Updating a Kubernetes Sourcegraph instance

Always refer to this page before upgrading Sourcegraph, as it comprehensively describes any special manual migration steps you must perform per-version.

Upgrade procedure

  1. Read our update policy to learn about Sourcegraph updates.
  2. Find the relevant entry for your update in the update notes on this page. If the notes indicate a patch release exists, target the highest one.
  3. After checking the relevant update notes, refer to either of the following guides to upgrade your instance:

Multi-version upgrade procedure

  1. Read our update policy to learn about Sourcegraph updates.
  2. Find the relevant entry for your update in the update notes on this page. If the notes indicate a patch release exists, target the highest one. These notes may contain relevant information about the infrastructure update such as resource requirement changes or versions of depencies (Docker, Kubernetes, externalized databases).
  3. After checking the relevant update notes, refer to either of the following guides to upgrade your instance:

Unreleased

Upgrade notes for the next version will appear here.

v4.4.1 ➔ v4.4.2

Notes:

v4.3 ➔ v4.4.1

Upgrade notes for the next version will appear here.

v4.2 ➔ v4.3.1

Upgrade notes for the next version will appear here.

v4.1 ➔ v4.2.1

Notes:

  • The worker-executors Service object is now included in manifests generated using kustomize. This object was already introduced in the base manifest, but omitted from manifests generated using kustomize. Its purpose is to enable ingested executor metrics to be scraped by Prometheus. It should have no impact on behavior.

Notes:

  • minio has been replaced with blobstore. Please see the update notes here: https://docs.sourcegraph.com/admin/how-to/blobstore_update_notes

  • This upgrade adds a node-exporter DaemonSet, which collects crucial machine-level metrics that help Sourcegraph scale your deployment.

    • Note: Similarly to cadvisor, node-exporter:
      • runs as a DaemonSet
      • needs to mount various read-only directories from the host machine (/, /proc, and /sys)
      • ideally shares the machine's PID namespace

    For more information, see deploy-sourcegraph-helm's Changelog or contact customer support.

v4.0 ➔ v4.1.3

v3.43 ➔ v4.0

Patch releases:

  • v4.0.1

Notes:

  • jaeger-agent sidecars have been removed in favor of an OpenTelemetry Collector DaemonSet + Deployment configuration. See Configure a tracing backend section.
  • Exporting traces to an external observability backend is now available. Read the documentation to configure.
  • The bundled Jaeger instance is now disabled by default. It can be enabled if you do not wish to utilise your own external tracing backend.

v3.42 ➔ v3.43

Patch releases:

  • 3.43.1
  • 3.43.2

v3.41 ➔ v3.42

Patch releases:

  • 3.42.1
  • 3.42.2

v3.40 ➔ v3.41

Notes:

  • The Postgres DBs frontend and codeintel-db are now given 1 hour to begin accepting connections before Kubernetes restarts the containers. #4136

v3.39 ➔ v3.40

Patch releases:

  • v3.40.1
  • v3.40.2

Notes:

  • cadvisor now defaults to run in privileged mode. This allows cadvisor to collect out of memory events happening to containers which can be used to discover underprovisoned resources. This is disabled by default in non-privileged overlay. #4126
  • Updated the Nginx ingress controller to v1.2.0. Previously this image originated from quay.io, now it is pulled from the official k8s repository. A redeployment of the ingress controller may be necessary if your deployment used the manifests provided in configure/ingress-nginx. #4128
  • The alpine-3.12 docker images used as init containers for some deployments have been replaced with images based on alpine-3.14. #4129

v3.38 ➔ v3.39

Notes:

  • Thecodeinsights-db container no longer uses TimescaleDB and is now based on the standard Postgres image sourcegraph/deploy-sourcegraph#4103. Metrics scraping is also enabled.
  • CAUTION: If you use a custom Code Insights postgres config, you must update the shared_preload_libraries list to remove timescaledb. The above PR demonstrates this change.

v3.37 ➔ v3.38

No upgrade notes.

v3.36 ➔ v3.37

Notes:

  • This release adds a new migrator initContainer to the frontend deployment to run database migrations. Confirm the environment variables on this new container match your database settings. Docs

v3.35 ➔ v3.36

Notes:

  • The backend service has been removed, so if you deploy with a method other than kubectl-apply-all.sh, a manual removal of the service may be necessary.

v3.34 ➔ v3.35

Patch releases:

  • v3.35.1

Notes:

  • The query-runner deployment has been removed, so if you deploy with a method other than the kubectl-apply-all.sh, a manual removal of the deployment may be necessary. Follow the standard upgrade procedure to upgrade your deployment.
  • There is a known issue with the Code Insights out-of-band settings migration not reaching 100% complete when encountering deleted users or organizations.

v3.33 ➔ v3.34

No upgrade notes.

v3.32 ➔ v3.33

No upgrade notes.

v3.31 ➔ v3.32

No upgrade notes.

v3.30 ➔ v3.31

Notes:

  • The built-in main Postgres (pgsql) and codeintel (codeintel-db) databases have switched to an alpine-based Docker image. Upon upgrading, Sourcegraph will need to re-index the entire database. All users that use our bundled (built-in) database instances must read through the 3.31 upgrade guide before upgrading.

v3.29 ➔ v3.30

Patch releases:

  • v3.30.1
  • v3.30.2
  • v3.30.3

Notes:

  • This upgrade removes the non-root overlay, in favor of using only the non-privileged overlay for deploying Sourcegraph in secure environments. If you were previously deploying using the non-root overlay, you should now generate overlays using the non-privileged overlay.

v3.28 ➔ v3.29

Notes:

  • This upgrade adds a new worker service that runs a number of background jobs that were previously run in the frontend service. See notes on deploying workers for additional details. Good initial values for CPU and memory resources allocated to this new service should match the frontend service.

v3.27 ➔ v3.28

Notes:

v3.26 ➔ v3.27

Notes:

  • If you are using an external database, upgrade your database to Postgres 12 or above prior to upgrading Sourcegraph. No action is required if you are using the supplied database images.

v3.25 ➔ v3.26

No upgrade notes.

v3.24 ➔ v3.25

Notes:

  • Go 1.15 introduced changes to SSL/TLS connection validation which requires certificates to include a SAN. This field was not included in older certificates and clients relied on the CN field. You might see an error like x509: certificate relies on legacy Common Name field. We recommend that customers using Sourcegraph with an external database and and connecting to it using SSL/TLS check whether the certificate is up to date.

v3.23 ➔ v3.24

No upgrade notes.

v3.22 ➔ v3.23

No upgrade notes.

v3.21 ➔ v3.22

Notes:

v3.20 ➔ v3.21

Notes:

  • This release introduces a second database instance, codeintel-db. If you have configured Sourcegraph with an external database, then update the CODEINTEL_PG* environment variables to point to a new external database as described in the external database documentation. Again, these must not point to the same database or the Sourcegraph instance will refuse to start.

v3.19 ➔ v3.20

No upgrade notes.

v3.18 ➔ v3.19

Notes:

  • WARNING: If you use an overlay that does not reference one of the provided overlays, please add - ../bases/pvcs as an additional base to your kustomization.yaml file. Otherwise the PVCs could be pruned if kubectl apply -prune is used.

v3.17 ➔ v3.18

No upgrade notes.

v3.16 ➔ v3.17

No upgrade notes.

v3.15 ➔ v3.16

Notes:

  • The following deployments have had their strategy changed from rolling to recreate. This change was made to avoid two pods writing to the same volume and causing corruption. No special action is needed to apply the change.
    • redis-cache
    • redis-store
    • pgsql
    • precise-code-intel-bundle-manager
    • prometheus

v3.14 ➔ v3.15

Prometheus and Grafana resource requirements increase

Resource requests and limits for Grafana and Prometheus are now equal to the following:

  • Grafana 100Mi -> 512Mi
  • Prometheus: 500M -> 3G

This change was made to ensure that even if another Sourcegraph service starts consuming more memory than expected and the Kubernetes node has been over-provisioned, that Sourcegraph's monitoring will still have enough memory to run and monitor / send alerts to the site admin. For additional information see #638

You may run the following commands to remove the now unused resources:

kubectl delete svc lsif-server
kubectl delete deployment lsif-server
kubectl delete pvc lsif-server

Configuration

In Sourcegraph 3.0 all site configuration has been moved out of the config-file.ConfigMap.yaml and into the PostgreSQL database. We have an automatic migration if you use version 3.2 or before. Please do not upgrade directly from 2.x to 3.3 or higher.

After running 3.0, you should visit the configuration page (/site-admin/configuration) and the management console and ensure that your configuration is as expected. In some rare cases, automatic migration may not be able to properly carry over some settings and you may need to reconfigure them.

A new sourcegraph-frontend service type

The type of the sourcegraph-frontend service (base/frontend/sourcegraph-frontend.Service.yaml) has changed from NodePort to ClusterIP. Directly applying this change will fail. Instead, you must delete the old service and then create the new one (this will result in a few seconds of downtime):

kubectl delete svc sourcegraph-frontend
kubectl apply -f base/frontend/sourcegraph-frontend.Service.yaml