Updating a Kubernetes Sourcegraph instance

This document describes the exact changes needed to update a Kubernetes Sourcegraph instance. Follow the recommended method of upgrading a Kubernetes cluster. Always refer to this page before upgrading Sourcegraph, as it comprehensively describes the steps needed to upgrade, and any manual migration steps you must perform.

  1. Read our update policy to learn about Sourcegraph updates.
  2. Find the relevant entry for your update in the update notes on this page.
  3. After checking the relevant update notes, refer to the Sourcegraph with Kubernetes upgrade guide to upgrade your instance.

3.38 -> 3.39

Thecodeinsights-db container no longer uses TimescaleDB and is now based on the standard Postgres image sourcegraph/deploy-sourcegraph#4103. Metrics scraping is also enabled.

CAUTION If you use a custom Code Insights postgres config, you must update the shared_preload_libraries list to remove timescaledb. The above PR demonstrates this change.

Follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.38.0 -> 3.38.1

Follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.37 -> 3.38

Follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.36 -> 3.37

This release adds a new migrator initContainer to the frontend deployment to run database migrations. Confirm the environment variables on this new container match your database settings. Docs

Follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.35 -> 3.36

The backend service has been removed, so if you deploy with a method other than kubectl-apply-all.sh, a manual removal of the service may be necessary.

Follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.34 -> 3.35.1

Due to issues related to Code Insights on the 3.35.0 release, Users are advised to upgrade directly to 3.35.1.

The query-runner deployment has been removed, so if you deploy with a method other than the kubectl-apply-all.sh, a manual removal of the deployment may be necessary. Follow the standard upgrade procedure to upgrade your deployment.

There is a known issue with the Code Insights out-of-band settings migration not reaching 100% complete when encountering deleted users or organizations.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.33 -> 3.34

No manual migration is required - follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.32 -> 3.33

No manual migration is required - follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.31 -> 3.32

No manual migration is required - follow the standard upgrade procedure to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.30.3 -> 3.31

The built-in main Postgres (pgsql) and codeintel (codeintel-db) databases have switched to an alpine-based Docker image. Upon upgrading, Sourcegraph will need to re-index the entire database.

If you have already upgraded to 3.30.3, which uses the new alpine-based Docker images, all users that use our bundled (built-in) database instances should have already performed the necessary re-indexing.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.30.x -> 3.31

The built-in main Postgres (pgsql) and codeintel (codeintel-db) databases have switched to an alpine-based Docker image. Upon upgrading, Sourcegraph will need to re-index the entire database.

All users that use our bundled (built-in) database instances must read through the 3.31 upgrade guide before upgrading.

3.29 -> 3.30.3

This upgrade removes the non-root overlay, in favor of using only the non-privileged overlay for deploying Sourcegraph in secure environments. If you were previously deploying using the non-root overlay, you should now generate overlays using the non-privileged overlay.

No other manual migration is required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.28 -> 3.29

This upgrade adds a new worker service that runs a number of background jobs that were previously run in the frontend service. See notes on deploying workers for additional details. Good initial values for CPU and memory resources allocated to this new service should match the frontend service.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.27 -> 3.28

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.26 -> 3.27

Warning: ⚠️ Sourcegraph 3.27 now requires Postgres 12+.

If you are using an external database, upgrade your database to Postgres 12 or above prior to upgrading Sourcegraph. No action is required if you are using the supplied database images.

Note: The Postgres 12 database migration scales with the size of your database, and the resources provided to the container. Expect to have downtime relative to the size of your database. Additionally, you must ensure that have enough storage space to accommodate the migration. A rough guide would be 2x the current on-disk database size

Warning: ⚠️ We have updated the default replicas for sourcegraph-frontend and precise-code-intel-worker to 2. If you use a custom value, make sure you do not merge the replica change.

Afterwards, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.25 -> 3.26

No manual migration required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.24 -> 3.25

  • Go 1.15 introduced changes to SSL/TLS connection validation which requires certificates to include a SAN. This field was not included in older certificates and clients relied on the CN field. You might see an error like x509: certificate relies on legacy Common Name field. We recommend that customers using Sourcegraph with an external database and and connecting to it using SSL/TLS check whether the certificate is up to date.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.23 -> 3.24

No manual migration required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.22 -> 3.23

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.21 -> 3.22

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

This upgrade removes the code intel bundle manager. This service has been deprecated and all references to it have been removed.

This upgrade also adds a MinIO container that doesn't require any custom configuration. You can find more detailed documentation in https://docs.sourcegraph.com/admin/external_services/object_storage.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.20 -> 3.21

Follow the standard upgrade method to upgrade your deployment.

This release introduces a second database instance, codeintel-db. If you have configured Sourcegraph with an external database, then update the CODEINTEL_PG* environment variables to point to a new external database as described in the external database documentation. Again, these must not point to the same database or the Sourcegraph instance will refuse to start.

If you wish to keep existing LSIF data

Warning: Do not upgrade out of the 3.21.x release branch until you have seen the log message indicating the completion of the LSIF data migration, or verified that the /lsif-storage/dbs directory on the precise-code-intel-bundle-manager volume is empty. Otherwise, you risk data loss for precise code intelligence.

If you had LSIF data uploaded prior to upgrading to 3.21.0, there is a background migration that moves all existing LSIF data into the codeintel-db upon upgrade. Once this process completes, the /lsif-storage/dbs directory on the precise-code-intel-bundle-manager volume should be empty, and the bundle manager should print the following log message:

Migration to Postgres has completed. All existing LSIF bundles have moved to the path /lsif-storage/db-backups and can be removed from the filesystem to reclaim space.

Wait for the above message to be printed in docker logs precise-code-intel-bundle-manager before upgrading to the next Sourcegraph version.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.20

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.19

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

Warning: If you use an overlay that does not reference one of the provided overlays, please add - ../bases/pvcs as an additional base to your kustomization.yaml file. Otherwise the PVCs could be pruned if kubectl apply -prune is used.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.18

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.17

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.16

No manual migration is required, follow the standard upgrade method to upgrade your deployment.

Note: The following deployments have had their strategy changed from rolling to recreate:

  • redis-cache
  • redis-store
  • pgsql
  • precise-code-intel-bundle-manager
  • prometheus

This change was made to avoid two pods writing to the same volume and causing corruption. No special action is needed to apply the change.

How smooth was this upgrade process for you? You can give us your feedback on this upgrade by filling out this feedback form.

3.15

Note: Prometheus and Grafana resource requirements increase

Resource requests and limits for Grafana and Prometheus are now equal to the following:

  • Grafana 100Mi -> 512Mi
  • Prometheus: 500M -> 3G

This change was made to ensure that even if another Sourcegraph service starts consuming more memory than expected and the Kubernetes node has been over-provisioned, that Sourcegraph's monitoring will still have enough memory to run and monitor / send alerts to the site admin. For additional information see #638

(optional) Keep LSIF data through manual migration

If you have previously uploaded LSIF precise code intelligence data and wish to retain it after upgrading, you will need to perform this migration.

Skipping the migration

If you choose not to migrate the data, Sourcegraph will use search-based code intelligence until you upload LSIF data again.

You may run the following commands to remove the now unused resources:

kubectl delete svc lsif-server
kubectl delete deployment lsif-server
kubectl delete pvc lsif-server

Migrating

The lsif-server service has been replaced by a trio of services defined in precise-code-intel, and the persistent volume claim in which lsif-server stored converted LSIF uploads has been replaced by bundle storage.

Upgrading to 3.15 will create a new empty volume for LSIF data. Without any action, the LSIF data previously uploaded to the instance will be lost. To retain old LSIF data, perform the following migration steps. This will cause some temporary downtime for precise code intelligence.

Migrating

  1. Deploy 3.15. This will create a bundle-manager persistent volume claim.
  2. Release the claims to old and new persistent volumes by taking down lsif-server and precise-code-intel-bundle-manager.
kubectl delete svc lsif-server
kubectl delete deployment lsif-server
kubectl delete deployment precise-code-intel-bundle-manager
  1. Deploy the lsif-server-migrator deployment to transfer the data from the old volume to the new volume.
kubectl apply -f configure/lsif-server-migrator/lsif-server-migrator.Deployment.yaml
  1. Watch the output of the lsif-server-migrator until the copy completes ('Copy complete!').
kubectl logs lsif-server-migrator
  1. Tear down the deployment and re-create the bundle manager deployment.
kubectl delete deployment lsif-server-migrator
./kubectl-apply-all.sh
  1. Remove the old persistent volume claim.
kubectl delete pvc lsif-server

3.11

In 3.11 we removed the management console. If you make use of CRITICAL_CONFIG_FILE or SITE_CONFIG_FILE, please refer to the migration notes for Sourcegraph 3.11+.

3.10

In 3.9 we migrated indexed-search to a StatefulSet. However, we didn't migrate the indexed-search service to a headless service. You can't mutate a service, so you will need to replace the service before running kubectl-apply-all.sh:

# Replace since we can't mutate services
kubectl replace --force -f base/indexed-search/indexed-search.Service.yaml

# Now apply all so frontend knows how to speak to the new service address
# for indexed-search
./kubectl-apply-all.sh

3.9

In 3.9 indexed-search is migrated from a Kubernetes Deployment to a StatefulSet. By default Kubernetes will assign a new volume to indexed-search, leading to it being unavailable while it reindexes. To avoid that we need to update the PersistentVolume's claim to the new indexed-search pod (from indexed-search to data-indexed-search-0. This can be achieved by running the commands in the script below before upgrading. Please read the script closely to understand what it does before following it.

# Set the reclaim policy to retain so when we delete the volume claim the volume is not deleted.
kubectl patch pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' $(kubectl get pv -o json | jq -r '.items[] | select(.spec.claimRef.name == "indexed-search").metadata.name')

# Stop indexed search so we can migrate it. This means indexed search will be down!
kubectl scale deploy/indexed-search --replicas=0

# Remove the existing claim on the volume
kubectl delete pvc indexed-search

# Move the claim to data-indexed-search-0, which is the name created by stateful set.
kubectl patch pv -p '{"spec":{"claimRef":{"name":"data-indexed-search-0","uuid":null}}}' $(kubectl get pv -o json | jq -r '.items[] | select(.spec.claimRef.name == "indexed-search").metadata.name')

# Create the stateful set
kubectl apply -f base/indexed-search/indexed-search.StatefulSet.yaml

3.8

If you're deploying Sourcegraph into a non-default namespace, refer to "Use non-default namespace" in docs/configure.md for further configuration instructions.

3.7.2

Before upgrading or downgrading 3.7, please consult the v3.7.2 migration guide to ensure you have enough free disk space.

3.0

🚨 If you have not migrated off of helm yet, please refer to helm.migrate.md before reading the following notes for migrating to Sourcegraph 3.0.

🚨 Please upgrade your Sourcegraph instance to 2.13.x before reading the following notes for migrating to Sourcegraph 3.0.

Configuration

In Sourcegraph 3.0 all site configuration has been moved out of the config-file.ConfigMap.yaml and into the PostgreSQL database. We have an automatic migration if you use version 3.2 or before. Please do not upgrade directly from 2.x to 3.3 or higher.

After running 3.0, you should visit the configuration page (/site-admin/configuration) and the management console and ensure that your configuration is as expected. In some rare cases, automatic migration may not be able to properly carry over some settings and you may need to reconfigure them.

sourcegraph-frontend service type

The type of the sourcegraph-frontend service (base/frontend/sourcegraph-frontend.Service.yaml) has changed from NodePort to ClusterIP. Directly applying this change will fail. Instead, you must delete the old service and then create the new one (this will result in a few seconds of downtime):

kubectl delete svc sourcegraph-frontend
kubectl apply -f base/frontend/sourcegraph-frontend.Service.yaml

Language server deployment

Sourcegraph 3.0 removed lsp-proxy and automatic language server deployment in favor of Sourcegraph extensions. As a consequence, Sourcegraph 3.0 does not automatically run or manage language servers. If you had code intelligence enabled in 2.x, you will need to follow the instructions for each language extension and deploy them individually. Read the code intelligence documentation.

HTTPS / TLS

Sourcegraph 3.0 removed HTTPS / TLS features from Sourcegraph in favor of relying on Kubernetes Ingress Resources. As a consequence, Sourcegraph 3.0 does not expose TLS as the NodePort 30433. Instead you need to ensure you have setup and configured either an ingress controller (recommended) or an explicit NGINX service. See ingress controller documentation, NGINX service documentation, and configure TLS/SSL documentation.

If you previously configured TLS_KEY and TLS_CERT environment variables, you can remove them from base/frontend/sourcegraph-frontend.Deployment.yaml

Postgres 11.1

Sourcegraph 3.0 ships with Postgres 11.1. The upgrade procedure is mostly automatic. Please read this page for detailed information.

2.12

Beginning in version 2.12.0, Sourcegraph's Kubernetes deployment requires an Enterprise license key. Follow the steps in docs/configure.md.