Operations guides for Sourcegraph with Kubernetes

Operations guides specific to managing Sourcegraph with Kubernetes installations.

Trying to deploy Sourcegraph with Kubernetes? Refer to our installation guide.

Featured guides

Configure
Configure your Sourcegraph deployment with our deployment reference. Upgrade
Upgrade your deployment to the latest Sourcegraph release. Troubleshoot
Troubleshoot common issues with your Sourcegraph instance.

Deploy

Refer to our installation guide for details on how to deploy Sourcegraph.

Migrating from another deployment type? Refer to our migration guides.

Applying manifests

In general, Sourcegraph with Kubernetes is deployed by applying the Kubernetes manifests in our deploy-sourcegraph reference repository - see our configuration guide for more details.

We provide a kubectl-apply-all.sh script that you can use to do this, usually by running the following from the root directory of the deploy-sourcegraph reference repository:

./kubectl-apply-all.sh

Once you have applied your changes:

Watch - verify your deployment has started:
```
kubectl get pods -A -o wide --watch
```
Port-foward - verify Sourcegraph is running by temporarily making the frontend port accessible:
```
kubectl port-forward svc/sourcegraph-frontend 3080:30080
```
Log in - browse to your Sourcegraph deployment, login, and verify the instance is working as expected.

Configure

We strongly recommend referring to our Configuration guide to learn about how to configure your Sourcegraph with Kubernetes instance.

Upgrade

See the Updating Sourcegraph docs on how to upgrade.
See the Updating a Kubernetes Sourcegraph instance docs for details on changes in each version to determine if manual migration steps are necessary.

List pods in cluster

List all pods in your cluster and the corresponding health status of each pod:

kubectl get pods -o=wide

Tail logs for specific pod

Tail the logs for the specified pod:

kubectl logs -f $POD_NAME

If Sourcegraph is unavailable and the sourcegraph-frontend-* pod(s) are not in status Running, then view their logs with kubectl logs -f sourcegraph-frontend-$POD_ID (filling in $POD_ID from the kubectl get pods output). Inspect both the log messages printed at startup (at the beginning of the log output) and recent log messages.

Retrieving resource information

Display detailed information about the status of a single pod:

kubectl describe $POD_NAME

List all Persistent Volume Claims (PVCs) and their statuses:

kubectl get pvc

List all Persistent Volumes (PVs) that have been provisioned. In a healthy cluster, there should be a one-to-one mapping between PVs and PVCs:

kubectl get pv

List all events in the cluster's history:

kubectl get events

Delete failing pod so it gets recreated, possibly on a different node:

kubectl delete pod $POD_NAME

Remove all pods from a node and mark it as unschedulable to prevent new pods from arriving

kubectl drain --force --ignore-daemonsets --delete-local-data $NODE

Restarting Sourcegraph Instance:

kubectl rollout restart deployment sourcegraph-frontend

Access the database

Get the id of one pgsql Pod:

kubectl get pods -l app=pgsql
NAME                     READY     STATUS    RESTARTS   AGE
pgsql-76a4bfcd64-rt4cn   2/2       Running   0          19m

Make sure you are operating under the correct namespace (i.e. add -n prod if your pod is under the prod namespace).

Open a PostgreSQL interactive terminal:

kubectl exec -it pgsql-76a4bfcd64-rt4cn -- psql -U sg

Run your SQL query:

SELECT * FROM users;

Backup and restore

The following instructions are specific to backing up and restoring the sourcegraph databases in a Kubernetes deployment. These do not apply to other deployment types.

Back up Sourcegraph databases

These instructions will back up the primary sourcegraph database and the codeintel database.

A. Verify deployment running

kubectl get pods -A

B. Stop all connections to the database by removing the frontend deployment

kubectl scale --replicas=0 deployment/sourcegraph-frontend
# or
kubectl delete deployment sourcegraph-frontend

C. Check for corrupt database indexes. If amcheck returns errors, please reach out to [email protected]

create extension amcheck;

select bt_index_parent_check(c.oid, true), c.relname, c.relpages
from pg_index i
join pg_opclass op ON i.indclass[0] = op.oid
join pg_am am ON op.opcmethod = am.oid
join pg_class c ON i.indexrelid = c.oid
join pg_namespace n ON c.relnamespace = n.oid
where am.amname = 'btree'
-- Don't check temp tables, which may be from another session:
and c.relpersistence != 't'
-- Function may throw an error when this is omitted:
and i.indisready AND i.indisvalid;

D. Generate the database dumps

kubectl exec -it $pgsql_POD_NAME -- bash -c 'pg_dump -C --username sg sg' > sourcegraph_db.out
kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'pg_dump -C --username sg sg' > codeintel_db.out

Ensure the sourcegraph_db.out and codeintel_db.out files are moved to a safe and secure location.

Restore Sourcegraph databases

Restoring Sourcegraph databases into a new environment

The following instructions apply only if you are restoring your databases into a new deployment of Sourcegraph ie: a new virtual machine

If you are restoring a previously running environment, see the instructions for restoring a previously running deployment

A. Copy the database dump files (eg. sourcegraph_db.out and codeintel_db.out) into the root of the deploy-sourcegraph directory

B. Start the database services by running the following command from the root of the deploy-sourcegraph directory

kubectl rollout restart deployment pgsql
kubectl rollout restart deployment codeintel-db

C. Copy the database files into the pods by running the following command from the root of the deploy-sourcegraph directory

kubectl cp sourcegraph_db.out $NAMESPACE/$pgsql_POD_NAME:/tmp/sourcegraph_db.out
kubectl cp codeintel_db.out $NAMESPACE/$codeintel-db_POD_NAME:/tmp/codeintel_db.out

D. Restore the databases

kubectl exec -it $pgsql_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg'
kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/condeintel_db.out sg'

E. Check for corrupt database indexes. If amcheck returns errors, please reach out to [email protected]

create extension amcheck;

select bt_index_parent_check(c.oid, true), c.relname, c.relpages
from pg_index i
join pg_opclass op ON i.indclass[0] = op.oid
join pg_am am ON op.opcmethod = am.oid
join pg_class c ON i.indexrelid = c.oid
join pg_namespace n ON c.relnamespace = n.oid
where am.amname = 'btree'
-- Don't check temp tables, which may be from another session:
and c.relpersistence != 't'
-- Function may throw an error when this is omitted:
and i.indisready AND i.indisvalid;

F. Start the remaining Sourcegraph services by following the steps in applying manifests.

Restoring Sourcegraph databases into an existing environment

A. Stop the existing deployment by removing the frontend deployment

kubectl scale --replicas=0 deployment/sourcegraph-frontend
# or
kubectl delete deployment sourcegraph-frontend

B. Remove any existing volumes for the databases in the existing deployment

kubectl delete pvc pgsql
kubectl delete pvc codeintel-db
kubectl delete pv $pgsql_PV_NAME --force
kubectl delete pv $codeintel-db_PV_NAME --force

C. Copy the database dump files (eg. sourcegraph_db.out and codeintel_db.out) into the root of the deploy-sourcegraph directory

D. Start the database services only

kubectl rollout restart deployment pgsql
kubectl rollout restart deployment codeintel-db

E. Copy the database files into the pods by running the following command from the root of the deploy-sourcegraph directory

kubectl cp sourcegraph_db.out $NAMESPACE/$pgsql_POD_NAME:/tmp/sourcegraph_db.out
kubectl cp codeintel_db.out $NAMESPACE/$codeintel-db_POD_NAME:/tmp/codeintel_db.out

F. Restore the databases

kubectl exec -it $pgsql_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/sourcegraph_db.out sg'
kubectl exec -it $codeintel-db_POD_NAME -- bash -c 'psql -v ERROR_ON_STOP=1 --username sg -f /tmp/condeintel_db.out sg'

G. Check for corrupt database indexes. If amcheck returns errors, please reach out to [email protected]

create extension amcheck;

select bt_index_parent_check(c.oid, true), c.relname, c.relpages
from pg_index i
join pg_opclass op ON i.indclass[0] = op.oid
join pg_am am ON op.opcmethod = am.oid
join pg_class c ON i.indexrelid = c.oid
join pg_namespace n ON c.relnamespace = n.oid
where am.amname = 'btree'
-- Don't check temp tables, which may be from another session:
and c.relpersistence != 't'
-- Function may throw an error when this is omitted:
and i.indisready AND i.indisvalid;

H. Start the remaining Sourcegraph services by following the steps in applying manifests.

Troubleshoot

See the Troubleshooting docs.