Administration troubleshooting
New state of 'nil' is invalid
Docker Toolbox on Windows: If you are using Docker Toolbox on Windows to run Sourcegraph, you may see an error in the frontend
log output:
frontend | frontend | frontend | frontend | New state of 'nil' is invalid.
After this error, no more frontend
log output is printed.
This problem is caused by docker/toolbox#695 in Docker Toolbox on Windows. To work around it, set the environment variable LOGO=false
, as in:
docker container run -e LOGO=false ... sourcegraph/server
See sourcegraph/sourcegraph#398 for more information.
Submitting a metrics dump
If you encounter performance or instability issues with Sourcegraph, we may ask you to submit a metrics dump to us. This allows us to inspect the performance and health of various parts of your Sourcegraph instance in the past and can often be the most effective way for us to identify the cause of your issue.
The metrics dump includes non-sensitive aggregate statistics of Sourcegraph like CPU & memory usage, number of successful and error requests between Sourcegraph services, and more. It does NOT contain sensitive information like code, repository names, user names, etc.
Docker Compose deployments
To create a metrics dump from a docker-compose deployment, follow these steps:
- Open a shell to the running
prometheus
container:
docker exec -it prometheus /bin/bash
- Inside the container bash shell trigger the creation of a Prometheus snapshot:
wget --post-data "" http://localhost:9090/api/v1/admin/tsdb/snapshot
- Find the created snapshot's name:
ls /prometheus/snapshots/
- Tar up the created snapshot
cd /prometheus/snapshots && tar -czvf /tmp/sourcegraph-metrics-dump.tgz <snapshot-name>
- Switch back to local shell (
exit
) and copy the metrics dump file to the host machine:
docker cp prometheus:/tmp/sourcegraph-metrics-dump.tgz sourcegraph-metrics-dump.tgz
Please then upload the sourcegraph-metrics-dump.tgz
file to Sourcegraph support so we can inspect it.
sourcegraph/server
deployments
Single-container To create a metrics dump from a single-container sourcegraph/server
deployment, follow these steps:
- Open a shell to the running container:
- Run
docker ps
to get the name of the Sourcegraph server container. - Run
docker exec -it <container name> /bin/bash
to open a bash shell.
- Run
- Inside the container bash shell trigger the creation of a Prometheus snapshot:
wget --post-data "" http://localhost:9090/api/v1/admin/tsdb/snapshot
- Tar up the created snapshot
cd ~/.sourcegraph/data/prometheus/snapshots && tar -czvf /tmp/sourcegraph-metrics-dump.tgz <snapshot-name>
- If needed, you can download the metrics dump to your local machine (current directory) using
scp
:
scp -r username@hostname:/tmp/sourcegraph-metrics-dump.tgz .
Please then upload the sourcegraph-metrics-dump.tgz
for Sourcegraph support to access it. If desired, we can send you a shared private Google Drive folder for the upload as it can sometimes be a few gigabytes.
Kubernetes deployments
If you're using the Kubernetes cluster deployment option,you can create a metrics dump as follows:
-
Open a shell to the running container
- Run
kubectl get pods
to get the name of the Prometheus pod. - Run
kubectl exec -it <pod-name> -- /bin/bash
.
- Run
-
Inside the container bash shell trigger the creation of a Prometheus snapshot:
wget --post-data "" http://localhost:9090/api/v1/admin/tsdb/snapshot
- Tar up the created snapshot
cd /prometheus/snapshots && tar -czvf /tmp/sourcegraph-metrics-dump.tgz <snapshot-name>
- Switch back to local shell and copy the metrics dump file over:
kubectl cp <podname>:/tmp/sourcegraph-metrics-dump.tgz /tmp/sourcegraph-metrics-dump.tgz
Again please then upload the sourcegraph-metrics-dump.tgz
for Sourcegraph support to access it.
Generating pprof profiles
Please follow these instructions to generate pprof profiles.