This is a guide for investigating errors and performance issues, with the goal of resolving the
issue or generating a high-quality issue report.
How to use this guide:
Scan through Specific scenarios to see if any of these applies to you. If
any do, follow the instructions in the subsection.
Scan through General scenarios to find which scenario(s) applies to you. If any do,
follow the instructions to update your instance or collect information for the issue report.
If you cannot resolve the issue on your own, file an issue on the Sourcegraph issue
tracker with the collected
information. Enterprise customers may alternatively file a support ticket or email
[email protected].
Specific scenarios
Scenario: search and code pages take a long time to load
If this is the case, this could indicate high gitserver load. To confirm, take the following steps:
Go to the Sourcegraph Internal > Gitserver rev2 dashboard.
Examine the "Echo Duration Seconds" dashboard (tracks the src_gitserver_echo_duration_seconds
metric) and "Commands running concurrently" dashboard (tracks the src_gitserver_exec_running
metric). If either of these is high (> 1s echo duration or 100s simultaneous execs), then this
indicates gitserver is under heavy load and likely the bottleneck.
Confirm your gitserver is not under-provisioned, by e.g. comparing its allocated resources with what the resource estimator shows.
Solution: set USE_ENHANCED_LANGUAGE_DETECTION=false in the Sourcegraph runtime
environment.
Scenario: no cloning, syncing, updating or deleting is happening
Observed state: Sourcegraph instance does not react to any updates to code hosts and no cloning is happening.
The cause of this state could be repo-updater queries that are too large for the limits of the running Postgres DB.
One symptom is seeing a line like the one below in the repo-updater logs:
t=2020-05-28T18:41:02+0000 lvl=eror msg=Syncer error="syncer.sync.store.upsert-repos: delete: driver: bad connection
or seeing the same error in the "Code host status panel" (Clicking the cloud icon).
The fix is to increase the memory on Postgres DB which will increase certain Postgres-internal limits and will allow
the queries from repo-updater to go through.
Another cause could be that the repo-updater is in a crash loop for some reason. If there are large numbers of repos
to be updated it could be from Out of memory errors. A fix here is to increase the memory for repo-updater instead.
General scenarios
This section contains a list of scenarios, each of which contains instructions that include
actions that are appropriate to the scenario. Note that more than one scenario may apply
to a given issue.
Scenario: the issue is NOT performance-related and there is a consistent reproduction.
Record the following information in the issue report:
When was the most recent update or deployment change applied?
Scenario: the issue is NOT performance-related, but it is hard to reproduce.
Without a consistent reproduction, the issue will be harder to diagnose, so we recommend trying to
find a repro if possible. If that isn't possible, file an issue with the following information:
Steps the user took before encountering the issue, including as much detail as possible.
Bad example: "User encountered a 502 error when trying to search for something."
Good example: "User encountered a 502 error on the search results page when trying to conduct a
global search for the following query. On refresh, the search worked with no error. The desired
result appears in our main repository, which is rather large (takes about a minute to fully
clone). The issue doesn't reproduce consistently, but we saw two other reports like this around
2pm PT yesterday, during peak usage hours."
If you know the approximate time the issue occurred or if there is a spike in error rate around a
certain time, copy the logs around that time.
Note any pattern in the issue reports. E.g., did users encountering the issue all visit the same
repository or belong to the same organization. Do site admins encounter the issue or only
non-admin users?
When was the most recent update or deployment change applied?
Scenario: the issue is performance-related and there is a consistent reproduction
Include the reproduction steps in the error report, along with relevant context (e.g., repository
size).
Bad example: "User did a search and it timed out."
Good example: "User issued the following search query in the following repository. The
repository is one of our larger repositories (takes about 1 minute to fully clone and the size
of the .git directory is 5GB). The results page took about 60 seconds to load and when it
finally did, the results was an error message that said 'timeout'".
Use Jaeger to drill down into slow requests and understand which
components of the request are slow. Remember that many Sourcegraph API requests identify the
Jaeger trace ID in the x-trace HTTP response header, which makes it easy to look up the trace
corresponding to a particular request.
Scenario: the issue is performance-related and there is NOT a consistent reproduction
Without a consistent reproduction, the issue will be harder to diagnose, so we recommend trying to
find a repro if possible. If that isn't possible, try the following:
Are there spikes in latencies or error rate over time?
Are there spikes in usage or traffic over time that correlate with when the issue is reported.
Are there spikes in memory usage, CPU, or disk usage over time?
If you know the approximate time the issue occurred or if there is a suspicious spike in metrics
around a certain time, check the logs around that time.
If the issue is ongoing or if you know the time during which the issue occurred, search
Jaeger for long-running request traces in the appropriate time window.
If Jaeger is unavailable, you can alternatively use the Go net/trace endpoint. (You will have
to scan the traces for each service to look for slow traces.)
If tracing points to a specific service as the source of high latency, examine the
logs and net/trace info for that service.
Scenario: multiple actions are slow or Sourcegraph as a whole feels sluggish
If Sourcegraph feels sluggish overall, the likely culprit is resource allocation.
If the metrics indicate high resource consumption, adjust the resource allocation higher.
If metrics are unavailable or inaccessible, here is a rough correspondence between end-user
slowness and the services that are usually the culprit:
Global search (i.e., no repository scope is specified) results page takes a long time to load.
Increase indexed-search memory limit or CPU limit. The number of indexed-search
shards can also be increased if using Sourcegraph on Kubernetes.
Search results show up quickly, but code snippets take awhile to populate. File contents take
awhile to load.
Increase gitserver memory usage. Gitserver memory may be the bottleneck, especially if there
are many repositories or repositories are large.
Increase number of gitserver shards. This can help if memory is the bottleneck. It can also
help if there are too many repositories per shard. Gitserver "shells out" to git for every
repository data request, so a high volume of user traffic that generates many simultaneous
requests for many repositories can lead to a spike in Linux process exec latency.
Increase memory and CPU limit of syntect-server. This helps if syntax highlighting is the
bottleneck.
Multiple UI pages take awhile to load.
Increase frontend CPU and memory limit.
Searches show intermittent HTTP 502 errors or timeouts, possibly concurrent with frontend
container restarts.
Increase frontend memory and CPU. This may indicate the frontend is running out of memory
when loading search results. This can be a problem when dealing with large monorepos.
If it is unclear which service is underallocated, examine Jaeger to
identify long-running traces and see which services take up the most time.
Once the dashboard appears, include screenshots of the entire dashboard in the issue report.
Include the logs of zoekt-webserver container in the indexed-search pods. If you are using single Docker container enable debug logs first.
Scenario: zoekt-webserver is in a CrashloopBackOff and err cannot allocate memory
Sourcegraph uses a mmap to store its indices. The default operating system limits on mmap counts may to be too low, which may result in out of memory exceptions.
If you are seeing this error on large scale deployments with a lot repos to be indexed, please use the following steps to verify the source of the errors:
Ensure pods are not actually running out of allocated memory and being OOMKilled by running (if they are, then you should give them more memory and the below will not help you):
kubectl top pod indexed-search-<pod_number>
kubectl describe nodes
On the host operating system execute sudo sysctl -n vm.max_map_count.
$ sysctl -n vm.max_map_count
65530
Calculate the number of repos in your deployment divided by the number of indexed-search replicas. For example:
250,000 repositories / 2 indexed-search repliacas = 125,000 repos to index per replica.
If the vm.max_map_count is lower than the result of the above calculation. Adjust the vm.max_map_count by executing:
sudo sysctl -w vm.max_map_count=262144
Verify the change.
$ sudo sysctl -n vm.max_map_count
262144
Ensure the change will persist a system reboot by updating the vm.max_map_count setting in /etc/sysctl.conf.
Actions
This section contains various actions that can be taken to collect information or update Sourcegraph
in order to resolve an error or performance issue. You should typically not read this section
directly, but start with the General scenarios section to determine which actions are
appropriate.
Check browser console
Open the browser JavaScript console (right-click in the browser > Inspect to open developer tools,
then click the Console tab).
Check browser network panel
Open the browser developer network page (right-click in the browser > Inspect to open developer tools,
then click the Network tab).
Check the waterfall diagram at the top and the Waterfall column in the list of network requests to
quickly identify high-latency requests.
Clicking on a request will open up a panel that provides additional details about the request.
If a GraphQL request is taking a long time, you should obtain its Jaeger trace ID by inspecting
the Headers tab of this panel and finding the X-Trace or x-trace response header value. Once
you've obtained this trace ID, look it up in Jaeger.
You can check Preserve log to preserve the list of requests across page loads and reloads.
Memory:process_resident_memory_bytes is a gauge that tracks memory usage per backend process.
Example: process_resident_memory_bytes{app="indexed-search"} shows memory usage for each
indexed-search instance.
CPU:process_cpu_seconds_total is a counter that tracks cumulative CPU seconds used .
Example: rate(process_cpu_seconds_total{app="sourcegraph-frontend"}[1m]) shows average CPU usage
for each sourcegraph-frontend instance over the last minute.
Disk:gitserver_disk_free_percent is a gauge that tracks free disk space on gitserver.
Check end-user stats
Go to /site-admin/usage-statistics to view daily, weekly, and monthly user statistics.
To drill down (e.g., into sub-daily traffic, visits per page type, latencies, etc.), access
Grafana and visit the Sourcegraph Internal > HTTP dashboard page, which
includes the following panels:
QPS by Status Code
QPS by URL Route
P90 Request Duration (request latency at the 90th percentile)
Grafana contains ready-made dashboards derived from Prometheus metrics. Any chart in Grafana
mentioned here can be viewed in Prometheus by clicking the dropdown menu next to the Grafana panel
title > Edit > copying the expression in the Metrics field.
Search for a matching span by setting the appropriate fields in the sidebar.
2 ways: start with a span ID, or manually locate your span by searching the Jaeger GUI
Examine logs
If you are using the single Docker container or Docker Compose deployment option, logs are printed
to stdout and stderr. You should be able to access these using your infrastructure provider's
standard log viewing mechanism.
If you are using Kubernetes,
Retrieve logs with kubectl logs $POD_ID.
Tail logs with kubectl logs -f $POD_ID.
If a pod container died, you can access the previous container logs with kubectl logs -p $POD_ID. This can be useful for diagnosing why a container crashed.
You can tail logs for all pods associated with a given deployment: kubectl logs -f deployment/sourcegraph-frontend --container=frontend --since=10m
Examine Go net/trace
Each core service has an endpoint which displays traces using Go's
net/trace package.
To access this data,
First ensure you are logged in as a site admin.
Go to the URL path /-/debug. This page should show a list of links with the names of each core
service (e.g., frontend, gitserver, etc.)
Click on the service you'd like to examine.
Click "Requests`. This brings you to a page where you can view traces for that service.
You can filter to traces by duration or error state.
You can show histograms of durations by minute, hour, or in total (since the process started)
On older versions of Sourcegraph on Kubernetes, the /-/debug URL path may be inaccessible. If this
is the case, you'll need to forward port 6060 on the main container of a given pod to access its
traces. For example, to access to traces of the first gitserver shard,
kubectl port-forward gitserver-0 6060
Go to http://localhost:6060 in your browser, and click on "Requests".
Copy configuration
Go the the URL path /site-admin/report-bug to obtain an all-in-one text box of all Sourcegraph
configuration (which includes site configuration, code host configuration, and global
settings). This lets you easily copy all configuration to an issue report (NOTE: remember to redact
any secrets).
Collect instance stats
The following statistics are useful background context when reporting a performance issue:
Number of repositories (can be found on the /site-admin/repositories page, search for "repositories total")
Size distribution of repositories (e.g., are there one or more large "monorepos" that contain most of the code?)
Number of users and daily usage stats from /site-admin/usage-statistics