Alert solutions

This document contains possible solutions for when you find alerts are firing in Sourcegraph's monitoring. If your alert isn't mentioned here, or if the solution doesn't help, contact us for assistance.

To learn more about Sourcegraph's alerting and how to set up alerts, see our alerting guide.

frontend: 99th_percentile_search_request_duration

99th percentile successful search request duration over 5m

Descriptions

warning frontend: 20s+ 99th percentile successful search request duration over 5m

Possible solutions

Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_request_duration"
]

_{Managed by the Sourcegraph Search team.}

frontend: 90th_percentile_search_request_duration

90th percentile successful search request duration over 5m

Descriptions

warning frontend: 15s+ 90th percentile successful search request duration over 5m

Possible solutions

Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_request_duration"
]

_{Managed by the Sourcegraph Search team.}

frontend: hard_timeout_search_responses

hard timeout search responses every 5m

Descriptions

warning frontend: 2%+ hard timeout search responses every 5m for 15m0s
critical frontend: 5%+ hard timeout search responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_timeout_search_responses",
  "critical_frontend_hard_timeout_search_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: hard_error_search_responses

hard error search responses every 5m

Descriptions

warning frontend: 2%+ hard error search responses every 5m for 15m0s
critical frontend: 5%+ hard error search responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_responses",
  "critical_frontend_hard_error_search_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: partial_timeout_search_responses

partial timeout search responses every 5m

Descriptions

warning frontend: 5%+ partial timeout search responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: search_alert_user_suggestions

search alert user suggestions shown every 5m

Descriptions

warning frontend: 5%+ search alert user suggestions shown every 5m for 15m0s

Possible solutions

This indicates your user`s are making syntax errors or similar user errors.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_search_alert_user_suggestions"
]

_{Managed by the Sourcegraph Search team.}

frontend: page_load_latency

90th percentile page load latency over all routes over 10m

Descriptions

critical frontend: 2s+ 90th percentile page load latency over all routes over 10m

Possible solutions

Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels.
Explore the data returned by the query in the dashboard panel and filter by different labels to identify any patterns
Trace a request to see what the slowest part is: https://docs.sourcegraph.com/admin/observability/tracing
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_page_load_latency"
]

_{Managed by the Sourcegraph Cloud Software-as-a-Service team.}

frontend: blob_load_latency

90th percentile blob load latency over 10m

Descriptions

critical frontend: 5s+ 90th percentile blob load latency over 10m

Possible solutions

Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels.
Trace a request to see what the slowest part is: https://docs.sourcegraph.com/admin/observability/tracing
Check that gitserver containers have enough CPU/memory and are not getting throttled.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_blob_load_latency"
]

_{Managed by the Sourcegraph Core application team.}

frontend: 99th_percentile_search_codeintel_request_duration

99th percentile code-intel successful search request duration over 5m

Descriptions

warning frontend: 20s+ 99th percentile code-intel successful search request duration over 5m

Possible solutions

Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
This alert may indicate that your instance is struggling to process symbols queries on a monorepo, learn more here.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_codeintel_request_duration"
]

_{Managed by the Sourcegraph Search team.}

frontend: 90th_percentile_search_codeintel_request_duration

90th percentile code-intel successful search request duration over 5m

Descriptions

warning frontend: 15s+ 90th percentile code-intel successful search request duration over 5m

Possible solutions

Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
This alert may indicate that your instance is struggling to process symbols queries on a monorepo, learn more here.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_codeintel_request_duration"
]

_{Managed by the Sourcegraph Search team.}

frontend: hard_timeout_search_codeintel_responses

hard timeout search code-intel responses every 5m

Descriptions

warning frontend: 2%+ hard timeout search code-intel responses every 5m for 15m0s
critical frontend: 5%+ hard timeout search code-intel responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_timeout_search_codeintel_responses",
  "critical_frontend_hard_timeout_search_codeintel_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: hard_error_search_codeintel_responses

hard error search code-intel responses every 5m

Descriptions

warning frontend: 2%+ hard error search code-intel responses every 5m for 15m0s
critical frontend: 5%+ hard error search code-intel responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_codeintel_responses",
  "critical_frontend_hard_error_search_codeintel_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: partial_timeout_search_codeintel_responses

partial timeout search code-intel responses every 5m

Descriptions

warning frontend: 5%+ partial timeout search code-intel responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_codeintel_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: search_codeintel_alert_user_suggestions

search code-intel alert user suggestions shown every 5m

Descriptions

warning frontend: 5%+ search code-intel alert user suggestions shown every 5m for 15m0s

Possible solutions

This indicates a bug in Sourcegraph, please open an issue.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_search_codeintel_alert_user_suggestions"
]

_{Managed by the Sourcegraph Search team.}

frontend: 99th_percentile_search_api_request_duration

99th percentile successful search API request duration over 5m

Descriptions

warning frontend: 50s+ 99th percentile successful search API request duration over 5m

Possible solutions

Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_api_request_duration"
]

_{Managed by the Sourcegraph Search team.}

frontend: 90th_percentile_search_api_request_duration

90th percentile successful search API request duration over 5m

Descriptions

warning frontend: 40s+ 90th percentile successful search API request duration over 5m

Possible solutions

Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_api_request_duration"
]

_{Managed by the Sourcegraph Search team.}

frontend: hard_error_search_api_responses

hard error search API responses every 5m

Descriptions

warning frontend: 2%+ hard error search API responses every 5m for 15m0s
critical frontend: 5%+ hard error search API responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_api_responses",
  "critical_frontend_hard_error_search_api_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: partial_timeout_search_api_responses

partial timeout search API responses every 5m

Descriptions

warning frontend: 5%+ partial timeout search API responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_api_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: search_api_alert_user_suggestions

search API alert user suggestions shown every 5m

Descriptions

warning frontend: 5%+ search API alert user suggestions shown every 5m

Possible solutions

This indicates your user`s search API requests have syntax errors or a similar user error. Check the responses the API sends back for an explanation.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_search_api_alert_user_suggestions"
]

_{Managed by the Sourcegraph Search team.}

frontend: internal_indexed_search_error_responses

internal indexed search error responses every 5m

Descriptions

warning frontend: 5%+ internal indexed search error responses every 5m for 15m0s

Possible solutions

Check the Zoekt Web Server dashboard for indications it might be unhealthy.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_internal_indexed_search_error_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: internal_unindexed_search_error_responses

internal unindexed search error responses every 5m

Descriptions

warning frontend: 5%+ internal unindexed search error responses every 5m for 15m0s

Possible solutions

Check the Searcher dashboard for indications it might be unhealthy.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_internal_unindexed_search_error_responses"
]

_{Managed by the Sourcegraph Search team.}

frontend: internalapi_error_responses

internal API error responses every 5m by route

Descriptions

warning frontend: 5%+ internal API error responses every 5m by route for 15m0s

Possible solutions

May not be a substantial issue, check the frontend logs for potential causes.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_internalapi_error_responses"
]

_{Managed by the Sourcegraph Cloud Software-as-a-Service team.}

frontend: 99th_percentile_gitserver_duration

99th percentile successful gitserver query duration over 5m

Descriptions

warning frontend: 20s+ 99th percentile successful gitserver query duration over 5m

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_gitserver_duration"
]

_{Managed by the Sourcegraph Core application team.}

frontend: gitserver_error_responses

gitserver error responses every 5m

Descriptions

warning frontend: 5%+ gitserver error responses every 5m for 15m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_gitserver_error_responses"
]

_{Managed by the Sourcegraph Core application team.}

frontend: observability_test_alert_warning

warning test alert metric

Descriptions

warning frontend: 1+ warning test alert metric

Possible solutions

This alert is triggered via the triggerObservabilityTestAlert GraphQL endpoint, and will automatically resolve itself.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_observability_test_alert_warning"
]

_{Managed by the Sourcegraph Devops team.}

frontend: observability_test_alert_critical

critical test alert metric

Descriptions

critical frontend: 1+ critical test alert metric

Possible solutions

This alert is triggered via the triggerObservabilityTestAlert GraphQL endpoint, and will automatically resolve itself.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_observability_test_alert_critical"
]

_{Managed by the Sourcegraph Devops team.}

frontend: cloudkms_cryptographic_requests

cryptographic requests to Cloud KMS every 1m

Descriptions

warning frontend: 15000+ cryptographic requests to Cloud KMS every 1m for 5m0s
critical frontend: 30000+ cryptographic requests to Cloud KMS every 1m for 5m0s

Possible solutions

Revert recent commits that cause extensive listing from "external_services" and/or "user_external_accounts" tables.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_cloudkms_cryptographic_requests",
  "critical_frontend_cloudkms_cryptographic_requests"
]

_{Managed by the Sourcegraph Core application team.}

frontend: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

warning frontend: 0.05s+ mean blocked seconds per conn request for 10m0s
critical frontend: 0.1s+ mean blocked seconds per conn request for 15m0s

Possible solutions

Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Scale up Postgres memory / cpus See our scaling guide
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_mean_blocked_seconds_per_conn_request",
  "critical_frontend_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Devops team.}

frontend: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning frontend: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_container_cpu_usage"
]

_{Managed by the Sourcegraph Devops team.}

frontend: container_memory_usage

container memory usage by instance

Descriptions

warning frontend: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_container_memory_usage"
]

_{Managed by the Sourcegraph Devops team.}

frontend: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning frontend: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the (frontend|sourcegraph-frontend) service.
Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

frontend: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning frontend: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the (frontend|sourcegraph-frontend) service.
Docker Compose: Consider increasing memory: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

frontend: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning frontend: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

frontend: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning frontend: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

frontend: go_goroutines

maximum active goroutines

Descriptions

warning frontend: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_go_goroutines"
]

_{Managed by the Sourcegraph Devops team.}

frontend: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning frontend: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Devops team.}

frontend: pods_available_percentage

percentage pods available

Descriptions

critical frontend: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_pods_available_percentage"
]

_{Managed by the Sourcegraph Devops team.}

frontend: mean_successful_sentinel_duration_over_2h

mean successful sentinel search duration over 2h

Descriptions

warning frontend: 5s+ mean successful sentinel search duration over 2h for 15m0s
critical frontend: 8s+ mean successful sentinel search duration over 2h for 30m0s

Possible solutions

Look at the breakdown by query to determine if a specific query type is being affected
Check for high CPU usage on zoekt-webserver
Check Honeycomb for unusual activity
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_mean_successful_sentinel_duration_over_2h",
  "critical_frontend_mean_successful_sentinel_duration_over_2h"
]

_{Managed by the Sourcegraph Search team.}

frontend: mean_sentinel_stream_latency_over_2h

mean successful sentinel stream latency over 2h

Descriptions

warning frontend: 2s+ mean successful sentinel stream latency over 2h for 15m0s
critical frontend: 3s+ mean successful sentinel stream latency over 2h for 30m0s

Possible solutions

Look at the breakdown by query to determine if a specific query type is being affected
Check for high CPU usage on zoekt-webserver
Check Honeycomb for unusual activity
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_mean_sentinel_stream_latency_over_2h",
  "critical_frontend_mean_sentinel_stream_latency_over_2h"
]

_{Managed by the Sourcegraph Search team.}

frontend: 90th_percentile_successful_sentinel_duration_over_2h

90th percentile successful sentinel search duration over 2h

Descriptions

warning frontend: 5s+ 90th percentile successful sentinel search duration over 2h for 15m0s
critical frontend: 10s+ 90th percentile successful sentinel search duration over 2h for 3h30m0s

Possible solutions

Look at the breakdown by query to determine if a specific query type is being affected
Check for high CPU usage on zoekt-webserver
Check Honeycomb for unusual activity
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_successful_sentinel_duration_over_2h",
  "critical_frontend_90th_percentile_successful_sentinel_duration_over_2h"
]

_{Managed by the Sourcegraph Search team.}

frontend: 90th_percentile_sentinel_stream_latency_over_2h

90th percentile successful sentinel stream latency over 2h

Descriptions

warning frontend: 4s+ 90th percentile successful sentinel stream latency over 2h for 15m0s
critical frontend: 6s+ 90th percentile successful sentinel stream latency over 2h for 3h30m0s

Possible solutions

Look at the breakdown by query to determine if a specific query type is being affected
Check for high CPU usage on zoekt-webserver
Check Honeycomb for unusual activity
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_sentinel_stream_latency_over_2h",
  "critical_frontend_90th_percentile_sentinel_stream_latency_over_2h"
]

_{Managed by the Sourcegraph Search team.}

gitserver: disk_space_remaining

disk space remaining by instance

Descriptions

warning gitserver: less than 25% disk space remaining by instance
critical gitserver: less than 15% disk space remaining by instance

Possible solutions

Provision more disk space: Sourcegraph will begin deleting least-used repository clones at 10% disk space remaining which may result in decreased performance, users having to wait for repositories to clone, etc.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_disk_space_remaining",
  "critical_gitserver_disk_space_remaining"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: running_git_commands

git commands running on each gitserver instance

Descriptions

warning gitserver: 50+ git commands running on each gitserver instance for 2m0s
critical gitserver: 100+ git commands running on each gitserver instance for 5m0s

Possible solutions

Check if the problem may be an intermittent and temporary peak using the "Container monitoring" section at the bottom of the Git Server dashboard.
Single container deployments: Consider upgrading to a Docker Compose deployment which offers better scalability and resource isolation.
Kubernetes and Docker Compose: Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the Sourcegraph resource estimator.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_running_git_commands",
  "critical_gitserver_running_git_commands"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: repository_clone_queue_size

repository clone queue size

Descriptions

warning gitserver: 25+ repository clone queue size

Possible solutions

If you just added several repositories, the warning may be expected.
Check which repositories need cloning, by visiting e.g. https://sourcegraph.example.com/site-admin/repositories?filter=not-cloned
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_repository_clone_queue_size"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: repository_existence_check_queue_size

repository existence check queue size

Descriptions

warning gitserver: 25+ repository existence check queue size

Possible solutions

Check the code host status indicator for errors: on the Sourcegraph app homepage, when signed in as an admin click the cloud icon in the top right corner of the page.
Check if the issue continues to happen after 30 minutes, it may be temporary.
Check the gitserver logs for more information.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_repository_existence_check_queue_size"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: frontend_internal_api_error_responses

frontend-internal API error responses every 5m by route

Descriptions

warning gitserver: 2%+ frontend-internal API error responses every 5m by route for 5m0s

Possible solutions

Single-container deployments: Check docker logs $CONTAINER_ID for logs starting with repo-updater that indicate requests to the frontend service are failing.
Kubernetes:
- Confirm that kubectl get pods shows the frontend pods are healthy.
- Check kubectl logs gitserver for logs indicate request failures to frontend or frontend-internal.
Docker Compose:
- Confirm that docker ps shows the frontend-internal container is healthy.
- Check docker logs gitserver for logs indicating request failures to frontend or frontend-internal.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_frontend_internal_api_error_responses"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

warning gitserver: 0.05s+ mean blocked seconds per conn request for 10m0s
critical gitserver: 0.1s+ mean blocked seconds per conn request for 15m0s

Possible solutions

Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Scale up Postgres memory / cpus See our scaling guide
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_mean_blocked_seconds_per_conn_request",
  "critical_gitserver_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Devops team.}

gitserver: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning gitserver: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_container_cpu_usage"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: container_memory_usage

container memory usage by instance

Descriptions

warning gitserver: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of gitserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_container_memory_usage"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning gitserver: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the gitserver service.
Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning gitserver: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: go_goroutines

maximum active goroutines

Descriptions

warning gitserver: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_go_goroutines"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning gitserver: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Core application team.}

gitserver: pods_available_percentage

percentage pods available

Descriptions

critical gitserver: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_gitserver_pods_available_percentage"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: github_proxy_waiting_requests

number of requests waiting on the global mutex

Descriptions

warning github-proxy: 100+ number of requests waiting on the global mutex for 5m0s

Possible solutions

  						- **Check github-proxy logs for network connection issues.
  						- **Check github status.

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_github_proxy_waiting_requests"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning github-proxy: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the github-proxy container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_container_cpu_usage"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: container_memory_usage

container memory usage by instance

Descriptions

warning github-proxy: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of github-proxy container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_container_memory_usage"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning github-proxy: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the github-proxy service.
Docker Compose: Consider increasing cpus: of the github-proxy container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning github-proxy: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the github-proxy service.
Docker Compose: Consider increasing memory: of the github-proxy container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning github-proxy: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the github-proxy container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning github-proxy: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of github-proxy container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: go_goroutines

maximum active goroutines

Descriptions

warning github-proxy: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_go_goroutines"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning github-proxy: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_github-proxy_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Core application team.}

github-proxy: pods_available_percentage

percentage pods available

Descriptions

critical github-proxy: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_github-proxy_pods_available_percentage"
]

_{Managed by the Sourcegraph Core application team.}

postgres: connections

active connections

Descriptions

warning postgres: less than 5 active connections for 5m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_connections"
]

_{Managed by the Sourcegraph Devops team.}

postgres: transaction_durations

maximum transaction durations

Descriptions

warning postgres: 0.3s+ maximum transaction durations for 5m0s
critical postgres: 0.5s+ maximum transaction durations for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_transaction_durations",
  "critical_postgres_transaction_durations"
]

_{Managed by the Sourcegraph Devops team.}

postgres: postgres_up

database availability

Descriptions

critical postgres: less than 0 database availability for 5m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_postgres_up"
]

_{Managed by the Sourcegraph Devops team.}

postgres: invalid_indexes

invalid indexes (unusable by the query planner)

Descriptions

critical postgres: 1+ invalid indexes (unusable by the query planner)

Possible solutions

Drop and re-create the invalid trigger - please contact Sourcegraph to supply the trigger definition.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_invalid_indexes"
]

_{Managed by the Sourcegraph Devops team.}

postgres: pg_exporter_err

errors scraping postgres exporter

Descriptions

warning postgres: 1+ errors scraping postgres exporter for 5m0s

Possible solutions

Ensure the Postgres exporter can access the Postgres database. Also, check the Postgres exporter logs for errors.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_pg_exporter_err"
]

_{Managed by the Sourcegraph Devops team.}

postgres: migration_in_progress

active schema migration

Descriptions

critical postgres: 1+ active schema migration for 5m0s

Possible solutions

The database migration has been in progress for 5 or more minutes - please contact Sourcegraph if this persists.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_migration_in_progress"
]

_{Managed by the Sourcegraph Devops team.}

postgres: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning postgres: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the (pgsql|codeintel-db|codeinsights-db) service.
Docker Compose: Consider increasing cpus: of the (pgsql|codeintel-db|codeinsights-db) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

postgres: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning postgres: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the (pgsql|codeintel-db|codeinsights-db) service.
Docker Compose: Consider increasing memory: of the (pgsql|codeintel-db|codeinsights-db) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

postgres: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning postgres: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the (pgsql|codeintel-db|codeinsights-db) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

postgres: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning postgres: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of (pgsql|codeintel-db|codeinsights-db) container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

postgres: pods_available_percentage

percentage pods available

Descriptions

critical postgres: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_pods_available_percentage"
]

_{Managed by the Sourcegraph Devops team.}

precise-code-intel-worker: codeintel_upload_queued_max_age

unprocessed upload record queue longest time in queue

Descriptions

critical precise-code-intel-worker: 18000s+ unprocessed upload record queue longest time in queue

Possible solutions

An alert here could be indicative of a few things: an upload surfacing a pathological performance characteristic, precise-code-intel-worker being underprovisioned for the required upload processing throughput, or a higher replica count being required for the volume of uploads.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_precise-code-intel-worker_codeintel_upload_queued_max_age"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: frontend_internal_api_error_responses

frontend-internal API error responses every 5m by route

Descriptions

warning precise-code-intel-worker: 2%+ frontend-internal API error responses every 5m by route for 5m0s

Possible solutions

Single-container deployments: Check docker logs $CONTAINER_ID for logs starting with repo-updater that indicate requests to the frontend service are failing.
Kubernetes:
- Confirm that kubectl get pods shows the frontend pods are healthy.
- Check kubectl logs precise-code-intel-worker for logs indicate request failures to frontend or frontend-internal.
Docker Compose:
- Confirm that docker ps shows the frontend-internal container is healthy.
- Check docker logs precise-code-intel-worker for logs indicating request failures to frontend or frontend-internal.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_frontend_internal_api_error_responses"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

warning precise-code-intel-worker: 0.05s+ mean blocked seconds per conn request for 10m0s
critical precise-code-intel-worker: 0.1s+ mean blocked seconds per conn request for 15m0s

Possible solutions

Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Scale up Postgres memory / cpus See our scaling guide
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_mean_blocked_seconds_per_conn_request",
  "critical_precise-code-intel-worker_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Devops team.}

precise-code-intel-worker: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning precise-code-intel-worker: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_cpu_usage"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: container_memory_usage

container memory usage by instance

Descriptions

warning precise-code-intel-worker: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_memory_usage"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning precise-code-intel-worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the precise-code-intel-worker service.
Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning precise-code-intel-worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the precise-code-intel-worker service.
Docker Compose: Consider increasing memory: of the precise-code-intel-worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning precise-code-intel-worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning precise-code-intel-worker: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: go_goroutines

maximum active goroutines

Descriptions

warning precise-code-intel-worker: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_go_goroutines"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning precise-code-intel-worker: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code-intel team.}

precise-code-intel-worker: pods_available_percentage

percentage pods available

Descriptions

critical precise-code-intel-worker: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_precise-code-intel-worker_pods_available_percentage"
]

_{Managed by the Sourcegraph Code-intel team.}

redis: redis-store_up

redis-store availability

Descriptions

critical redis: less than 1 redis-store availability for 10s

Possible solutions

Ensure redis-store is running
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_redis-store_up"
]

_{Managed by the Sourcegraph Devops team.}

redis: redis-cache_up

redis-cache availability

Descriptions

critical redis: less than 1 redis-cache availability for 10s

Possible solutions

Ensure redis-cache is running
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_redis-cache_up"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the redis-cache service.
Docker Compose: Consider increasing cpus: of the redis-cache container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the redis-cache service.
Docker Compose: Consider increasing memory: of the redis-cache container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the redis-cache container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning redis: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of redis-cache container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the redis-store service.
Docker Compose: Consider increasing cpus: of the redis-store container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the redis-store service.
Docker Compose: Consider increasing memory: of the redis-store container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the redis-store container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning redis: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of redis-store container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

redis: pods_available_percentage

percentage pods available

Descriptions

critical redis: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_pods_available_percentage"
]

_{Managed by the Sourcegraph Devops team.}

redis: pods_available_percentage

percentage pods available

Descriptions

critical redis: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_pods_available_percentage"
]

_{Managed by the Sourcegraph Devops team.}

worker: worker_job_codeintel-janitor_count

number of worker instances running the codeintel-janitor job

Descriptions

warning worker: less than 1 number of worker instances running the codeintel-janitor job for 1m0s
critical worker: less than 1 number of worker instances running the codeintel-janitor job for 5m0s

Possible solutions

Ensure your instance defines a worker container such that:
- WORKER_JOB_ALLOWLIST contains "codeintel-janitor" (or "all"), and
- WORKER_JOB_BLOCKLIST does not contain "codeintel-janitor"
Ensure that such a container is not failing to start or stay active
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-janitor_count",
  "critical_worker_worker_job_codeintel-janitor_count"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: worker_job_codeintel-commitgraph_count

number of worker instances running the codeintel-commitgraph job

Descriptions

warning worker: less than 1 number of worker instances running the codeintel-commitgraph job for 1m0s
critical worker: less than 1 number of worker instances running the codeintel-commitgraph job for 5m0s

Possible solutions

Ensure your instance defines a worker container such that:
- WORKER_JOB_ALLOWLIST contains "codeintel-commitgraph" (or "all"), and
- WORKER_JOB_BLOCKLIST does not contain "codeintel-commitgraph"
Ensure that such a container is not failing to start or stay active
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-commitgraph_count",
  "critical_worker_worker_job_codeintel-commitgraph_count"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: worker_job_codeintel-auto-indexing_count

number of worker instances running the codeintel-auto-indexing job

Descriptions

warning worker: less than 1 number of worker instances running the codeintel-auto-indexing job for 1m0s
critical worker: less than 1 number of worker instances running the codeintel-auto-indexing job for 5m0s

Possible solutions

Ensure your instance defines a worker container such that:
- WORKER_JOB_ALLOWLIST contains "codeintel-auto-indexing" (or "all"), and
- WORKER_JOB_BLOCKLIST does not contain "codeintel-auto-indexing"
Ensure that such a container is not failing to start or stay active
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-auto-indexing_count",
  "critical_worker_worker_job_codeintel-auto-indexing_count"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: codeintel_commit_graph_queued_max_age

repository queue longest time in queue

Descriptions

critical worker: 3600s+ repository queue longest time in queue

Possible solutions

An alert here is generally indicative of either underprovisioned worker instance(s) and/or an underprovisioned main postgres instance.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_worker_codeintel_commit_graph_queued_max_age"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: insights_queue_unutilized_size

insights queue size that is not utilized (not processing)

Descriptions

warning worker: 0+ insights queue size that is not utilized (not processing) for 30m0s

Possible solutions

Verify code insights worker job has successfully started. Restart worker service and monitoring startup logs, looking for worker panics.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_insights_queue_unutilized_size"
]

_{Managed by the Sourcegraph Code-insights team.}

worker: frontend_internal_api_error_responses

frontend-internal API error responses every 5m by route

Descriptions

warning worker: 2%+ frontend-internal API error responses every 5m by route for 5m0s

Possible solutions

Single-container deployments: Check docker logs $CONTAINER_ID for logs starting with repo-updater that indicate requests to the frontend service are failing.
Kubernetes:
- Confirm that kubectl get pods shows the frontend pods are healthy.
- Check kubectl logs worker for logs indicate request failures to frontend or frontend-internal.
Docker Compose:
- Confirm that docker ps shows the frontend-internal container is healthy.
- Check docker logs worker for logs indicating request failures to frontend or frontend-internal.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_frontend_internal_api_error_responses"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

warning worker: 0.05s+ mean blocked seconds per conn request for 10m0s
critical worker: 0.1s+ mean blocked seconds per conn request for 15m0s

Possible solutions

Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Scale up Postgres memory / cpus See our scaling guide
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_mean_blocked_seconds_per_conn_request",
  "critical_worker_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Devops team.}

worker: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning worker: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_container_cpu_usage"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: container_memory_usage

container memory usage by instance

Descriptions

warning worker: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_container_memory_usage"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the worker service.
Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the worker service.
Docker Compose: Consider increasing memory: of the worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning worker: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: go_goroutines

maximum active goroutines

Descriptions

warning worker: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_go_goroutines"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning worker: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code-intel team.}

worker: pods_available_percentage

percentage pods available

Descriptions

critical worker: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_worker_pods_available_percentage"
]

_{Managed by the Sourcegraph Code-intel team.}

repo-updater: src_repoupdater_max_sync_backoff

time since oldest sync

Descriptions

critical repo-updater: 32400s+ time since oldest sync for 10m0s

Possible solutions

An alert here indicates that no code host connections have synced in at least 9h0m0s. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts.
Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
Check the repo-updater logs for errors about syncing.
Confirm that outbound network connections are allowed where repo-updater is deployed.
Check back in an hour to see if the issue has resolved itself.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_src_repoupdater_max_sync_backoff"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: src_repoupdater_syncer_sync_errors_total

site level external service sync error rate

Descriptions

warning repo-updater: 0.5+ site level external service sync error rate for 10m0s
critical repo-updater: 1+ site level external service sync error rate for 10m0s

Possible solutions

An alert here indicates errors syncing site level repo metadata with code hosts. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts.
Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
Check the repo-updater logs for errors about syncing.
Confirm that outbound network connections are allowed where repo-updater is deployed.
Check back in an hour to see if the issue has resolved itself.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_syncer_sync_errors_total",
  "critical_repo-updater_src_repoupdater_syncer_sync_errors_total"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: syncer_sync_start

repo metadata sync was started

Descriptions

warning repo-updater: less than 0 repo metadata sync was started for 9h0m0s

Possible solutions

Check repo-updater logs for errors.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_syncer_sync_start"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: syncer_sync_duration

95th repositories sync duration

Descriptions

warning repo-updater: 30s+ 95th repositories sync duration for 5m0s

Possible solutions

Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_syncer_sync_duration"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: source_duration

95th repositories source duration

Descriptions

warning repo-updater: 30s+ 95th repositories source duration for 5m0s

Possible solutions

Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_source_duration"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: syncer_synced_repos

repositories synced

Descriptions

warning repo-updater: less than 0 repositories synced for 9h0m0s

Possible solutions

Check network connectivity to code hosts
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_syncer_synced_repos"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: sourced_repos

repositories sourced

Descriptions

warning repo-updater: less than 0 repositories sourced for 9h0m0s

Possible solutions

Check network connectivity to code hosts
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sourced_repos"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: user_added_repos

total number of user added repos

Descriptions

critical repo-updater: 720000+ total number of user added repos for 5m0s

Possible solutions

Check for unusual spikes in user added repos. Each user is only allowed to add 2000 and we have a site wide limit of 800k.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_user_added_repos"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: purge_failed

repositories purge failed

Descriptions

warning repo-updater: 0+ repositories purge failed for 5m0s

Possible solutions

Check repo-updater`s connectivity with gitserver and gitserver logs
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_purge_failed"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: sched_auto_fetch

repositories scheduled due to hitting a deadline

Descriptions

warning repo-updater: less than 0 repositories scheduled due to hitting a deadline for 9h0m0s

Possible solutions

Check repo-updater logs. This is expected to fire if there are no user added code hosts
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sched_auto_fetch"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: sched_known_repos

repositories managed by the scheduler

Descriptions

warning repo-updater: less than 0 repositories managed by the scheduler for 10m0s

Possible solutions

Check repo-updater logs. This is expected to fire if there are no user added code hosts
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sched_known_repos"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: sched_update_queue_length

rate of growth of update queue length over 5 minutes

Descriptions

critical repo-updater: 0+ rate of growth of update queue length over 5 minutes for 2h0m0s

Possible solutions

Check repo-updater logs for indications that the queue is not being processed. The queue length should trend downwards over time as items are sent to GitServer
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_sched_update_queue_length"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: sched_loops

scheduler loops

Descriptions

warning repo-updater: less than 0 scheduler loops for 9h0m0s

Possible solutions

Check repo-updater logs for errors. This is expected to fire if there are no user added code hosts
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sched_loops"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: src_repoupdater_stale_repos

repos that haven't been fetched in more than 8 hours

Descriptions

warning repo-updater: 1+ repos that haven't been fetched in more than 8 hours for 25m0s

Possible solutions

  					Check repo-updater logs for errors.
  					Check for rows in gitserver_repos where LastError is not an empty string.

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_stale_repos"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: sched_error

repositories schedule error rate

Descriptions

critical repo-updater: 1+ repositories schedule error rate for 25m0s

Possible solutions

Check repo-updater logs for errors
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_sched_error"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_perms

time gap between least and most up to date permissions

Descriptions

warning repo-updater: 259200s+ time gap between least and most up to date permissions for 5m0s

Possible solutions

Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_perms_syncer_perms"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_stale_perms

number of entities with stale permissions

Descriptions

warning repo-updater: 100+ number of entities with stale permissions for 5m0s

Possible solutions

Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_perms_syncer_stale_perms"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_no_perms

number of entities with no permissions

Descriptions

warning repo-updater: 100+ number of entities with no permissions for 5m0s

Possible solutions

Enabled permissions for the first time: Wait for few minutes and see if the number goes down.
Otherwise: Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_perms_syncer_no_perms"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_outdated_perms

number of entities with outdated permissions

Descriptions

warning repo-updater: 100+ number of entities with outdated permissions for 5m0s

Possible solutions

Enabled permissions for the first time: Wait for few minutes and see if the number goes down.
Otherwise: Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_perms_syncer_outdated_perms"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_sync_duration

95th permissions sync duration

Descriptions

warning repo-updater: 30s+ 95th permissions sync duration for 5m0s

Possible solutions

Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_perms_syncer_sync_duration"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_queue_size

permissions sync queued items

Descriptions

warning repo-updater: 100+ permissions sync queued items for 5m0s

Possible solutions

Enabled permissions for the first time: Wait for few minutes and see if the number goes down.
Otherwise: Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_perms_syncer_queue_size"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: perms_syncer_sync_errors

permissions sync error rate

Descriptions

critical repo-updater: 1+ permissions sync error rate for 1m0s

Possible solutions

Check the network connectivity the Sourcegraph and the code host.
Check if API rate limit quota is exhausted on the code host.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_perms_syncer_sync_errors"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: src_repoupdater_external_services_total

the total number of external services

Descriptions

critical repo-updater: 20000+ the total number of external services for 1h0m0s

Possible solutions

Check for spikes in external services, could be abuse
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_src_repoupdater_external_services_total"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: src_repoupdater_user_external_services_total

the total number of user added external services

Descriptions

warning repo-updater: 20000+ the total number of user added external services for 1h0m0s

Possible solutions

Check for spikes in external services, could be abuse
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_user_external_services_total"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: repoupdater_queued_sync_jobs_total

the total number of queued sync jobs

Descriptions

warning repo-updater: 100+ the total number of queued sync jobs for 1h0m0s

Possible solutions

Check if jobs are failing to sync: "SELECT * FROM external_service_sync_jobs WHERE state = errored";
Increase the number of workers using the repoConcurrentExternalServiceSyncers site config.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_queued_sync_jobs_total"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: repoupdater_completed_sync_jobs_total

the total number of completed sync jobs

Descriptions

warning repo-updater: 100000+ the total number of completed sync jobs for 1h0m0s

Possible solutions

Check repo-updater logs. Jobs older than 1 day should have been removed.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_completed_sync_jobs_total"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: repoupdater_errored_sync_jobs_percentage

the percentage of external services that have failed their most recent sync

Descriptions

warning repo-updater: 10%+ the percentage of external services that have failed their most recent sync for 1h0m0s

Possible solutions

Check repo-updater logs. Check code host connectivity
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_errored_sync_jobs_percentage"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: github_graphql_rate_limit_remaining

remaining calls to GitHub graphql API before hitting the rate limit

Descriptions

critical repo-updater: less than 250 remaining calls to GitHub graphql API before hitting the rate limit

Possible solutions

Try restarting the pod to get a different public IP.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_github_graphql_rate_limit_remaining"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: github_rest_rate_limit_remaining

remaining calls to GitHub rest API before hitting the rate limit

Descriptions

critical repo-updater: less than 250 remaining calls to GitHub rest API before hitting the rate limit

Possible solutions

Try restarting the pod to get a different public IP.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_github_rest_rate_limit_remaining"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: github_search_rate_limit_remaining

remaining calls to GitHub search API before hitting the rate limit

Descriptions

critical repo-updater: less than 5 remaining calls to GitHub search API before hitting the rate limit

Possible solutions

Try restarting the pod to get a different public IP.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_github_search_rate_limit_remaining"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: gitlab_rest_rate_limit_remaining

remaining calls to GitLab rest API before hitting the rate limit

Descriptions

critical repo-updater: less than 30 remaining calls to GitLab rest API before hitting the rate limit

Possible solutions

Try restarting the pod to get a different public IP.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_gitlab_rest_rate_limit_remaining"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: frontend_internal_api_error_responses

frontend-internal API error responses every 5m by route

Descriptions

warning repo-updater: 2%+ frontend-internal API error responses every 5m by route for 5m0s

Possible solutions

Single-container deployments: Check docker logs $CONTAINER_ID for logs starting with repo-updater that indicate requests to the frontend service are failing.
Kubernetes:
- Confirm that kubectl get pods shows the frontend pods are healthy.
- Check kubectl logs repo-updater for logs indicate request failures to frontend or frontend-internal.
Docker Compose:
- Confirm that docker ps shows the frontend-internal container is healthy.
- Check docker logs repo-updater for logs indicating request failures to frontend or frontend-internal.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_frontend_internal_api_error_responses"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

warning repo-updater: 0.05s+ mean blocked seconds per conn request for 10m0s
critical repo-updater: 0.1s+ mean blocked seconds per conn request for 15m0s

Possible solutions

Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Scale up Postgres memory / cpus See our scaling guide
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_mean_blocked_seconds_per_conn_request",
  "critical_repo-updater_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Devops team.}

repo-updater: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning repo-updater: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_container_cpu_usage"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: container_memory_usage

container memory usage by instance

Descriptions

critical repo-updater: 90%+ container memory usage by instance for 10m0s

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_container_memory_usage"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning repo-updater: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the repo-updater service.
Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning repo-updater: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the repo-updater service.
Docker Compose: Consider increasing memory: of the repo-updater container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning repo-updater: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning repo-updater: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: go_goroutines

maximum active goroutines

Descriptions

warning repo-updater: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_go_goroutines"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning repo-updater: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Core application team.}

repo-updater: pods_available_percentage

percentage pods available

Descriptions

critical repo-updater: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_pods_available_percentage"
]

_{Managed by the Sourcegraph Core application team.}

searcher: unindexed_search_request_errors

unindexed search request errors every 5m by code

Descriptions

warning searcher: 5%+ unindexed search request errors every 5m by code for 5m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_unindexed_search_request_errors"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: replica_traffic

requests per second over 10m

Descriptions

warning searcher: 5+ requests per second over 10m

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_replica_traffic"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

warning searcher: 0.05s+ mean blocked seconds per conn request for 10m0s
critical searcher: 0.1s+ mean blocked seconds per conn request for 15m0s

Possible solutions

Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Scale up Postgres memory / cpus See our scaling guide
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_mean_blocked_seconds_per_conn_request",
  "critical_searcher_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Devops team.}

searcher: frontend_internal_api_error_responses

frontend-internal API error responses every 5m by route

Descriptions

warning searcher: 2%+ frontend-internal API error responses every 5m by route for 5m0s

Possible solutions

Single-container deployments: Check docker logs $CONTAINER_ID for logs starting with repo-updater that indicate requests to the frontend service are failing.
Kubernetes:
- Confirm that kubectl get pods shows the frontend pods are healthy.
- Check kubectl logs searcher for logs indicate request failures to frontend or frontend-internal.
Docker Compose:
- Confirm that docker ps shows the frontend-internal container is healthy.
- Check docker logs searcher for logs indicating request failures to frontend or frontend-internal.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_frontend_internal_api_error_responses"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning searcher: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the searcher container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_container_cpu_usage"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: container_memory_usage

container memory usage by instance

Descriptions

warning searcher: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of searcher container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_container_memory_usage"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning searcher: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the searcher service.
Docker Compose: Consider increasing cpus: of the searcher container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning searcher: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the searcher service.
Docker Compose: Consider increasing memory: of the searcher container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning searcher: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the searcher container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning searcher: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of searcher container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: go_goroutines

maximum active goroutines

Descriptions

warning searcher: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_go_goroutines"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning searcher: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Search-core team.}

searcher: pods_available_percentage

percentage pods available

Descriptions

critical searcher: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_searcher_pods_available_percentage"
]

_{Managed by the Sourcegraph Search-core team.}

symbols: frontend_internal_api_error_responses

frontend-internal API error responses every 5m by route

Descriptions

warning symbols: 2%+ frontend-internal API error responses every 5m by route for 5m0s

Possible solutions

Single-container deployments: Check docker logs $CONTAINER_ID for logs starting with repo-updater that indicate requests to the frontend service are failing.
Kubernetes:
- Confirm that kubectl get pods shows the frontend pods are healthy.
- Check kubectl logs symbols for logs indicate request failures to frontend or frontend-internal.
Docker Compose:
- Confirm that docker ps shows the frontend-internal container is healthy.
- Check docker logs symbols for logs indicating request failures to frontend or frontend-internal.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_frontend_internal_api_error_responses"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning symbols: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the symbols container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_container_cpu_usage"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: container_memory_usage

container memory usage by instance

Descriptions

warning symbols: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of symbols container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_container_memory_usage"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning symbols: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the symbols service.
Docker Compose: Consider increasing cpus: of the symbols container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning symbols: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the symbols service.
Docker Compose: Consider increasing memory: of the symbols container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning symbols: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the symbols container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning symbols: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of symbols container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: go_goroutines

maximum active goroutines

Descriptions

warning symbols: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_go_goroutines"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning symbols: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code-intel team.}

symbols: pods_available_percentage

percentage pods available

Descriptions

critical symbols: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_symbols_pods_available_percentage"
]

_{Managed by the Sourcegraph Code-intel team.}

syntect-server: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning syntect-server: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the syntect-server container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_container_cpu_usage"
]

_{Managed by the Sourcegraph Core application team.}

syntect-server: container_memory_usage

container memory usage by instance

Descriptions

warning syntect-server: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of syntect-server container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_container_memory_usage"
]

_{Managed by the Sourcegraph Core application team.}

syntect-server: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning syntect-server: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the syntect-server service.
Docker Compose: Consider increasing cpus: of the syntect-server container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

syntect-server: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning syntect-server: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the syntect-server service.
Docker Compose: Consider increasing memory: of the syntect-server container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Core application team.}

syntect-server: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning syntect-server: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the syntect-server container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

syntect-server: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning syntect-server: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of syntect-server container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Core application team.}

syntect-server: pods_available_percentage

percentage pods available

Descriptions

critical syntect-server: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_syntect-server_pods_available_percentage"
]

_{Managed by the Sourcegraph Core application team.}

zoekt: average_resolve_revision_duration

average resolve revision duration over 5m

Descriptions

warning zoekt: 15s+ average resolve revision duration over 5m
critical zoekt: 30s+ average resolve revision duration over 5m

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_average_resolve_revision_duration",
  "critical_zoekt_average_resolve_revision_duration"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: get_index_options_error_increase

the number of repositories we failed to get indexing options over 5m

Descriptions

warning zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 5m0s
critical zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 35m0s

Possible solutions

View error rates on gitserver and frontend to identify root cause.
Rollback frontend/gitserver deployment if due to a bad code change.
View error logs for getIndexOptions via net/trace debug interface. For example click on a indexed-search-indexer- on https://sourcegraph.com/-/debug/. Then click on Traces. Replace sourcegraph.com with your instance address.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_get_index_options_error_increase",
  "critical_zoekt_get_index_options_error_increase"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: indexed_search_request_errors

indexed search request errors every 5m by code

Descriptions

warning zoekt: 5%+ indexed search request errors every 5m by code for 5m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_indexed_search_request_errors"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning zoekt: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the zoekt-indexserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_cpu_usage"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: container_memory_usage

container memory usage by instance

Descriptions

warning zoekt: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of zoekt-indexserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_memory_usage"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning zoekt: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the zoekt-webserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_cpu_usage"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: container_memory_usage

container memory usage by instance

Descriptions

warning zoekt: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of zoekt-webserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_memory_usage"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning zoekt: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the zoekt-indexserver service.
Docker Compose: Consider increasing cpus: of the zoekt-indexserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning zoekt: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the zoekt-indexserver service.
Docker Compose: Consider increasing memory: of the zoekt-indexserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning zoekt: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the zoekt-indexserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning zoekt: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of zoekt-indexserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning zoekt: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the zoekt-webserver service.
Docker Compose: Consider increasing cpus: of the zoekt-webserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning zoekt: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the zoekt-webserver service.
Docker Compose: Consider increasing memory: of the zoekt-webserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning zoekt: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the zoekt-webserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning zoekt: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of zoekt-webserver container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Search-core team.}

zoekt: pods_available_percentage

percentage pods available

Descriptions

critical zoekt: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_zoekt_pods_available_percentage"
]

_{Managed by the Sourcegraph Search-core team.}

prometheus: prometheus_rule_eval_duration

average prometheus rule group evaluation duration over 10m by rule group

Descriptions

warning prometheus: 30s+ average prometheus rule group evaluation duration over 10m by rule group

Possible solutions

Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary.
If the rule group taking a long time to evaluate belongs to /sg_prometheus_addons, try reducing the complexity of any custom Prometheus rules provided.
If the rule group taking a long time to evaluate belongs to /sg_config_prometheus, please open an issue.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_rule_eval_duration"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: prometheus_rule_eval_failures

failed prometheus rule evaluations over 5m by rule group

Descriptions

warning prometheus: 0+ failed prometheus rule evaluations over 5m by rule group

Possible solutions

Check Prometheus logs for messages related to rule group evaluation (generally with log field component="rule manager").
If the rule group failing to evaluate belongs to /sg_prometheus_addons, ensure any custom Prometheus configuration provided is valid.
If the rule group taking a long time to evaluate belongs to /sg_config_prometheus, please open an issue.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_rule_eval_failures"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: alertmanager_notification_latency

alertmanager notification latency over 1m by integration

Descriptions

warning prometheus: 1s+ alertmanager notification latency over 1m by integration

Possible solutions

Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary.
Ensure that your observability.alerts configuration (in site configuration) is valid.
Check if the relevant alert integration service is experiencing downtime or issues.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_alertmanager_notification_latency"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: alertmanager_notification_failures

failed alertmanager notifications over 1m by integration

Descriptions

warning prometheus: 0+ failed alertmanager notifications over 1m by integration

Possible solutions

Ensure that your observability.alerts configuration (in site configuration) is valid.
Check if the relevant alert integration service is experiencing downtime or issues.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_alertmanager_notification_failures"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: prometheus_config_status

prometheus configuration reload status

Descriptions

warning prometheus: less than 1 prometheus configuration reload status

Possible solutions

Check Prometheus logs for messages related to configuration loading.
Ensure any custom configuration you have provided Prometheus is valid.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_config_status"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: alertmanager_config_status

alertmanager configuration reload status

Descriptions

warning prometheus: less than 1 alertmanager configuration reload status

Possible solutions

Ensure that your observability.alerts configuration (in site configuration) is valid.
More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_alertmanager_config_status"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: prometheus_tsdb_op_failure

prometheus tsdb failures by operation over 1m by operation

Descriptions

warning prometheus: 0+ prometheus tsdb failures by operation over 1m by operation

Possible solutions

Check Prometheus logs for messages related to the failing operation.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_tsdb_op_failure"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: prometheus_target_sample_exceeded

prometheus scrapes that exceed the sample limit over 10m

Descriptions

warning prometheus: 0+ prometheus scrapes that exceed the sample limit over 10m

Possible solutions

Check Prometheus logs for messages related to target scrape failures.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_target_sample_exceeded"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: prometheus_target_sample_duplicate

prometheus scrapes rejected due to duplicate timestamps over 10m

Descriptions

warning prometheus: 0+ prometheus scrapes rejected due to duplicate timestamps over 10m

Possible solutions

Check Prometheus logs for messages related to target scrape failures.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_target_sample_duplicate"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

warning prometheus: 99%+ container cpu usage total (1m average) across all cores by instance

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the prometheus container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_container_cpu_usage"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: container_memory_usage

container memory usage by instance

Descriptions

warning prometheus: 99%+ container memory usage by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of prometheus container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_container_memory_usage"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

warning prometheus: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the prometheus service.
Docker Compose: Consider increasing cpus: of the prometheus container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

warning prometheus: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Possible solutions

Kubernetes: Consider increasing memory limits in the Deployment.yaml for the prometheus service.
Docker Compose: Consider increasing memory: of the prometheus container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

warning prometheus: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Possible solutions

Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
Docker Compose: Consider increasing cpus: of the prometheus container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

warning prometheus: 90%+ container memory usage (5m maximum) by instance

Possible solutions

Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
Docker Compose: Consider increasing memory: of prometheus container in docker-compose.yml.
Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Devops team.}

prometheus: pods_available_percentage

percentage pods available

Descriptions

critical prometheus: less than 90% percentage pods available for 10m0s

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_prometheus_pods_available_percentage"
]

_{Managed by the Sourcegraph Devops team.}

executor: go_goroutines

maximum active goroutines

Descriptions

warning executor: 10000+ maximum active goroutines for 10m0s

Possible solutions

More help interpreting this metric is available in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_executor_go_goroutines"
]

_{Managed by the Sourcegraph Code-intel team.}

executor: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

warning executor: 2s+ maximum go garbage collection duration

Possible solutions

Learn more about the related dashboard panel in the dashboards reference.
Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_executor_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code-intel team.}