Alerts reference

This document contains a complete reference of all alerts in Sourcegraph's monitoring, and next steps for when you find alerts that are firing. If your alert isn't mentioned here, or if the next steps don't help, contact us for assistance.

To learn more about Sourcegraph's alerting and how to set up alerts, see our alerting guide.

frontend: 99th_percentile_search_request_duration

99th percentile successful search request duration over 5m

Descriptions

• warning frontend: 20s+ 99th percentile successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_request_duration"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.99, sum by (le) (rate(src_search_streaming_latency_seconds_bucket{source="browser"}[5m])))) >= 20)

frontend: 90th_percentile_search_request_duration

90th percentile successful search request duration over 5m

Descriptions

• warning frontend: 15s+ 90th percentile successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_request_duration"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.9, sum by (le) (rate(src_search_streaming_latency_seconds_bucket{source="browser"}[5m])))) >= 15)

frontend: hard_timeout_search_responses

hard timeout search responses every 5m

Descriptions

• warning frontend: 2%+ hard timeout search responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_timeout_search_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max(((sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser",status="timeout"}[5m])) + sum(increase(src_graphql_search_response{alert_type="timed_out",request_name!="CodeIntelSearch",source="browser",status="alert"}[5m]))) / sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)

frontend: hard_error_search_responses

hard error search responses every 5m

Descriptions

• warning frontend: 2%+ hard error search responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (status) (increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser",status=~"error"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)

frontend: partial_timeout_search_responses

partial timeout search responses every 5m

Descriptions

• warning frontend: 5%+ partial timeout search responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (status) (increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser",status="partial_timeout"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 5)

frontend: search_alert_user_suggestions

search alert user suggestions shown every 5m

Descriptions

• warning frontend: 5%+ search alert user suggestions shown every 5m for 15m0s

Next steps

• This indicates your user`s are making syntax errors or similar user errors.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_search_alert_user_suggestions"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (alert_type) (increase(src_graphql_search_response{alert_type!~"timed_out|no_results__suggest_quotes",request_name!="CodeIntelSearch",source="browser",status="alert"}[5m])) / ignoring (alert_type) group_left () sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 5)

frontend: page_load_latency

90th percentile page load latency over all routes over 10m

Descriptions

• warning frontend: 2s+ 90th percentile page load latency over all routes over 10m

Next steps

• Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels.
• Investigate potential sources of latency by selecting Explore and modifying the sum by(le) section to include additional labels: for example, sum by(le, job) or sum by (le, instance).
• Trace a request to see what the slowest part is: https://sourcegraph.com/docs/admin/observability/tracing
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_page_load_latency"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.9, sum by (le) (rate(src_http_request_duration_seconds_bucket{route!="blob",route!="raw",route!~"graphql.*"}[10m])))) >= 2)

frontend: 99th_percentile_search_codeintel_request_duration

99th percentile code-intel successful search request duration over 5m

Descriptions

• warning frontend: 20s+ 99th percentile code-intel successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• This alert may indicate that your instance is struggling to process symbols queries on a monorepo, learn more here.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_codeintel_request_duration"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.99, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",request_name="CodeIntelSearch",source="browser",type="Search"}[5m])))) >= 20)

frontend: 90th_percentile_search_codeintel_request_duration

90th percentile code-intel successful search request duration over 5m

Descriptions

• warning frontend: 15s+ 90th percentile code-intel successful search request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• This alert may indicate that your instance is struggling to process symbols queries on a monorepo, learn more here.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_codeintel_request_duration"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.9, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",request_name="CodeIntelSearch",source="browser",type="Search"}[5m])))) >= 15)

frontend: hard_timeout_search_codeintel_responses

hard timeout search code-intel responses every 5m

Descriptions

• warning frontend: 2%+ hard timeout search code-intel responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_timeout_search_codeintel_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max(((sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status="timeout"}[5m])) + sum(increase(src_graphql_search_response{alert_type="timed_out",request_name="CodeIntelSearch",source="browser",status="alert"}[5m]))) / sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)

frontend: hard_error_search_codeintel_responses

hard error search code-intel responses every 5m

Descriptions

• warning frontend: 2%+ hard error search code-intel responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_codeintel_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (status) (increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status=~"error"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)

frontend: partial_timeout_search_codeintel_responses

partial timeout search code-intel responses every 5m

Descriptions

• warning frontend: 5%+ partial timeout search code-intel responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_codeintel_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (status) (increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status="partial_timeout"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status="partial_timeout"}[5m])) * 100) >= 5)

frontend: search_codeintel_alert_user_suggestions

search code-intel alert user suggestions shown every 5m

Descriptions

• warning frontend: 5%+ search code-intel alert user suggestions shown every 5m for 15m0s

Next steps

• This indicates a bug in Sourcegraph, please open an issue.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_search_codeintel_alert_user_suggestions"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (alert_type) (increase(src_graphql_search_response{alert_type!~"timed_out",request_name="CodeIntelSearch",source="browser",status="alert"}[5m])) / ignoring (alert_type) group_left () sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser"}[5m])) * 100) >= 5)

frontend: 99th_percentile_search_api_request_duration

99th percentile successful search API request duration over 5m

Descriptions

• warning frontend: 50s+ 99th percentile successful search API request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 20, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_search_api_request_duration"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.99, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",source="other",type="Search"}[5m])))) >= 50)

frontend: 90th_percentile_search_api_request_duration

90th percentile successful search API request duration over 5m

Descriptions

• warning frontend: 40s+ 90th percentile successful search API request duration over 5m

Next steps

• Get details on the exact queries that are slow by configuring "observability.logSlowSearches": 15, in the site configuration and looking for frontend warning logs prefixed with slow search request for additional details.
• Check that most repositories are indexed by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
• Kubernetes: Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the indexed-search.Deployment.yaml if regularly hitting max CPU utilization.
• Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_search_api_request_duration"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.9, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",source="other",type="Search"}[5m])))) >= 40)

frontend: hard_error_search_api_responses

hard error search API responses every 5m

Descriptions

• warning frontend: 2%+ hard error search API responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_hard_error_search_api_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (status) (increase(src_graphql_search_response{source="other",status=~"error"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{source="other"}[5m]))) >= 2)

frontend: partial_timeout_search_api_responses

partial timeout search API responses every 5m

Descriptions

• warning frontend: 5%+ partial timeout search API responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_partial_timeout_search_api_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum(increase(src_graphql_search_response{source="other",status="partial_timeout"}[5m])) / sum(increase(src_graphql_search_response{source="other"}[5m]))) >= 5)

frontend: search_api_alert_user_suggestions

search API alert user suggestions shown every 5m

Descriptions

• warning frontend: 5%+ search API alert user suggestions shown every 5m

Next steps

• This indicates your user`s search API requests have syntax errors or a similar user error. Check the responses the API sends back for an explanation.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_search_api_alert_user_suggestions"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (alert_type) (increase(src_graphql_search_response{alert_type!~"timed_out|no_results__suggest_quotes",source="other",status="alert"}[5m])) / ignoring (alert_type) group_left () sum(increase(src_graphql_search_response{source="other",status="alert"}[5m]))) >= 5)

frontend: frontend_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "frontend" instances)

Descriptions

• critical frontend: 300s+ maximum duration since last successful site configuration update (all "frontend" instances)

Next steps

• This indicates that one or more "frontend" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "frontend" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_frontend_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~"(sourcegraph-)?frontend"}[1m]))) >= 300)

frontend: internal_indexed_search_error_responses

internal indexed search error responses every 5m

Descriptions

• warning frontend: 5%+ internal indexed search error responses every 5m for 15m0s

Next steps

• Check the Zoekt Web Server dashboard for indications it might be unhealthy.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_internal_indexed_search_error_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (code) (increase(src_zoekt_request_duration_seconds_count{code!~"2.."}[5m])) / ignoring (code) group_left () sum(increase(src_zoekt_request_duration_seconds_count[5m])) * 100) >= 5)

frontend: internal_unindexed_search_error_responses

internal unindexed search error responses every 5m

Descriptions

• warning frontend: 5%+ internal unindexed search error responses every 5m for 15m0s

Next steps

• Check the Searcher dashboard for indications it might be unhealthy.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_internal_unindexed_search_error_responses"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum by (code) (increase(searcher_service_request_total{code!~"2.."}[5m])) / ignoring (code) group_left () sum(increase(searcher_service_request_total[5m])) * 100) >= 5)

frontend: 99th_percentile_gitserver_duration

99th percentile successful gitserver query duration over 5m

Descriptions

• warning frontend: 20s+ 99th percentile successful gitserver query duration over 5m

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_99th_percentile_gitserver_duration"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.99, sum by (le, category) (rate(src_gitserver_request_duration_seconds_bucket{job=~"(sourcegraph-)?frontend"}[5m])))) >= 20)

frontend: gitserver_error_responses

gitserver error responses every 5m

Descriptions

• warning frontend: 5%+ gitserver error responses every 5m for 15m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_gitserver_error_responses"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum by (category) (increase(src_gitserver_request_duration_seconds_count{code!~"2..",job=~"(sourcegraph-)?frontend"}[5m])) / ignoring (code) group_left () sum by (category) (increase(src_gitserver_request_duration_seconds_count{job=~"(sourcegraph-)?frontend"}[5m])) * 100) >= 5)

frontend: observability_test_alert_warning

warning test alert metric

Descriptions

• warning frontend: 1+ warning test alert metric

Next steps

• This alert is triggered via the triggerObservabilityTestAlert GraphQL endpoint, and will automatically resolve itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_observability_test_alert_warning"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (owner) (observability_test_metric_warning)) >= 1)

frontend: observability_test_alert_critical

critical test alert metric

Descriptions

• critical frontend: 1+ critical test alert metric

Next steps

• This alert is triggered via the triggerObservabilityTestAlert GraphQL endpoint, and will automatically resolve itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_observability_test_alert_critical"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max by (owner) (observability_test_metric_critical)) >= 1)

frontend: cloudkms_cryptographic_requests

cryptographic requests to Cloud KMS every 1m

Descriptions

• warning frontend: 15000+ cryptographic requests to Cloud KMS every 1m for 5m0s
• critical frontend: 30000+ cryptographic requests to Cloud KMS every 1m for 5m0s

Next steps

• Revert recent commits that cause extensive listing from "external_services" and/or "user_external_accounts" tables.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_cloudkms_cryptographic_requests",
  "critical_frontend_cloudkms_cryptographic_requests"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum(increase(src_cloudkms_cryptographic_total[1m]))) >= 15000)

Generated query for critical alert: max((sum(increase(src_cloudkms_cryptographic_total[1m]))) >= 30000)

frontend: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning frontend: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical frontend: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_mean_blocked_seconds_per_conn_request",
  "critical_frontend_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="frontend"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="frontend"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="frontend"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="frontend"}[5m]))) >= 0.5)

frontend: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning frontend: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_container_cpu_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}) >= 99)

frontend: container_memory_usage

container memory usage by instance

Descriptions

• warning frontend: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_container_memory_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}) >= 99)

frontend: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning frontend: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the (frontend|sourcegraph-frontend) service.
• Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[1d])) >= 80)

frontend: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning frontend: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the (frontend|sourcegraph-frontend) service.
• Docker Compose: Consider increasing memory: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[1d])) >= 80)

frontend: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning frontend: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[5m])) >= 90)

frontend: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning frontend: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[5m])) >= 90)

frontend: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning frontend: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (frontend|sourcegraph-frontend) container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^(frontend|sourcegraph-frontend).*"})) >= 1)

frontend: go_goroutines

maximum active goroutines

Descriptions

• warning frontend: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_go_goroutines"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*(frontend|sourcegraph-frontend)"})) >= 10000)

frontend: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning frontend: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*(frontend|sourcegraph-frontend)"})) >= 2)

frontend: pods_available_percentage

percentage pods available

Descriptions

• critical frontend: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod (frontend|sourcegraph-frontend) (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p (frontend|sourcegraph-frontend).
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_frontend_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*(frontend|sourcegraph-frontend)"}) / count by (app) (up{app=~".*(frontend|sourcegraph-frontend)"}) * 100) <= 90)

frontend: email_delivery_failures

email delivery failure rate over 30 minutes

Descriptions

• warning frontend: 0%+ email delivery failure rate over 30 minutes
• critical frontend: 10%+ email delivery failure rate over 30 minutes

Next steps

• Check your SMTP configuration in site configuration.
• Check sourcegraph-frontend logs for more detailed error messages.
• Check your SMTP provider for more detailed error messages.
• Use sum(increase(src_email_send{success="false"}[30m])) to check the raw count of delivery failures.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_email_delivery_failures",
  "critical_frontend_email_delivery_failures"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum(increase(src_email_send{success="false"}[30m])) / sum(increase(src_email_send[30m])) * 100) > 0)

Generated query for critical alert: max((sum(increase(src_email_send{success="false"}[30m])) / sum(increase(src_email_send[30m])) * 100) >= 10)

frontend: mean_successful_sentinel_duration_over_2h

mean successful sentinel search duration over 2h

Descriptions

• warning frontend: 5s+ mean successful sentinel search duration over 2h for 15m0s
• critical frontend: 8s+ mean successful sentinel search duration over 2h for 30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_mean_successful_sentinel_duration_over_2h",
  "critical_frontend_mean_successful_sentinel_duration_over_2h"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum(rate(src_search_response_latency_seconds_sum{source=~"searchblitz.*",status="success"}[2h])) / sum(rate(src_search_response_latency_seconds_count{source=~"searchblitz.*",status="success"}[2h]))) >= 5)

Generated query for critical alert: max((sum(rate(src_search_response_latency_seconds_sum{source=~"searchblitz.*",status="success"}[2h])) / sum(rate(src_search_response_latency_seconds_count{source=~"searchblitz.*",status="success"}[2h]))) >= 8)

frontend: mean_sentinel_stream_latency_over_2h

mean successful sentinel stream latency over 2h

Descriptions

• warning frontend: 2s+ mean successful sentinel stream latency over 2h for 15m0s
• critical frontend: 3s+ mean successful sentinel stream latency over 2h for 30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_mean_sentinel_stream_latency_over_2h",
  "critical_frontend_mean_sentinel_stream_latency_over_2h"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((sum(rate(src_search_streaming_latency_seconds_sum{source=~"searchblitz.*"}[2h])) / sum(rate(src_search_streaming_latency_seconds_count{source=~"searchblitz.*"}[2h]))) >= 2)

Generated query for critical alert: max((sum(rate(src_search_streaming_latency_seconds_sum{source=~"searchblitz.*"}[2h])) / sum(rate(src_search_streaming_latency_seconds_count{source=~"searchblitz.*"}[2h]))) >= 3)

frontend: 90th_percentile_successful_sentinel_duration_over_2h

90th percentile successful sentinel search duration over 2h

Descriptions

• warning frontend: 5s+ 90th percentile successful sentinel search duration over 2h for 15m0s
• critical frontend: 10s+ 90th percentile successful sentinel search duration over 2h for 3h30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_successful_sentinel_duration_over_2h",
  "critical_frontend_90th_percentile_successful_sentinel_duration_over_2h"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_response_latency_seconds_bucket{source=~"searchblitz.*",status="success"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 5)

Generated query for critical alert: max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_response_latency_seconds_bucket{source=~"searchblitz.*",status="success"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 10)

frontend: 90th_percentile_sentinel_stream_latency_over_2h

90th percentile successful sentinel stream latency over 2h

Descriptions

• warning frontend: 4s+ 90th percentile successful sentinel stream latency over 2h for 15m0s
• critical frontend: 6s+ 90th percentile successful sentinel stream latency over 2h for 3h30m0s

Next steps

• Look at the breakdown by query to determine if a specific query type is being affected
• Check for high CPU usage on zoekt-webserver
• Check Honeycomb for unusual activity
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_frontend_90th_percentile_sentinel_stream_latency_over_2h",
  "critical_frontend_90th_percentile_sentinel_stream_latency_over_2h"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_streaming_latency_seconds_bucket{source=~"searchblitz.*"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 4)

Generated query for critical alert: max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_streaming_latency_seconds_bucket{source=~"searchblitz.*"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 6)

gitserver: cpu_throttling_time

container CPU throttling time %

Descriptions

• warning gitserver: 75%+ container CPU throttling time % for 2m0s
• critical gitserver: 90%+ container CPU throttling time % for 5m0s

Next steps

• • Consider increasing the CPU limit for the container.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_cpu_throttling_time",
  "critical_gitserver_cpu_throttling_time"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum by (container_label_io_kubernetes_pod_name) ((rate(container_cpu_cfs_throttled_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m]) / rate(container_cpu_cfs_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m])) * 100)) >= 75)

Generated query for critical alert: max((sum by (container_label_io_kubernetes_pod_name) ((rate(container_cpu_cfs_throttled_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m]) / rate(container_cpu_cfs_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m])) * 100)) >= 90)

gitserver: disk_space_remaining

disk space remaining

Descriptions

• warning gitserver: less than 15% disk space remaining
• critical gitserver: less than 10% disk space remaining for 10m0s

Next steps

• On a warning alert, you may want to provision more disk space: Disk pressure may result in decreased performance, users having to wait for repositories to clone, etc.
• On a critical alert, you need to provision more disk space. Running out of disk space will result in decreased performance, or complete service outage.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_disk_space_remaining",
  "critical_gitserver_disk_space_remaining"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min(((src_gitserver_disk_space_available / src_gitserver_disk_space_total) * 100) < 15)

Generated query for critical alert: min(((src_gitserver_disk_space_available / src_gitserver_disk_space_total) * 100) < 10)

gitserver: running_git_commands

git commands running on each gitserver instance

Descriptions

• warning gitserver: 50+ git commands running on each gitserver instance for 2m0s
• critical gitserver: 100+ git commands running on each gitserver instance for 5m0s

Next steps

• Check if the problem may be an intermittent and temporary peak using the "Container monitoring" section at the bottom of the Git Server dashboard.
• Single container deployments: Consider upgrading to a Docker Compose deployment which offers better scalability and resource isolation.
• Kubernetes and Docker Compose: Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the Sourcegraph resource estimator.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_running_git_commands",
  "critical_gitserver_running_git_commands"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum by (instance, cmd) (src_gitserver_exec_running)) >= 50)

Generated query for critical alert: max((sum by (instance, cmd) (src_gitserver_exec_running)) >= 100)

gitserver: echo_command_duration_test

echo test command duration

Descriptions

• warning gitserver: 0.02s+ echo test command duration for 30s
• critical gitserver: 1s+ echo test command duration for 1m0s

Next steps

• Single container deployments: Upgrade to a Docker Compose deployment which offers better scalability and resource isolation.
• Kubernetes and Docker Compose: Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the Sourcegraph resource estimator.
• If your persistent volume is slow, you may want to provision more IOPS, usually by increasing the volume size.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_echo_command_duration_test",
  "critical_gitserver_echo_command_duration_test"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(src_gitserver_echo_duration_seconds)) >= 0.02)

Generated query for critical alert: max((max(src_gitserver_echo_duration_seconds)) >= 1)

gitserver: repo_corrupted

number of times a repo corruption has been identified

Descriptions

• critical gitserver: 0+ number of times a repo corruption has been identified

Next steps

• Check the corruption logs for details. gitserver_repos.corruption_logs contains more information.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_gitserver_repo_corrupted"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((sum(rate(src_gitserver_repo_corrupted[5m]))) > 0)

gitserver: repository_clone_queue_size

repository clone queue size

Descriptions

• warning gitserver: 25+ repository clone queue size

Next steps

• If you just added several repositories, the warning may be expected.
• Check which repositories need cloning, by visiting e.g. https://sourcegraph.example.com/site-admin/repositories?filter=not-cloned
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_repository_clone_queue_size"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum(src_gitserver_clone_queue)) >= 25)

gitserver: gitserver_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "gitserver" instances)

Descriptions

• critical gitserver: 300s+ maximum duration since last successful site configuration update (all "gitserver" instances)

Next steps

• This indicates that one or more "gitserver" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "gitserver" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_gitserver_gitserver_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*gitserver"}[1m]))) >= 300)

gitserver: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning gitserver: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical gitserver: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_mean_blocked_seconds_per_conn_request",
  "critical_gitserver_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="gitserver"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="gitserver"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="gitserver"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="gitserver"}[5m]))) >= 0.5)

gitserver: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning gitserver: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_container_cpu_usage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^gitserver.*"}) >= 99)

gitserver: container_memory_usage

container memory usage by instance

Descriptions

• warning gitserver: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_container_memory_usage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^gitserver.*"}) >= 99)

gitserver: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning gitserver: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the gitserver service.
• Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^gitserver.*"}[1d])) >= 80)

gitserver: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning gitserver: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the gitserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^gitserver.*"}[5m])) >= 90)

gitserver: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning gitserver: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of gitserver container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^gitserver.*"})) >= 1)

gitserver: go_goroutines

maximum active goroutines

Descriptions

• warning gitserver: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_go_goroutines"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*gitserver"})) >= 10000)

gitserver: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning gitserver: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_gitserver_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*gitserver"})) >= 2)

gitserver: pods_available_percentage

percentage pods available

Descriptions

• critical gitserver: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod gitserver (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p gitserver.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_gitserver_pods_available_percentage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*gitserver"}) / count by (app) (up{app=~".*gitserver"}) * 100) <= 90)

postgres: connections

active connections

Descriptions

• warning postgres: less than 5 active connections for 5m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_connections"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: min((sum by (job) (pg_stat_activity_count{datname!~"template.*|postgres|cloudsqladmin"}) or sum by (job) (pg_stat_activity_count{datname!~"template.*|cloudsqladmin",job="codeinsights-db"})) <= 5)

postgres: usage_connections_percentage

connection in use

Descriptions

• warning postgres: 80%+ connection in use for 5m0s
• critical postgres: 100%+ connection in use for 5m0s

Next steps

• Consider increasing max_connections of the database instance, learn more
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_usage_connections_percentage",
  "critical_postgres_usage_connections_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (job) (pg_stat_activity_count) / (sum by (job) (pg_settings_max_connections) - sum by (job) (pg_settings_superuser_reserved_connections)) * 100) >= 80)

Generated query for critical alert: max((sum by (job) (pg_stat_activity_count) / (sum by (job) (pg_settings_max_connections) - sum by (job) (pg_settings_superuser_reserved_connections)) * 100) >= 100)

postgres: transaction_durations

maximum transaction durations

Descriptions

• warning postgres: 0.3s+ maximum transaction durations for 5m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_transaction_durations"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (job) (pg_stat_activity_max_tx_duration{datname!~"template.*|postgres|cloudsqladmin",job!="codeintel-db"}) or sum by (job) (pg_stat_activity_max_tx_duration{datname!~"template.*|cloudsqladmin",job="codeinsights-db"})) >= 0.3)

postgres: postgres_up

database availability

Descriptions

• critical postgres: less than 0 database availability for 5m0s

Next steps

• Kubernetes:
  • Determine if the pod was OOM killed using kubectl describe pod (pgsql|codeintel-db|codeinsights) (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
  • Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p (pgsql|codeintel-db|codeinsights).
  • Check if there is any OOMKILL event using the provisioning panels
  • Check kernel logs using dmesg for OOMKILL events on worker nodes
• Docker Compose:
  • Determine if the pod was OOM killed using docker inspect -f '{{json .State}}' (pgsql|codeintel-db|codeinsights) (look for "OOMKilled":true) and, if so, consider increasing the memory limit of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
  • Check the logs before the container restarted to see if there are panic: messages or similar using docker logs (pgsql|codeintel-db|codeinsights) (note this will include logs from the previous and currently running container).
  • Check if there is any OOMKILL event using the provisioning panels
  • Check kernel logs using dmesg for OOMKILL events
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_postgres_up"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((pg_up) <= 0)

postgres: invalid_indexes

invalid indexes (unusable by the query planner)

Descriptions

• critical postgres: 1+ invalid indexes (unusable by the query planner)

Next steps

• Drop and re-create the invalid trigger - please contact Sourcegraph to supply the trigger definition.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_invalid_indexes"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: sum((max by (relname) (pg_invalid_index_count)) >= 1)

postgres: pg_exporter_err

errors scraping postgres exporter

Descriptions

• warning postgres: 1+ errors scraping postgres exporter for 5m0s

Next steps

• Ensure the Postgres exporter can access the Postgres database. Also, check the Postgres exporter logs for errors.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_pg_exporter_err"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((pg_exporter_last_scrape_error) >= 1)

postgres: migration_in_progress

active schema migration

Descriptions

• critical postgres: 1+ active schema migration for 5m0s

Next steps

• The database migration has been in progress for 5 or more minutes - please contact Sourcegraph if this persists.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_migration_in_progress"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((pg_sg_migration_status) >= 1)

postgres: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning postgres: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the (pgsql|codeintel-db|codeinsights) service.
• Docker Compose: Consider increasing cpus: of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[1d])) >= 80)

postgres: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning postgres: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the (pgsql|codeintel-db|codeinsights) service.
• Docker Compose: Consider increasing memory: of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[1d])) >= 80)

postgres: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning postgres: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[5m])) >= 90)

postgres: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning postgres: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[5m])) >= 90)

postgres: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning postgres: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of (pgsql|codeintel-db|codeinsights) container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_postgres_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^(pgsql|codeintel-db|codeinsights).*"})) >= 1)

postgres: pods_available_percentage

percentage pods available

Descriptions

• critical postgres: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod (pgsql|codeintel-db|codeinsights) (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p (pgsql|codeintel-db|codeinsights).
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_postgres_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*(pgsql|codeintel-db|codeinsights)"}) / count by (app) (up{app=~".*(pgsql|codeintel-db|codeinsights)"}) * 100) <= 90)

precise-code-intel-worker: codeintel_upload_queued_max_age

unprocessed upload record queue longest time in queue

Descriptions

• warning precise-code-intel-worker: 18000s+ unprocessed upload record queue longest time in queue

Next steps

• An alert here could be indicative of a few things: an upload surfacing a pathological performance characteristic, precise-code-intel-worker being underprovisioned for the required upload processing throughput, or a higher replica count being required for the volume of uploads.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_codeintel_upload_queued_max_age"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max(src_codeintel_upload_queued_duration_seconds_total{job=~"^precise-code-intel-worker.*"})) >= 18000)

precise-code-intel-worker: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning precise-code-intel-worker: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical precise-code-intel-worker: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_mean_blocked_seconds_per_conn_request",
  "critical_precise-code-intel-worker_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="precise-code-intel-worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="precise-code-intel-worker"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="precise-code-intel-worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="precise-code-intel-worker"}[5m]))) >= 0.5)

precise-code-intel-worker: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning precise-code-intel-worker: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_cpu_usage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 99)

precise-code-intel-worker: container_memory_usage

container memory usage by instance

Descriptions

• warning precise-code-intel-worker: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_memory_usage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 99)

precise-code-intel-worker: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning precise-code-intel-worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the precise-code-intel-worker service.
• Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[1d])) >= 80)

precise-code-intel-worker: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning precise-code-intel-worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the precise-code-intel-worker service.
• Docker Compose: Consider increasing memory: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[1d])) >= 80)

precise-code-intel-worker: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning precise-code-intel-worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[5m])) >= 90)

precise-code-intel-worker: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning precise-code-intel-worker: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[5m])) >= 90)

precise-code-intel-worker: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning precise-code-intel-worker: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of precise-code-intel-worker container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^precise-code-intel-worker.*"})) >= 1)

precise-code-intel-worker: go_goroutines

maximum active goroutines

Descriptions

• warning precise-code-intel-worker: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_go_goroutines"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*precise-code-intel-worker"})) >= 10000)

precise-code-intel-worker: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning precise-code-intel-worker: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_precise-code-intel-worker_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*precise-code-intel-worker"})) >= 2)

precise-code-intel-worker: pods_available_percentage

percentage pods available

Descriptions

• critical precise-code-intel-worker: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod precise-code-intel-worker (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p precise-code-intel-worker.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_precise-code-intel-worker_pods_available_percentage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*precise-code-intel-worker"}) / count by (app) (up{app=~".*precise-code-intel-worker"}) * 100) <= 90)

redis: redis-store_up

redis-store availability

Descriptions

• critical redis: less than 1 redis-store availability for 10s

Next steps

• Ensure redis-store is running
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_redis-store_up"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((redis_up{app="redis-store"}) < 1)

redis: redis-cache_up

redis-cache availability

Descriptions

• critical redis: less than 1 redis-cache availability for 10s

Next steps

• Ensure redis-cache is running
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_redis-cache_up"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((redis_up{app="redis-cache"}) < 1)

redis: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the redis-cache service.
• Docker Compose: Consider increasing cpus: of the redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^redis-cache.*"}[1d])) >= 80)

redis: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the redis-cache service.
• Docker Compose: Consider increasing memory: of the redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-cache.*"}[1d])) >= 80)

redis: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^redis-cache.*"}[5m])) >= 90)

redis: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning redis: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-cache container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-cache.*"}[5m])) >= 90)

redis: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning redis: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-cache container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^redis-cache.*"})) >= 1)

redis: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the redis-store service.
• Docker Compose: Consider increasing cpus: of the redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^redis-store.*"}[1d])) >= 80)

redis: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the redis-store service.
• Docker Compose: Consider increasing memory: of the redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-store.*"}[1d])) >= 80)

redis: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^redis-store.*"}[5m])) >= 90)

redis: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning redis: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-store container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-store.*"}[5m])) >= 90)

redis: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning redis: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of redis-store container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_redis_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^redis-store.*"})) >= 1)

redis: pods_available_percentage

percentage pods available

Descriptions

• critical redis: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod redis-cache (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p redis-cache.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*redis-cache"}) / count by (app) (up{app=~".*redis-cache"}) * 100) <= 90)

redis: pods_available_percentage

percentage pods available

Descriptions

• critical redis: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod redis-store (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p redis-store.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_redis_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*redis-store"}) / count by (app) (up{app=~".*redis-store"}) * 100) <= 90)

worker: worker_job_codeintel-upload-janitor_count

number of worker instances running the codeintel-upload-janitor job

Descriptions

• warning worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 1m0s
• critical worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 5m0s

Next steps

• Ensure your instance defines a worker container such that:
  • WORKER_JOB_ALLOWLIST contains "codeintel-upload-janitor" (or "all"), and
  • WORKER_JOB_BLOCKLIST does not contain "codeintel-upload-janitor"
• Ensure that such a container is not failing to start or stay active
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-upload-janitor_count",
  "critical_worker_worker_job_codeintel-upload-janitor_count"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: (min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) == 1)

Generated query for critical alert: (min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) == 1)

worker: worker_job_codeintel-commitgraph-updater_count

number of worker instances running the codeintel-commitgraph-updater job

Descriptions

• warning worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 1m0s
• critical worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 5m0s

Next steps

• Ensure your instance defines a worker container such that:
  • WORKER_JOB_ALLOWLIST contains "codeintel-commitgraph-updater" (or "all"), and
  • WORKER_JOB_BLOCKLIST does not contain "codeintel-commitgraph-updater"
• Ensure that such a container is not failing to start or stay active
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-commitgraph-updater_count",
  "critical_worker_worker_job_codeintel-commitgraph-updater_count"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: (min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) == 1)

Generated query for critical alert: (min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) == 1)

worker: worker_job_codeintel-autoindexing-scheduler_count

number of worker instances running the codeintel-autoindexing-scheduler job

Descriptions

• warning worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 1m0s
• critical worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 5m0s

Next steps

• Ensure your instance defines a worker container such that:
  • WORKER_JOB_ALLOWLIST contains "codeintel-autoindexing-scheduler" (or "all"), and
  • WORKER_JOB_BLOCKLIST does not contain "codeintel-autoindexing-scheduler"
• Ensure that such a container is not failing to start or stay active
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_worker_job_codeintel-autoindexing-scheduler_count",
  "critical_worker_worker_job_codeintel-autoindexing-scheduler_count"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: (min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) == 1)

Generated query for critical alert: (min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) == 1)

worker: codeintel_commit_graph_queued_max_age

repository queue longest time in queue

Descriptions

• warning worker: 3600s+ repository queue longest time in queue

Next steps

• An alert here is generally indicative of either underprovisioned worker instance(s) and/or an underprovisioned main postgres instance.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_codeintel_commit_graph_queued_max_age"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max(src_codeintel_commit_graph_queued_duration_seconds_total{job=~"^worker.*"})) >= 3600)

worker: perms_syncer_outdated_perms

number of entities with outdated permissions

Descriptions

• warning worker: 100+ number of entities with outdated permissions for 5m0s

Next steps

• Enabled permissions for the first time: Wait for few minutes and see if the number goes down.
• Otherwise: Increase the API rate limit to GitHub, GitLab or Bitbucket Server.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_perms_syncer_outdated_perms"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (type) (src_repo_perms_syncer_outdated_perms)) >= 100)

worker: perms_syncer_sync_duration

95th permissions sync duration

Descriptions

• warning worker: 30s+ 95th permissions sync duration for 5m0s

Next steps

• Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_perms_syncer_sync_duration"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.95, max by (le, type) (rate(src_repo_perms_syncer_sync_duration_seconds_bucket[1m])))) >= 30)

worker: perms_syncer_sync_errors

permissions sync error rate

Descriptions

• critical worker: 1+ permissions sync error rate for 1m0s

Next steps

• Check the network connectivity the Sourcegraph and the code host.
• Check if API rate limit quota is exhausted on the code host.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_worker_perms_syncer_sync_errors"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((max by (type) (ceil(rate(src_repo_perms_syncer_sync_errors_total[1m])))) >= 1)

worker: insights_queue_unutilized_size

insights queue size that is not utilized (not processing)

Descriptions

• warning worker: 0+ insights queue size that is not utilized (not processing) for 30m0s

Next steps

• Verify code insights worker job has successfully started. Restart worker service and monitoring startup logs, looking for worker panics.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_insights_queue_unutilized_size"
]

_{Managed by the Sourcegraph Code Search team.}

Technical details

Generated query for warning alert: max((max(src_query_runner_worker_total{job=~"^worker.*"}) > 0 and on (job) sum by (op) (increase(src_workerutil_dbworker_store_insights_query_runner_jobs_store_total{job=~"^worker.*",op="Dequeue"}[5m])) < 1) > 0)

worker: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning worker: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical worker: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_mean_blocked_seconds_per_conn_request",
  "critical_worker_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="worker"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="worker"}[5m]))) >= 0.5)

worker: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning worker: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_container_cpu_usage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}) >= 99)

worker: container_memory_usage

container memory usage by instance

Descriptions

• warning worker: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_container_memory_usage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^worker.*"}) >= 99)

worker: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the worker service.
• Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}[1d])) >= 80)

worker: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the worker service.
• Docker Compose: Consider increasing memory: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^worker.*"}[1d])) >= 80)

worker: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}[5m])) >= 90)

worker: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning worker: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^worker.*"}[5m])) >= 90)

worker: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning worker: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of worker container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^worker.*"})) >= 1)

worker: go_goroutines

maximum active goroutines

Descriptions

• warning worker: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_go_goroutines"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*worker"})) >= 10000)

worker: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning worker: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_worker_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*worker"})) >= 2)

worker: pods_available_percentage

percentage pods available

Descriptions

• critical worker: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod worker (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p worker.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_worker_pods_available_percentage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*worker"}) / count by (app) (up{app=~".*worker"}) * 100) <= 90)

worker: worker_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "worker" instances)

Descriptions

• critical worker: 300s+ maximum duration since last successful site configuration update (all "worker" instances)

Next steps

• This indicates that one or more "worker" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "worker" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_worker_worker_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~"^worker.*"}[1m]))) >= 300)

repo-updater: src_repoupdater_max_sync_backoff

time since oldest sync

Descriptions

• critical repo-updater: 32400s+ time since oldest sync for 10m0s

Next steps

• An alert here indicates that no code host connections have synced in at least 9h0m0s. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts.
• Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
• Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
• Check the repo-updater logs for errors about syncing.
• Confirm that outbound network connections are allowed where repo-updater is deployed.
• Check back in an hour to see if the issue has resolved itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_src_repoupdater_max_sync_backoff"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((max(src_repoupdater_max_sync_backoff)) >= 32400)

repo-updater: src_repoupdater_syncer_sync_errors_total

site level external service sync error rate

Descriptions

• warning repo-updater: 0.5+ site level external service sync error rate for 10m0s
• critical repo-updater: 1+ site level external service sync error rate for 10m0s

Next steps

• An alert here indicates errors syncing site level repo metadata with code hosts. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts.
• Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
• Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
• Check the repo-updater logs for errors about syncing.
• Confirm that outbound network connections are allowed where repo-updater is deployed.
• Check back in an hour to see if the issue has resolved itself.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_syncer_sync_errors_total",
  "critical_repo-updater_src_repoupdater_syncer_sync_errors_total"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (family) (rate(src_repoupdater_syncer_sync_errors_total{owner!="user",reason!="internal_rate_limit",reason!="invalid_npm_path"}[5m]))) > 0.5)

Generated query for critical alert: max((max by (family) (rate(src_repoupdater_syncer_sync_errors_total{owner!="user",reason!="internal_rate_limit",reason!="invalid_npm_path"}[5m]))) > 1)

repo-updater: syncer_sync_start

repo metadata sync was started

Descriptions

• warning repo-updater: less than 0 repo metadata sync was started for 9h0m0s

Next steps

• Check repo-updater logs for errors.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_syncer_sync_start"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max by (family) (rate(src_repoupdater_syncer_start_sync{family="Syncer.SyncExternalService"}[9h]))) <= 0)

repo-updater: syncer_sync_duration

95th repositories sync duration

Descriptions

• warning repo-updater: 30s+ 95th repositories sync duration for 5m0s

Next steps

• Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_syncer_sync_duration"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.95, max by (le, family, success) (rate(src_repoupdater_syncer_sync_duration_seconds_bucket[1m])))) >= 30)

repo-updater: source_duration

95th repositories source duration

Descriptions

• warning repo-updater: 30s+ 95th repositories source duration for 5m0s

Next steps

• Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_source_duration"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((histogram_quantile(0.95, max by (le) (rate(src_repoupdater_source_duration_seconds_bucket[1m])))) >= 30)

repo-updater: syncer_synced_repos

repositories synced

Descriptions

• warning repo-updater: less than 0 repositories synced for 9h0m0s

Next steps

• Check network connectivity to code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_syncer_synced_repos"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(rate(src_repoupdater_syncer_synced_repos_total[1m]))) <= 0)

repo-updater: sourced_repos

repositories sourced

Descriptions

• warning repo-updater: less than 0 repositories sourced for 9h0m0s

Next steps

• Check network connectivity to code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sourced_repos"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max(rate(src_repoupdater_source_repos_total[1m]))) <= 0)

repo-updater: purge_failed

repositories purge failed

Descriptions

• warning repo-updater: 0+ repositories purge failed for 5m0s

Next steps

• Check repo-updater`s connectivity with gitserver and gitserver logs
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_purge_failed"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(rate(src_repoupdater_purge_failed[1m]))) > 0)

repo-updater: sched_auto_fetch

repositories scheduled due to hitting a deadline

Descriptions

• warning repo-updater: less than 0 repositories scheduled due to hitting a deadline for 9h0m0s

Next steps

• Check repo-updater logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sched_auto_fetch"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max(rate(src_repoupdater_sched_auto_fetch[1m]))) <= 0)

repo-updater: sched_known_repos

repositories managed by the scheduler

Descriptions

• warning repo-updater: less than 0 repositories managed by the scheduler for 10m0s

Next steps

• Check repo-updater logs. This is expected to fire if there are no user added code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sched_known_repos"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max(src_repoupdater_sched_known_repos)) <= 0)

repo-updater: sched_update_queue_length

rate of growth of update queue length over 5 minutes

Descriptions

• critical repo-updater: 0+ rate of growth of update queue length over 5 minutes for 2h0m0s

Next steps

• Check repo-updater logs for indications that the queue is not being processed. The queue length should trend downwards over time as items are sent to GitServer
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_sched_update_queue_length"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((max(deriv(src_repoupdater_sched_update_queue_length[5m]))) > 0)

repo-updater: sched_loops

scheduler loops

Descriptions

• warning repo-updater: less than 0 scheduler loops for 9h0m0s

Next steps

• Check repo-updater logs for errors. This is expected to fire if there are no user added code hosts
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_sched_loops"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max(rate(src_repoupdater_sched_loops[1m]))) <= 0)

repo-updater: src_repoupdater_stale_repos

repos that haven't been fetched in more than 8 hours

Descriptions

• warning repo-updater: 1+ repos that haven't been fetched in more than 8 hours for 25m0s

Next steps

•   					Check repo-updater logs for errors.
    					Check for rows in gitserver_repos where LastError is not an empty string.
  

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_src_repoupdater_stale_repos"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(src_repoupdater_stale_repos)) >= 1)

repo-updater: sched_error

repositories schedule error rate

Descriptions

• critical repo-updater: 1+ repositories schedule error rate for 25m0s

Next steps

• Check repo-updater logs for errors
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_sched_error"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((max(rate(src_repoupdater_sched_error[1m]))) >= 1)

repo-updater: src_repoupdater_external_services_total

the total number of external services

Descriptions

• critical repo-updater: 20000+ the total number of external services for 1h0m0s

Next steps

• Check for spikes in external services, could be abuse
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_src_repoupdater_external_services_total"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((max(src_repoupdater_external_services_total)) >= 20000)

repo-updater: repoupdater_queued_sync_jobs_total

the total number of queued sync jobs

Descriptions

• warning repo-updater: 100+ the total number of queued sync jobs for 1h0m0s

Next steps

• Check if jobs are failing to sync: "SELECT * FROM external_service_sync_jobs WHERE state = errored";
• Increase the number of workers using the repoConcurrentExternalServiceSyncers site config.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_queued_sync_jobs_total"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(src_repoupdater_queued_sync_jobs_total)) >= 100)

repo-updater: repoupdater_completed_sync_jobs_total

the total number of completed sync jobs

Descriptions

• warning repo-updater: 100000+ the total number of completed sync jobs for 1h0m0s

Next steps

• Check repo-updater logs. Jobs older than 1 day should have been removed.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_completed_sync_jobs_total"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(src_repoupdater_completed_sync_jobs_total)) >= 100000)

repo-updater: repoupdater_errored_sync_jobs_percentage

the percentage of external services that have failed their most recent sync

Descriptions

• warning repo-updater: 10%+ the percentage of external services that have failed their most recent sync for 1h0m0s

Next steps

• Check repo-updater logs. Check code host connectivity
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_repoupdater_errored_sync_jobs_percentage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max(src_repoupdater_errored_sync_jobs_percentage)) > 10)

repo-updater: github_graphql_rate_limit_remaining

remaining calls to GitHub graphql API before hitting the rate limit

Descriptions

• warning repo-updater: less than 250 remaining calls to GitHub graphql API before hitting the rate limit

Next steps

• Consider creating a new token for the indicated resource (the name label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_github_graphql_rate_limit_remaining"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max by (name) (src_github_rate_limit_remaining_v2{resource="graphql"})) <= 250)

repo-updater: github_rest_rate_limit_remaining

remaining calls to GitHub rest API before hitting the rate limit

Descriptions

• warning repo-updater: less than 250 remaining calls to GitHub rest API before hitting the rate limit

Next steps

• Consider creating a new token for the indicated resource (the name label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_github_rest_rate_limit_remaining"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max by (name) (src_github_rate_limit_remaining_v2{resource="rest"})) <= 250)

repo-updater: github_search_rate_limit_remaining

remaining calls to GitHub search API before hitting the rate limit

Descriptions

• warning repo-updater: less than 5 remaining calls to GitHub search API before hitting the rate limit

Next steps

• Consider creating a new token for the indicated resource (the name label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_github_search_rate_limit_remaining"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: min((max by (name) (src_github_rate_limit_remaining_v2{resource="search"})) <= 5)

repo-updater: gitlab_rest_rate_limit_remaining

remaining calls to GitLab rest API before hitting the rate limit

Descriptions

• critical repo-updater: less than 30 remaining calls to GitLab rest API before hitting the rate limit

Next steps

• Try restarting the pod to get a different public IP.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_gitlab_rest_rate_limit_remaining"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: min((max by (name) (src_gitlab_rate_limit_remaining{resource="rest"})) <= 30)

repo-updater: repo_updater_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "repo_updater" instances)

Descriptions

• critical repo-updater: 300s+ maximum duration since last successful site configuration update (all "repo_updater" instances)

Next steps

• This indicates that one or more "repo_updater" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "repo_updater" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_repo_updater_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*repo-updater"}[1m]))) >= 300)

repo-updater: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning repo-updater: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical repo-updater: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_mean_blocked_seconds_per_conn_request",
  "critical_repo-updater_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="repo-updater"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="repo-updater"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="repo-updater"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="repo-updater"}[5m]))) >= 0.5)

repo-updater: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning repo-updater: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_container_cpu_usage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^repo-updater.*"}) >= 99)

repo-updater: container_memory_usage

container memory usage by instance

Descriptions

• critical repo-updater: 90%+ container memory usage by instance for 10m0s

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_container_memory_usage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^repo-updater.*"}) >= 90)

repo-updater: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning repo-updater: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the repo-updater service.
• Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^repo-updater.*"}[1d])) >= 80)

repo-updater: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning repo-updater: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the repo-updater service.
• Docker Compose: Consider increasing memory: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^repo-updater.*"}[1d])) >= 80)

repo-updater: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning repo-updater: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^repo-updater.*"}[5m])) >= 90)

repo-updater: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning repo-updater: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^repo-updater.*"}[5m])) >= 90)

repo-updater: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning repo-updater: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of repo-updater container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^repo-updater.*"})) >= 1)

repo-updater: go_goroutines

maximum active goroutines

Descriptions

• warning repo-updater: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_go_goroutines"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*repo-updater"})) >= 10000)

repo-updater: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning repo-updater: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_repo-updater_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*repo-updater"})) >= 2)

repo-updater: pods_available_percentage

percentage pods available

Descriptions

• critical repo-updater: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod repo-updater (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p repo-updater.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_repo-updater_pods_available_percentage"
]

_{Managed by the Sourcegraph Source team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*repo-updater"}) / count by (app) (up{app=~".*repo-updater"}) * 100) <= 90)

searcher: replica_traffic

requests per second per replica over 10m

Descriptions

• warning searcher: 5+ requests per second per replica over 10m

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_replica_traffic"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((sum by (instance) (rate(searcher_service_request_total[10m]))) >= 5)

searcher: unindexed_search_request_errors

unindexed search request errors every 5m by code

Descriptions

• warning searcher: 5%+ unindexed search request errors every 5m by code for 5m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_unindexed_search_request_errors"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((sum by (code) (increase(searcher_service_request_total{code!="200",code!="canceled"}[5m])) / ignoring (code) group_left () sum(increase(searcher_service_request_total[5m])) * 100) >= 5)

searcher: searcher_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "searcher" instances)

Descriptions

• critical searcher: 300s+ maximum duration since last successful site configuration update (all "searcher" instances)

Next steps

• This indicates that one or more "searcher" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "searcher" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_searcher_searcher_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*searcher"}[1m]))) >= 300)

searcher: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning searcher: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical searcher: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_mean_blocked_seconds_per_conn_request",
  "critical_searcher_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="searcher"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="searcher"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="searcher"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="searcher"}[5m]))) >= 0.5)

searcher: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning searcher: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the searcher container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_container_cpu_usage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^searcher.*"}) >= 99)

searcher: container_memory_usage

container memory usage by instance

Descriptions

• warning searcher: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of searcher container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_container_memory_usage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^searcher.*"}) >= 99)

searcher: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning searcher: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the searcher service.
• Docker Compose: Consider increasing cpus: of the searcher container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^searcher.*"}[1d])) >= 80)

searcher: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning searcher: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the searcher service.
• Docker Compose: Consider increasing memory: of the searcher container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^searcher.*"}[1d])) >= 80)

searcher: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning searcher: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the searcher container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^searcher.*"}[5m])) >= 90)

searcher: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning searcher: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of searcher container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^searcher.*"}[5m])) >= 90)

searcher: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning searcher: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of searcher container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^searcher.*"})) >= 1)

searcher: go_goroutines

maximum active goroutines

Descriptions

• warning searcher: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_go_goroutines"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*searcher"})) >= 10000)

searcher: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning searcher: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_searcher_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*searcher"})) >= 2)

searcher: pods_available_percentage

percentage pods available

Descriptions

• critical searcher: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod searcher (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p searcher.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_searcher_pods_available_percentage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*searcher"}) / count by (app) (up{app=~".*searcher"}) * 100) <= 90)

symbols: symbols_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "symbols" instances)

Descriptions

• critical symbols: 300s+ maximum duration since last successful site configuration update (all "symbols" instances)

Next steps

• This indicates that one or more "symbols" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "symbols" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_symbols_symbols_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*symbols"}[1m]))) >= 300)

symbols: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning symbols: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical symbols: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_mean_blocked_seconds_per_conn_request",
  "critical_symbols_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="symbols"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="symbols"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="symbols"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="symbols"}[5m]))) >= 0.5)

symbols: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning symbols: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the symbols container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_container_cpu_usage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^symbols.*"}) >= 99)

symbols: container_memory_usage

container memory usage by instance

Descriptions

• warning symbols: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of symbols container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_container_memory_usage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^symbols.*"}) >= 99)

symbols: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning symbols: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the symbols service.
• Docker Compose: Consider increasing cpus: of the symbols container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^symbols.*"}[1d])) >= 80)

symbols: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning symbols: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the symbols service.
• Docker Compose: Consider increasing memory: of the symbols container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^symbols.*"}[1d])) >= 80)

symbols: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning symbols: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the symbols container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^symbols.*"}[5m])) >= 90)

symbols: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning symbols: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of symbols container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^symbols.*"}[5m])) >= 90)

symbols: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning symbols: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of symbols container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^symbols.*"})) >= 1)

symbols: go_goroutines

maximum active goroutines

Descriptions

• warning symbols: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_go_goroutines"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*symbols"})) >= 10000)

symbols: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning symbols: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_symbols_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*symbols"})) >= 2)

symbols: pods_available_percentage

percentage pods available

Descriptions

• critical symbols: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod symbols (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p symbols.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_symbols_pods_available_percentage"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*symbols"}) / count by (app) (up{app=~".*symbols"}) * 100) <= 90)

syntect-server: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning syntect-server: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the syntect-server container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_container_cpu_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^syntect-server.*"}) >= 99)

syntect-server: container_memory_usage

container memory usage by instance

Descriptions

• warning syntect-server: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of syntect-server container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_container_memory_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^syntect-server.*"}) >= 99)

syntect-server: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning syntect-server: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the syntect-server service.
• Docker Compose: Consider increasing cpus: of the syntect-server container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^syntect-server.*"}[1d])) >= 80)

syntect-server: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning syntect-server: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the syntect-server service.
• Docker Compose: Consider increasing memory: of the syntect-server container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntect-server.*"}[1d])) >= 80)

syntect-server: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning syntect-server: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the syntect-server container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^syntect-server.*"}[5m])) >= 90)

syntect-server: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning syntect-server: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of syntect-server container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntect-server.*"}[5m])) >= 90)

syntect-server: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning syntect-server: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of syntect-server container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_syntect-server_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^syntect-server.*"})) >= 1)

syntect-server: pods_available_percentage

percentage pods available

Descriptions

• critical syntect-server: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod syntect-server (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p syntect-server.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_syntect-server_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*syntect-server"}) / count by (app) (up{app=~".*syntect-server"}) * 100) <= 90)

zoekt: average_resolve_revision_duration

average resolve revision duration over 5m

Descriptions

• warning zoekt: 15s+ average resolve revision duration over 5m

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_average_resolve_revision_duration"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((sum(rate(resolve_revision_seconds_sum[5m])) / sum(rate(resolve_revision_seconds_count[5m]))) >= 15)

zoekt: get_index_options_error_increase

the number of repositories we failed to get indexing options over 5m

Descriptions

• warning zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 5m0s
• critical zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 35m0s

Next steps

• View error rates on gitserver and frontend to identify root cause.
• Rollback frontend/gitserver deployment if due to a bad code change.
• View error logs for getIndexOptions via net/trace debug interface. For example click on a indexed-search-indexer- on https://sourcegraph.com/-/debug/. Then click on Traces. Replace sourcegraph.com with your instance address.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_get_index_options_error_increase",
  "critical_zoekt_get_index_options_error_increase"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((sum(increase(get_index_options_error_total[5m]))) >= 100)

Generated query for critical alert: max((sum(increase(get_index_options_error_total[5m]))) >= 100)

zoekt: indexed_search_request_errors

indexed search request errors every 5m by code

Descriptions

• warning zoekt: 5%+ indexed search request errors every 5m by code for 5m0s

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_indexed_search_request_errors"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((sum by (code) (increase(src_zoekt_request_duration_seconds_count{code!~"2.."}[5m])) / ignoring (code) group_left () sum(increase(src_zoekt_request_duration_seconds_count[5m])) * 100) >= 5)

zoekt: memory_map_areas_percentage_used

process memory map areas percentage used (per instance)

Descriptions

• warning zoekt: 60%+ process memory map areas percentage used (per instance)
• critical zoekt: 80%+ process memory map areas percentage used (per instance)

Next steps

• If you are running out of memory map areas, you could resolve this by:

  • Enabling shard merging for Zoekt: Set SRC_ENABLE_SHARD_MERGING="1" for zoekt-indexserver. Use this option if your corpus of repositories has a high percentage of small, rarely updated repositories. See documentation.
  • Creating additional Zoekt replicas: This spreads all the shards out amongst more replicas, which means that each individual replica will have fewer shards. This, in turn, decreases the amount of memory map areas that a single replica can create (in order to load the shards into memory).
  • Increasing the virtual memory subsystem`s "max_map_count" parameter which defines the upper limit of memory areas a process can use. The default value of max_map_count is usually 65536. We recommend to set this value to 2x the number of repos to be indexed per Zoekt instance. This means, if you want to index 240k repositories with 3 Zoekt instances, set max_map_count to (240000 / 3) * 2 = 160000. The exact instructions for tuning this parameter can differ depending on your environment. See https://kernel.org/doc/Documentation/sysctl/vm.txt for more information.

• More help interpreting this metric is available in the dashboards reference.

• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_memory_map_areas_percentage_used",
  "critical_zoekt_memory_map_areas_percentage_used"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max(((proc_metrics_memory_map_current_count / proc_metrics_memory_map_max_limit) * 100) >= 60)

Generated query for critical alert: max(((proc_metrics_memory_map_current_count / proc_metrics_memory_map_max_limit) * 100) >= 80)

zoekt: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning zoekt: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the zoekt-indexserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_cpu_usage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-indexserver.*"}) >= 99)

zoekt: container_memory_usage

container memory usage by instance

Descriptions

• warning zoekt: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of zoekt-indexserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_memory_usage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-indexserver.*"}) >= 99)

zoekt: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning zoekt: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the zoekt-webserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_cpu_usage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-webserver.*"}) >= 99)

zoekt: container_memory_usage

container memory usage by instance

Descriptions

• warning zoekt: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of zoekt-webserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_memory_usage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-webserver.*"}) >= 99)

zoekt: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning zoekt: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the zoekt-indexserver service.
• Docker Compose: Consider increasing cpus: of the zoekt-indexserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-indexserver.*"}[1d])) >= 80)

zoekt: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning zoekt: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the zoekt-indexserver service.
• Docker Compose: Consider increasing memory: of the zoekt-indexserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-indexserver.*"}[1d])) >= 80)

zoekt: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning zoekt: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the zoekt-indexserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-indexserver.*"}[5m])) >= 90)

zoekt: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning zoekt: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of zoekt-indexserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-indexserver.*"}[5m])) >= 90)

zoekt: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning zoekt: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of zoekt-indexserver container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^zoekt-indexserver.*"})) >= 1)

zoekt: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning zoekt: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the zoekt-webserver service.
• Docker Compose: Consider increasing cpus: of the zoekt-webserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-webserver.*"}[1d])) >= 80)

zoekt: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning zoekt: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the zoekt-webserver service.
• Docker Compose: Consider increasing memory: of the zoekt-webserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-webserver.*"}[1d])) >= 80)

zoekt: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning zoekt: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the zoekt-webserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-webserver.*"}[5m])) >= 90)

zoekt: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning zoekt: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of zoekt-webserver container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-webserver.*"}[5m])) >= 90)

zoekt: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning zoekt: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of zoekt-webserver container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_zoekt_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^zoekt-webserver.*"})) >= 1)

zoekt: pods_available_percentage

percentage pods available

Descriptions

• critical zoekt: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod indexed-search (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p indexed-search.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_zoekt_pods_available_percentage"
]

_{Managed by the Sourcegraph Search Platform team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*indexed-search"}) / count by (app) (up{app=~".*indexed-search"}) * 100) <= 90)

prometheus: prometheus_rule_eval_duration

average prometheus rule group evaluation duration over 10m by rule group

Descriptions

• warning prometheus: 30s+ average prometheus rule group evaluation duration over 10m by rule group

Next steps

• Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary.
• If the rule group taking a long time to evaluate belongs to /sg_prometheus_addons, try reducing the complexity of any custom Prometheus rules provided.
• If the rule group taking a long time to evaluate belongs to /sg_config_prometheus, please open an issue.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_rule_eval_duration"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (rule_group) (avg_over_time(prometheus_rule_group_last_duration_seconds[10m]))) >= 30)

prometheus: prometheus_rule_eval_failures

failed prometheus rule evaluations over 5m by rule group

Descriptions

• warning prometheus: 0+ failed prometheus rule evaluations over 5m by rule group

Next steps

• Check Prometheus logs for messages related to rule group evaluation (generally with log field component="rule manager").
• If the rule group failing to evaluate belongs to /sg_prometheus_addons, ensure any custom Prometheus configuration provided is valid.
• If the rule group taking a long time to evaluate belongs to /sg_config_prometheus, please open an issue.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_rule_eval_failures"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (rule_group) (rate(prometheus_rule_evaluation_failures_total[5m]))) > 0)

prometheus: alertmanager_notification_latency

alertmanager notification latency over 1m by integration

Descriptions

• warning prometheus: 1s+ alertmanager notification latency over 1m by integration

Next steps

• Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary.
• Ensure that your observability.alerts configuration (in site configuration) is valid.
• Check if the relevant alert integration service is experiencing downtime or issues.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_alertmanager_notification_latency"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (integration) (rate(alertmanager_notification_latency_seconds_sum[1m]))) >= 1)

prometheus: alertmanager_notification_failures

failed alertmanager notifications over 1m by integration

Descriptions

• warning prometheus: 0+ failed alertmanager notifications over 1m by integration

Next steps

• Ensure that your observability.alerts configuration (in site configuration) is valid.
• Check if the relevant alert integration service is experiencing downtime or issues.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_alertmanager_notification_failures"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (integration) (rate(alertmanager_notifications_failed_total[1m]))) > 0)

prometheus: prometheus_config_status

prometheus configuration reload status

Descriptions

• warning prometheus: less than 1 prometheus configuration reload status

Next steps

• Check Prometheus logs for messages related to configuration loading.
• Ensure any custom configuration you have provided Prometheus is valid.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_config_status"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: min((prometheus_config_last_reload_successful) < 1)

prometheus: alertmanager_config_status

alertmanager configuration reload status

Descriptions

• warning prometheus: less than 1 alertmanager configuration reload status

Next steps

• Ensure that your observability.alerts configuration (in site configuration) is valid.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_alertmanager_config_status"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: min((alertmanager_config_last_reload_successful) < 1)

prometheus: prometheus_tsdb_op_failure

prometheus tsdb failures by operation over 1m by operation

Descriptions

• warning prometheus: 0+ prometheus tsdb failures by operation over 1m by operation

Next steps

• Check Prometheus logs for messages related to the failing operation.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_tsdb_op_failure"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((increase(label_replace({__name__=~"prometheus_tsdb_(.*)_failed_total"}, "operation", "$1", "__name__", "(.+)s_failed_total")[5m:1m])) > 0)

prometheus: prometheus_target_sample_exceeded

prometheus scrapes that exceed the sample limit over 10m

Descriptions

• warning prometheus: 0+ prometheus scrapes that exceed the sample limit over 10m

Next steps

• Check Prometheus logs for messages related to target scrape failures.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_target_sample_exceeded"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m])) > 0)

prometheus: prometheus_target_sample_duplicate

prometheus scrapes rejected due to duplicate timestamps over 10m

Descriptions

• warning prometheus: 0+ prometheus scrapes rejected due to duplicate timestamps over 10m

Next steps

• Check Prometheus logs for messages related to target scrape failures.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_prometheus_target_sample_duplicate"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((increase(prometheus_target_scrapes_sample_duplicate_timestamp_total[10m])) > 0)

prometheus: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning prometheus: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the prometheus container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_container_cpu_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^prometheus.*"}) >= 99)

prometheus: container_memory_usage

container memory usage by instance

Descriptions

• warning prometheus: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of prometheus container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_container_memory_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^prometheus.*"}) >= 99)

prometheus: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning prometheus: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the prometheus service.
• Docker Compose: Consider increasing cpus: of the prometheus container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^prometheus.*"}[1d])) >= 80)

prometheus: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning prometheus: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the prometheus service.
• Docker Compose: Consider increasing memory: of the prometheus container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^prometheus.*"}[1d])) >= 80)

prometheus: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning prometheus: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the prometheus container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^prometheus.*"}[5m])) >= 90)

prometheus: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning prometheus: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of prometheus container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^prometheus.*"}[5m])) >= 90)

prometheus: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning prometheus: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of prometheus container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_prometheus_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^prometheus.*"})) >= 1)

prometheus: pods_available_percentage

percentage pods available

Descriptions

• critical prometheus: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod prometheus (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p prometheus.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_prometheus_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*prometheus"}) / count by (app) (up{app=~".*prometheus"}) * 100) <= 90)

executor: executor_handlers

executor active handlers

Descriptions

• critical executor: 0 active executor handlers and > 0 queue size for 5m0s

Next steps

• Check to see the state of any compute VMs, they may be taking longer than expected to boot.
• Make sure the executors appear under Site Admin > Executors.
• Check the Grafana dashboard section for APIClient, it should do frequent requests to Dequeue and Heartbeat and those must not fail.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_executor_executor_handlers"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Custom query for critical alert: min(((sum(src_executor_processor_handlers{sg_job=~"^sourcegraph-executors.*"}) or vector(0)) == 0 and (sum by (queue) (src_executor_total{job=~"^sourcegraph-executors.*"})) > 0) <= 0)

executor: executor_processor_error_rate

executor operation error rate over 5m

Descriptions

• warning executor: 100%+ executor operation error rate over 5m for 1h0m0s

Next steps

• Determine the cause of failure from the auto-indexing job logs in the site-admin page.
• This alert fires if all executor jobs have been failing for the past hour. The alert will continue for up to 5 hours until the error rate is no longer 100%, even if there are no running jobs in that time, as the problem is not know to be resolved until jobs start succeeding again.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_executor_executor_processor_error_rate"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Custom query for warning alert: max((last_over_time(sum(increase(src_executor_processor_errors_total{sg_job=~"^sourcegraph-executors.*"}[5m]))[5h:]) / (last_over_time(sum(increase(src_executor_processor_total{sg_job=~"^sourcegraph-executors.*"}[5m]))[5h:]) + last_over_time(sum(increase(src_executor_processor_errors_total{sg_job=~"^sourcegraph-executors.*"}[5m]))[5h:])) * 100) >= 100)

executor: go_goroutines

maximum active goroutines

Descriptions

• warning executor: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_executor_go_goroutines"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (sg_instance) (go_goroutines{sg_job=~".*sourcegraph-executors"})) >= 10000)

executor: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning executor: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_executor_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max by (sg_instance) (go_gc_duration_seconds{sg_job=~".*sourcegraph-executors"})) >= 2)

codeintel-uploads: codeintel_commit_graph_queued_max_age

repository queue longest time in queue

Descriptions

• warning codeintel-uploads: 3600s+ repository queue longest time in queue

Next steps

• An alert here is generally indicative of either underprovisioned worker instance(s) and/or an underprovisioned main postgres instance.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_codeintel-uploads_codeintel_commit_graph_queued_max_age"
]

_{Managed by the Sourcegraph Code intelligence team.}

Technical details

Generated query for warning alert: max((max(src_codeintel_commit_graph_queued_duration_seconds_total)) >= 3600)

telemetry: telemetry_gateway_exporter_queue_growth

rate of growth of export queue over 30m

Descriptions

• warning telemetry: 1+ rate of growth of export queue over 30m for 1h0m0s
• critical telemetry: 1+ rate of growth of export queue over 30m for 36h0m0s

Next steps

• Check the "number of events exported per batch over 30m" dashboard panel to see if export throughput is at saturation.
• Increase TELEMETRY_GATEWAY_EXPORTER_EXPORT_BATCH_SIZE to export more events per batch.
• Reduce TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL to schedule more export jobs.
• See worker logs in the worker.telemetrygateway-exporter log scope for more details to see if any export errors are occuring - if logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_telemetry_telemetry_gateway_exporter_queue_growth",
  "critical_telemetry_telemetry_gateway_exporter_queue_growth"
]

_{Managed by the Sourcegraph Data & Analytics team.}

Technical details

Generated query for warning alert: max((max(deriv(src_telemetrygatewayexporter_queue_size[30m]))) > 1)

Generated query for critical alert: max((max(deriv(src_telemetrygatewayexporter_queue_size[30m]))) > 1)

telemetry: telemetrygatewayexporter_exporter_errors_total

events exporter operation errors every 30m

Descriptions

• warning telemetry: 0+ events exporter operation errors every 30m

Next steps

• Failures indicate that exporting of telemetry events from Sourcegraph are failing. This may affect the performance of the database as the backlog grows.
• See worker logs in the worker.telemetrygateway-exporter log scope for more details. If logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_telemetry_telemetrygatewayexporter_exporter_errors_total"
]

_{Managed by the Sourcegraph Data & Analytics team.}

Technical details

Generated query for warning alert: max((sum(increase(src_telemetrygatewayexporter_exporter_errors_total{job=~"^worker.*"}[30m]))) > 0)

telemetry: telemetrygatewayexporter_queue_cleanup_errors_total

export queue cleanup operation errors every 30m

Descriptions

• warning telemetry: 0+ export queue cleanup operation errors every 30m

Next steps

• Failures indicate that pruning of already-exported telemetry events from the database is failing. This may affect the performance of the database as the export queue table grows.
• See worker logs in the worker.telemetrygateway-exporter log scope for more details.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_telemetry_telemetrygatewayexporter_queue_cleanup_errors_total"
]

_{Managed by the Sourcegraph Data & Analytics team.}

Technical details

Generated query for warning alert: max((sum(increase(src_telemetrygatewayexporter_queue_cleanup_errors_total{job=~"^worker.*"}[30m]))) > 0)

telemetry: telemetrygatewayexporter_queue_metrics_reporter_errors_total

export backlog metrics reporting operation errors every 30m

Descriptions

• warning telemetry: 0+ export backlog metrics reporting operation errors every 30m

Next steps

• Failures indicate that reporting of telemetry events metrics is failing. This may affect the reliability of telemetry events export metrics.
• See worker logs in the worker.telemetrygateway-exporter log scope for more details.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_telemetry_telemetrygatewayexporter_queue_metrics_reporter_errors_total"
]

_{Managed by the Sourcegraph Data & Analytics team.}

Technical details

Generated query for warning alert: max((sum(increase(src_telemetrygatewayexporter_queue_metrics_reporter_errors_total{job=~"^worker.*"}[30m]))) > 0)

telemetry: telemetry_job_error_rate

usage data exporter operation error rate over 5m

Descriptions

• warning telemetry: 0%+ usage data exporter operation error rate over 5m for 30m0s

Next steps

• Involved cloud team to inspect logs of the managed instance to determine error sources.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_telemetry_telemetry_job_error_rate"
]

_{Managed by the Sourcegraph Data & Analytics team.}

Technical details

Generated query for warning alert: max((sum by (op) (increase(src_telemetry_job_errors_total{job=~"^worker.*"}[5m])) / (sum by (op) (increase(src_telemetry_job_total{job=~"^worker.*"}[5m])) + sum by (op) (increase(src_telemetry_job_errors_total{job=~"^worker.*"}[5m]))) * 100) > 0)

telemetry: telemetry_job_utilized_throughput

utilized percentage of maximum throughput

Descriptions

• warning telemetry: 90%+ utilized percentage of maximum throughput for 30m0s

Next steps

• Throughput utilization is high. This could be a signal that this instance is producing too many events for the export job to keep up. Configure more throughput using the maxBatchSize option.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_telemetry_telemetry_job_utilized_throughput"
]

_{Managed by the Sourcegraph Data & Analytics team.}

Technical details

Generated query for warning alert: max((rate(src_telemetry_job_total{op="SendEvents"}[1h]) / on () group_right () src_telemetry_job_max_throughput * 100) > 90)

otel-collector: otel_span_refused

spans refused per receiver

Descriptions

• warning otel-collector: 1+ spans refused per receiver for 5m0s

Next steps

• Check logs of the collector and configuration of the receiver
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_otel-collector_otel_span_refused"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (receiver) (rate(otelcol_receiver_refused_spans[1m]))) > 1)

otel-collector: otel_span_export_failures

span export failures by exporter

Descriptions

• warning otel-collector: 1+ span export failures by exporter for 5m0s

Next steps

• Check the configuration of the exporter and if the service being exported is up
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_otel-collector_otel_span_export_failures"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (exporter) (rate(otelcol_exporter_send_failed_spans[1m]))) > 1)

otel-collector: otelcol_exporter_enqueue_failed_spans

exporter enqueue failed spans

Descriptions

• warning otel-collector: 0+ exporter enqueue failed spans for 5m0s

Next steps

• Check the configuration of the exporter and if the service being exported is up. This may be caused by a queue full of unsettled elements, so you may need to decrease your sending rate or horizontally scale collectors.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_otel-collector_otelcol_exporter_enqueue_failed_spans"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (exporter) (rate(otelcol_exporter_enqueue_failed_spans{job=~"^.*"}[1m]))) > 0)

otel-collector: otelcol_processor_dropped_spans

spans dropped per processor per minute

Descriptions

• warning otel-collector: 0+ spans dropped per processor per minute for 5m0s

Next steps

• Check the configuration of the processor
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_otel-collector_otelcol_processor_dropped_spans"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((sum by (processor) (rate(otelcol_processor_dropped_spans[1m]))) > 0)

otel-collector: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning otel-collector: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the otel-collector container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_otel-collector_container_cpu_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^otel-collector.*"}) >= 99)

otel-collector: container_memory_usage

container memory usage by instance

Descriptions

• warning otel-collector: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of otel-collector container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_otel-collector_container_memory_usage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^otel-collector.*"}) >= 99)

otel-collector: pods_available_percentage

percentage pods available

Descriptions

• critical otel-collector: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod otel-collector (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p otel-collector.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_otel-collector_pods_available_percentage"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*otel-collector"}) / count by (app) (up{app=~".*otel-collector"}) * 100) <= 90)

embeddings: embeddings_site_configuration_duration_since_last_successful_update_by_instance

maximum duration since last successful site configuration update (all "embeddings" instances)

Descriptions

• critical embeddings: 300s+ maximum duration since last successful site configuration update (all "embeddings" instances)

Next steps

• This indicates that one or more "embeddings" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
• Check for relevant errors in the "embeddings" logs, as well as frontend`s logs.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_embeddings_embeddings_site_configuration_duration_since_last_successful_update_by_instance"
]

_{Managed by the Sourcegraph Infrastructure Org team.}

Technical details

Generated query for critical alert: max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*embeddings"}[1m]))) >= 300)

embeddings: mean_blocked_seconds_per_conn_request

mean blocked seconds per conn request

Descriptions

• warning embeddings: 0.1s+ mean blocked seconds per conn request for 10m0s
• critical embeddings: 0.5s+ mean blocked seconds per conn request for 10m0s

Next steps

• Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
• Scale up Postgres memory/cpus - see our scaling guide
• If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_mean_blocked_seconds_per_conn_request",
  "critical_embeddings_mean_blocked_seconds_per_conn_request"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="embeddings"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="embeddings"}[5m]))) >= 0.1)

Generated query for critical alert: max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="embeddings"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="embeddings"}[5m]))) >= 0.5)

embeddings: container_cpu_usage

container cpu usage total (1m average) across all cores by instance

Descriptions

• warning embeddings: 99%+ container cpu usage total (1m average) across all cores by instance

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the embeddings container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_container_cpu_usage"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((cadvisor_container_cpu_usage_percentage_total{name=~"^embeddings.*"}) >= 99)

embeddings: container_memory_usage

container memory usage by instance

Descriptions

• warning embeddings: 99%+ container memory usage by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of embeddings container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_container_memory_usage"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((cadvisor_container_memory_usage_percentage_total{name=~"^embeddings.*"}) >= 99)

embeddings: provisioning_container_cpu_usage_long_term

container cpu usage total (90th percentile over 1d) across all cores by instance

Descriptions

• warning embeddings: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the Deployment.yaml for the embeddings service.
• Docker Compose: Consider increasing cpus: of the embeddings container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_provisioning_container_cpu_usage_long_term"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^embeddings.*"}[1d])) >= 80)

embeddings: provisioning_container_memory_usage_long_term

container memory usage (1d maximum) by instance

Descriptions

• warning embeddings: 80%+ container memory usage (1d maximum) by instance for 336h0m0s

Next steps

• Kubernetes: Consider increasing memory limits in the Deployment.yaml for the embeddings service.
• Docker Compose: Consider increasing memory: of the embeddings container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_provisioning_container_memory_usage_long_term"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^embeddings.*"}[1d])) >= 80)

embeddings: provisioning_container_cpu_usage_short_term

container cpu usage total (5m maximum) across all cores by instance

Descriptions

• warning embeddings: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s

Next steps

• Kubernetes: Consider increasing CPU limits in the the relevant Deployment.yaml.
• Docker Compose: Consider increasing cpus: of the embeddings container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_provisioning_container_cpu_usage_short_term"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^embeddings.*"}[5m])) >= 90)

embeddings: provisioning_container_memory_usage_short_term

container memory usage (5m maximum) by instance

Descriptions

• warning embeddings: 90%+ container memory usage (5m maximum) by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of embeddings container in docker-compose.yml.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_provisioning_container_memory_usage_short_term"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^embeddings.*"}[5m])) >= 90)

embeddings: container_oomkill_events_total

container OOMKILL events total by instance

Descriptions

• warning embeddings: 1+ container OOMKILL events total by instance

Next steps

• Kubernetes: Consider increasing memory limit in relevant Deployment.yaml.
• Docker Compose: Consider increasing memory: of embeddings container in docker-compose.yml.
• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_container_oomkill_events_total"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((max by (name) (container_oom_events_total{name=~"^embeddings.*"})) >= 1)

embeddings: go_goroutines

maximum active goroutines

Descriptions

• warning embeddings: 10000+ maximum active goroutines for 10m0s

Next steps

• More help interpreting this metric is available in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_go_goroutines"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_goroutines{job=~".*embeddings"})) >= 10000)

embeddings: go_gc_duration_seconds

maximum go garbage collection duration

Descriptions

• warning embeddings: 2s+ maximum go garbage collection duration

Next steps

• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "warning_embeddings_go_gc_duration_seconds"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for warning alert: max((max by (instance) (go_gc_duration_seconds{job=~".*embeddings"})) >= 2)

embeddings: pods_available_percentage

percentage pods available

Descriptions

• critical embeddings: less than 90% percentage pods available for 10m0s

Next steps

• Determine if the pod was OOM killed using kubectl describe pod embeddings (look for OOMKilled: true) and, if so, consider increasing the memory limit in the relevant Deployment.yaml.
• Check the logs before the container restarted to see if there are panic: messages or similar using kubectl logs -p embeddings.
• Learn more about the related dashboard panel in the dashboards reference.
• Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:

"observability.silenceAlerts": [
  "critical_embeddings_pods_available_percentage"
]

_{Managed by the Sourcegraph Cody team.}

Technical details

Generated query for critical alert: min((sum by (app) (up{app=~".*embeddings"}) / count by (app) (up{app=~".*embeddings"}) * 100) <= 90)