Sourcegraph monitoring generator
The monitoring generator manages converting monitoring definitions into integrations with Sourcegraph's monitoring ecosystem.
Its purpose is to help enable a cohesive observability experience for site administrators, codify Sourcegraph's monitoring pillars, and make it easy for developers to add monitoring for their Sourcegraph services by generating integrations with Sourcegraph's monitoring ecosystem for free.
Reference
- Usage and development for developing the generator itself
- Monitoring API for interacting with the generator library
- How to add monitoring definitions for developers looking to add monitoring for their services
Features
Documentation generation
The generator automatically creates documentation from monitoring definitions that customers and engineers can reference. These include:
Links to generated documentation can be provided in our other generated integrations - for example, Slack alerts will provide a link to the appropriate alert solutions entry, and Grafana panels will link to the appropriate dashboards reference entry.
Grafana integration
The generator automatically generates and ships dashboards from monitoring definitions within the Sourcegraph Grafana distribution.
It also takes care of the following:
- Graphs within rows are sized appropriately
- Alerts visualization through the
ObservableAlertDefinitionAPI:- Overview graphs for alerts (both Sourcegraph-wide and per-service)
- Threshold lines for alerts of all levels are rendered in graphs
- Formatting of units, labels, and more (using either the defaults, or the
ObservablePanelOptionAPI) - Providing links to generated documentation
- Configuring annotation layers for alert events and version changes
- Maintaining a uniform look and feel across all dashboards
Links to generated documentation can be provided in our other generated integrations - for example, Slack alerts will provide a link to the appropriate service's dashboard.
Prometheus integration
The generator automatically generates and ships Prometheus recording rules and alerts within the Sourcegraph Prometheus distribution. This include the following, all with appropriate and consistent labels:
alert_countrecording rules- Native Prometheus alerts, leveraged by our Alertmanager integration
Generated Prometheus recording rules are leveraged by the Grafana integration.
Alertmanager integration
The generator's Prometheus integration is a critical part of the Sourcegraph's alerting capabilities, which handles alert routing by level and formatting of alert messages to include links to documentation and dashboards. Learn more about using Sourcegraph alerting in the alerting documentation. This is possible due to the labels generated by the Prometheus integration.
At Sourcegraph, extended routing based on team ownership (as defined by ObservableOwner) is also used to route customer support requests and on-call events through OpsGenie.