Alerts can be configured to notify site admins when there is something wrong or noteworthy on the Sourcegraph instance.
Visit your site configuration (e.g. https://sourcegraph.example.com/site-admin/configuration
) to configure alerts using the observability.alerts
field. As always, you can use Ctrl+Space
at any time to get hints about allowed fields as well as relevant documentation inside the configuration editor.
Once configured, Sourcegraph alerts will automatically be routed to the appropriate notification channels by severity level.
"observability.alerts": [ { "level": "critical", "notifier": { "type": "slack", // Slack incoming webhook URL. "url": "https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxx", } } ]
"observability.alerts": [ { "level": "critical", "notifier": { "type": "pagerduty", // Routing key for the PagerDuty Events API v2 "routingKey": "XXXXXXXX" } } ]
"observability.alerts": [ { "level": "critical", "notifier": { "type": "webhook", // Webhook URL. "url": "https://my.webhook.url" } } ]
Note that to receive email notifications, the email.address
and email.smtp
fields must be configured in site configuration.
"observability.alerts": [ { "level": "critical", "notifier": { "type": "email", // Address where alerts will be sent "address": "[email protected]" } } ]
If there is an alert you are aware of and you wish to silence notifications for it, add an entry to the observability.silenceAlerts
field. For example:
{ "observability.silenceAlerts": [ "warning_gitserver_disk_space_remaining" ] }
You can find the appropriate identifier for each alert in alert solutions.
Before configuring specific alerts in Grafana, you must set up alert channels. Each channel corresponds to an external service to which Grafana will push alerts.
/login
URL (e.g., http://localhost:3070/-/debug/grafana/login
or append /login
to your Grafana direct access URL if different). If you are doing this for the first time, user the username and password admin
and admin
.Notification channels
.New channel
and then specify the settings of your channel. The Type
field selects the type of external service (e.g., PagerDuty, Slack, Email). Some service types will require additional configuration in the service itself. Here are some examples:https://api.slack.com/apps
to create a new Slack App.Create an App
, give the app a name, and click Create App
.Incoming Webhooks
and toggle on Activate Incoming Webhooks
.Add New Webhook to Workspace
.Url
field.https://app.pagerduty.com/developer/apps
.Create New App
. Give the app a name and decription, and set the category to
"Application Performance Management". For "Would you like to publish the app for all
PagerDuty users?", select "No". Click Save
.Add
.Create
.Save
and then Save
again on the Configure App page.Integration Key
field.Send Test
to send a test notification. You should receive a notification from Grafana via your specified channel. If this worked, click Save
.After adding the appropriate notification channels, configure individual alerts to notify those channels.
Save As..
to create a
writeable copy.Edit
.Test Rule
or viewing State History
.See alert solutions for possible solutions when alerts are firing.