Event level data usage pipeline

This document outlines information about the ability to export raw user event data from Sourcegraph. This is limited to certain managed instances (cloud) where the customer has signed a corresponding data collection agreement.

What is it?

This process is a background job that can be enabled that will periodically scrape the event_logs table in the primary database and send it to Sourcegraph centralized analytics. Events stored in event_logs are product events performed by users or the system. More information can be found in RFC 719: Managed Instance Telemetry.

The job interval determines how often the job is executed. The batch size option determines how many records can be pulled in a single scrape. The batch size has a default value and can be overridden with a site setting:

  "exportUsageTelemetry": {
    "batchSize": 100,
  }

The scraping job maintains state using a bookmark stored in the primary postgres database in the table event_logs_scrape_state. If the bookmark is not found, one will be inserted such that the bookmark is the most recent event at the time.

The scraping process has a crude at-least once semantics guarantee. If any scrape should fail, the bookmark state will not be updated causing future scrapes to retry the same set of events.

Allow list

Only events that exist in an allow list will be scraped. Events are keyed in the allow list by the event_logs.name column. The allow list can be found in the primary postgres database in the table event_logs_export_allowlist.

Adding to the allow list

Modifying the allow list is performed using database migrations. To simplify this process and ensure consistency, use the sg tool:

sg telemetry allowlist add --migration [event]

sg telemetry allowlist remove --migration [event]

If you want to modify many events you can pass a file of newline delimited event names using xargs:

cat /location/of/my/events/file | xargs sg telemetry allowlist add --migration

Create a pull request and get a review from the Data Engineering team.

Determine if an event is in the allow list

Currently, there is not a single document that shows the entire allow list. There are two options:

  1. Start Sourcegraph and migrate to the latest version, and query the database
select * from event_logs_export_allowlist;
  1. Look through migration files to see if the event you are looking for has been added and not deleted

How to enable for a managed instance

  1. Ensure the managed instance has the appropriate IAM policy applied
  2. Update the managed instance deployment manifest to include the following environment variables:
    1. EXPORT_USAGE_DATA_ENABLED=true
    2. EXPORT_USAGE_DATA_TOPIC_NAME
    3. EXPORT_USAGE_DATA_TOPIC_PROJECT
  3. Deploy the updated deployment manifest and restart the worker service.

Monitoring

Each Sourcegraph instance with this export job enabled will emit metrics that are prefixed with src_telemetry_job.