Code Insights Pings

Code Insights pings allow us to quantitatively measure the usage and success of Code Insights. This page is a source of truth for detailed explanations, statuses, and implementations of our pings.

We keep this docs page up to date because pings are a vital component of our product knowledge and prioritization process, and a broken or incorrect ping impacts 3-5 months of data (because that's how long a fix takes to propagate).

Terminology

  • FE event - log events that we send by calling standard telemetry service on the frontend. These pings live only in the event_logs table. These typically represent user actions, such as hovers.
  • BE capture - pings that our BE sends to the ping store by checking/selecting data from database tables. Our backend periodically sends these pings to the event_logs table. These typically represent absolute counts across the entire instance.

Metrics

Additions count, edits count, and removals count

Type: FE event

Intended purpose: To track how many times customers have created, edited, and removed insights, by week.

Functional implementation: We track insight creating/editing/deleting events in the creation UI form and insight context menu component with standard telemetry service calls.

Other considerations: N/A

Hovers count

Type: FE event

Intended purpose: To track how many times users hover over a datapoint to see the tooltip on the graph, or "dig in" to the information.

Functional implementation: This ping works by firing an event on the client when a user hovers over a datapoint on a code insight.

Other considerations: N/A

UI customizations count

Type: FE event

Intended purpose: To track how many times users resize the insight graphs.

Functional implementation: This ping works by firing an event on the client when a user resizes a Code Insights graph on the page.

Other considerations: N/A

Data point clicks count

Type: FE event

Intended purpose: To track how many times users click a datapoint to get to a diff search.

Functional implementation: This ping works by firing an event on the client when a user clicks an individual data point of an insight graph, which takes them to a diff search.

Other considerations: Because this functionality does not yet exist for backend insights, it only tracks clicks on frontend insights.

Page views count

Type: FE event

Intended purpose: To track how many times users view insights pages.

Functional implementation: This ping works by firing an event on the client when a user views any /insights page, whether it's creating or viewing insights.

Other considerations: As we add new insights pages it's important to make sure we're adding pages to this counter.

Unique page views count

Type: FE event

Intended purpose: To track how many unique users are viewing insights pages each week.

Functional implementation: This ping works by firing an event on the client when a unique user views any /insights page for the first time that week, whether it's creating or viewing insights.

Other considerations: As we add new insights pages it's important to make sure we're adding pages to this counter.

  • Aggregation: By week
  • Event Code: InsightsUniquePageView
  • PRs: #17805
  • Version Added: 3.25
  • Version(s) broken: 3.25-3.26 (not weekly)(fix PR), 3.30 (broken when switching to dashboard pages, didn't track dashboard views)(fix PR)

Standalone insights page filters edits count

Type: FE event

Intended purpose: To track how many users actively re-filter insights through the standalone insight page's filter panel.

Functional implementation: This ping works by firing a telemetry event on the client when a user changes insights filters (include/exclude repository regexp, search context, etc).

Other considerations: N/A

Standalone insights page dashboard clicks count

Type: FE event

Intended purpose: To track how many users are discovering dashboards from this page.

Functional implementation: This ping works by firing a telemetry event on the client when a user clicks any dashboard pills on the standalone insight page.

Other considerations: N/A

Standalone insights page edit button clicks count

Type: FE event

Intended purpose: To track how many users are going to the edit page through the standalone insight page.

Functional implementation: This ping works by firing a telemetry event on the client when a user clicks on the edit button on the standalone insight page.

Other considerations: N/A

In-product landing page events (hover, data points click, template section clicks)

Type: FE events

Intended purpose: To track unique users' activity on the in-product (get started insights) and the cloud landing pages.

Other considerations: N/A.

Org-visible insights count (Total)

Type: BE capture

Intended purpose: To track how many insights are visible by more than just the creator of the insight.

Functional implementation: we gather this on the backend by joining the insight_view and insight_view_grants tables and counting the insights with org level grants.

Other considerations: N/A

  • Aggregation: total time, by insight type
  • Event Code: InsightOrgVisible
  • PRs: #21671
  • Version Added: 3.29
  • Version(s) broken: 3.31-3.35.0 (doesn't handle backend insights) fix PR

First time insight creators count

Type: FE event

Intended purpose: To track the week and count of the first time a user(s) creates a code insight, of any type, on an instance. The sum of first time insight creators count over all time is equal to the total number of unique creators who have made an insight.

Functional implementation: This metric queries the insight table for new addition events, then filters by unique IDs that appeared for the first time that week.

Other considerations: TODO does this ping include creators who create via the API?

  • Aggregation: By week
  • Event Code: WeeklyFirstTimeInsightCreators
  • Version Added: 3.25
  • Version(s) broken: 3.31-3.35.0 (doesn't handle backend insights, other bugs)

Total count of insights grouped by step size (days)

Type: BE capture

Intended purpose: To track the x-axis (time window) set by users on frontend insights, to help prioritize features related to setting time windows.

Functional implementation: this metric runs on the backend over all the insights.

Other considerations: N/A

Code Insights View/Click Creation Funnels

Type: FE event

Intended purpose: These pings allow us to both understand how the view/click/view/click conversion funnel works for the creation flows of all existing types of insights, as well as smell-check other pings. The reason we use both "view" and "button clicks" in this funnel is that it's possible to view a page without "funneling through" via the prior page's CTA (for example: you can reach the creation/edit screen by the "edit" button, which does not involve logging a click on the "create search insight" button).

Functional implementation: These events fire on the frontend when the user takes the below actions.

Other considerations:

View series counts

Type: BE capture

Intended purpose: To track the number of view series, grouped by presentation type and generation method. Note: a "view series" differs from a "series" by being attached to a particular insight. A series can be attached to more than one insight.

Functional implementation: This is calculated by joining the insight_series, insight_view_series, and insight_view tables.

Other considerations: N/A

Series counts

Type: BE capture

Intended purpose: To track the number of series, grouped by generation method.

Functional implementation: This is calculated using the insight_series table.

Other considerations: N/A

  • Aggregation: total
  • Event Code: SeriesCounts
  • Version added: 3.34

View counts

Type: BE capture

Intended purpose: To track the number of insight views, grouped by presentation type.

Functional implementation: This is calculated using the insight_view table. Unlike critical telemetry which only shows the number of unlocked insights for customers without full access, this ping shows the total number that are locked or unlocked (and may have been created during a trial or free beta phase).

Other considerations: N/A

  • Aggregation: total
  • Event Code: ViewCounts
  • Version added: 3.34

Total orgs with dashboards

Type: BE capture

Intended purpose: To track the number of orgs with at least one dashboard.

Functional implementation: This is calculated using the dashboard_grants table.

Other considerations: N/A

Total dashboard count

Type: BE capture

Intended purpose: To track the total number of dashboards.

Functional implementation: This is calculated using the dashboard table.

Other considerations: N/A

Insights per dashboard

Type: BE capture

Intended purpose: To track statistics (average, min, max, median, std dev,) about how many insights are on each dashboard.

Functional implementation: These are calculated using the dashboard_insight_view table.

Other considerations: N/A

Series backfill time

Type: BE capture

Intended purpose: To track how long on average it takes series to backfill.

Functional implementation: Exposes aggregate information using the backfill times found on insight_series.

Other considerations: N/A

Data export requests

Type: BE capture

Intended purpose: To track usage of data exporting functionality.

Functional implementation: Telemetry events are fired when a request reaches the backend HTTP handler, whether that comes from the webapp or the CLI.

Other considerations: The ping name contains click but this does indeed also record events from the CLI.

  • Aggregation: weekly
  • Event Code: WeeklyDataExportClicks, InsightsDataExportRequest in event_logs
  • Version added: 5.0

Search results aggregations metrics

Information icon hovers

Type: FE event

Intended purpose: To track interest in the feature.

Functional implementation: This ping works by firing a telemetry evente on the client when a user hovers over the information icon.

Type: FE event

Intended purpose: To track how users are using the different aggregation UI modes.

Functional implementation: These pings work by firing telemetry events on the client when a user expands or collapses the sidebar or full view panel.

Other considerations: For the expanded UI mode events we record which aggregation mode was toggled.

Aggregation modes clicks and hovers

Type: FE event

Intended purpose: To track aggregation mode usage and interest.

Functional implementation: These pings work by firing telemetry events on the client when a user clicks on a mode or hovers over a disabled mode.

Other considerations: The ping also includes data for current UI mode.

Aggregation chart clicks and hovers

Type: FE event

Intended purpose: To track if users are hovering over results and clicking through.

Functional implementation: These pings work by firing telemetry events on the client when a user hovers or clicks on a result.

Other considerations: The ping also includes data for the current UI mode and aggregation mode.

Search mode (proactive/extended) success rate

Type: FE event

Intended purpose: To track the number of aggregation searches that succeed or hit limit in either a proactive or extended search.

Functional implementation: These pings fire a telemetry event when an aggregation search completes or times out.