GitHub

Site admins can sync Git repositories hosted on GitHub.com and GitHub Enterprise with Sourcegraph so that users can search and navigate the repositories.

To connect GitHub to Sourcegraph:

Depending on whether you are a site admin or user:
1. Site admin: Go to Site admin > Manage repositories > Add repositories
2. User: Go to Settings > Manage repositories.
Select GitHub.
Configure the connection to GitHub using the action buttons above the text field, and additional fields can be added using Cmd/Ctrl+Space for auto-completion. See the configuration documentation below.
Press Add repositories.

Supported versions

GitHub.com
GitHub Enterprise v2.10 and newer

Selecting repositories for code search

There are four fields for configuring which repositories are mirrored/synchronized:

repos
A list of repositories in owner/name format. The order determines the order in which we sync repository metadata and is safe to change.
orgs
A list of organizations (every repository belonging to the organization will be cloned).
repositoryQuery
A list of strings with three pre-defined options (public, affiliated, none, none of which are subject to result limitations), and/or a GitHub advanced search query. Note: There is an existing limitation that requires the latter, GitHub advanced search queries, to return less than 1000 results. See this issue for ongoing work to address this limitation.
exclude
A list of repositories to exclude which takes precedence over the repos, orgs, and repositoryQuery fields.

Private repositories

A token that has the prerequisite scopes is required in order to clone private repositories for search, as well as at least read access to the relevant private repositories.

See GitHub API token and access for more details.

GitHub API token and access

The GitHub service requires a token in order to access their API. There are two different types of tokens you can supply:

Personal access token:
This gives Sourcegraph the same level of access to repositories as the account that created the token. If you're not wanting to mix your personal repositories with your organizations repositories, you could add an entry to the exclude array, or you can use a machine user token.
Machine user token:
Generates a token for a machine user that is affiliated with an organization instead of a user account.

No token scopes are required if you only want to sync public repositories and don't want to use any of the following features. Otherwise, the following token scopes are required for specific features:

Feature	Required token scopes
Sync private repositories	`repo`
Sync repository permissions	`repo`
Repository permissions caching	`repo`, `write:org`
Batch changes	`repo`, `read:org`, `user:email`, `read:discussion`, and `workflow` (learn more)

WARNING: In addition to the prerequisite token scopes, the account attached to the token must actually have the same level of access to the relevant resources that you are trying to grant. For example:

If read access to repositories is required, the token must have repo scope and the token's account must have read access to the relevant repositories. This can happen by being directly granted read access to repositories, being on a team with read access to the repository, and so on.
If write access to repositories is required, the token must have repo scope and the token's account must have write access to all repositories. This can happen by being added as a direct contributor, being on a team with write access to the repository, being an admin for the repository's organization, and so on.
If write access to organizations is required, the token must have write:org scope and the token's account must have write access for all organizations. This can happen by being an admin in all relevant organizations.

Learn more about how the GitHub API is used and what level of access is required in the corresponding feature documentation.

GitHub.com rate limits

You should always include a token in a configuration for a GitHub.com URL to avoid being denied service by GitHub's unauthenticated rate limits. If you don't want to automatically synchronize repositories from the account associated with your personal access token, you can create a token without a repo scope for the purposes of bypassing rate limit restrictions only.

Internal rate limits

Internal rate limiting can be configured to limit the rate at which requests are made from Sourcegraph to GitHub.

If enabled, the default rate is set at 5000 per hour which can be configured via the requestsPerHour field (see below):

For Sourcegraph <=3.38, if rate limiting is configured more than once for the same code host instance, the most restrictive limit will be used.
For Sourcegraph >=3.39, rate limiting should be enabled and configured for each individual code host connection.

NOTE Internal rate limiting is only currently applied when synchronising changesets in batch changes, repository permissions and repository metadata from code hosts.

Repository permissions

By default, all Sourcegraph users can view all repositories. To configure Sourcegraph to use GitHub's per-user repository permissions, see "Repository permissions".

User authentication

To configure GitHub as an authentication provider (which will enable sign-in via GitHub), see the authentication documentation.

Webhooks

The webhooks setting allows specifying the organization webhook secrets necessary to authenticate incoming webhook requests to /.api/github-webhooks.

"webhooks": [
  {"org": "your_org", "secret": "verylongrandomsecret"}
]

Using webhooks is highly recommended when using batch changes, since they speed up the syncing of pull request data between GitHub and Sourcegraph and make it more efficient.

To set up webhooks:

In Sourcegraph, go to Site admin > Manage code hosts and edit the GitHub configuration.
Add the "webhooks" property to the configuration (you can generate a secret with openssl rand -hex 32):
"webhooks": [{"org": "your_org", "secret": "verylongrandomsecret"}]
Click Update repositories.
Copy the webhook URL displayed below the Update repositories button.
On GitHub, go to the settings page of your organization. From there, click Settings, then Webhooks, then Add webhook.
Fill in the webhook form:
- Payload URL: the URL you copied above from Sourcegraph.
- Content type: this must be set to application/json.
- Secret: the secret token you configured Sourcegraph to use above.
- Which events: select Let me select individual events, and then enable:
  - Issue comments
  - Pull requests
  - Pull request reviews
  - Pull request review comments
  - Check runs
  - Check suites
  - Statuses
- Active: ensure this is enabled.
Click Add webhook.
Confirm that the new webhook is listed.

Done! Sourcegraph will now receive webhook events from GitHub and use them to sync pull request events, used by batch changes, faster and more efficiently.

Configuration

GitHub connections support the following configuration options, which are specified in the JSON editor in the site admin "Manage repositories" area.

`admin/external_service/github.schema.json`

{
	// If non-null, enforces GitHub repository permissions. This requires that there is an item in the [site configuration json](https://docs.sourcegraph.com/admin/config/site_config#auth-providers) `auth.providers` field, of type "github" with the same `url` field as specified in this `GitHubConnection`.
	"authorization": null,

	// TLS certificate of the GitHub Enterprise instance. This is only necessary if the certificate is self-signed or signed by an internal CA. To get the certificate run `openssl s_client -connect HOST:443 -showcerts < /dev/null 2> /dev/null | openssl x509 -outform PEM`. To escape the value into a JSON string, you may want to use a tool like https://json-escape-text.now.sh.
	"certificate": null,
	// Other example values:
	// - "-----BEGIN CERTIFICATE-----\n..."

	// Only used to override the cloud_default column from a config file specified by EXTSVC_CONFIG_FILE
	"cloudDefault": false,

	// When set to true, this external service will be chosen as our 'Global' GitHub service. Only valid on Sourcegraph.com. Only one service can have this flag set.
	"cloudGlobal": false,

	// A list of repositories to never mirror from this GitHub instance. Takes precedence over "orgs", "repos", and "repositoryQuery" configuration.
	//
	// Supports excluding by name ({"name": "owner/name"}) or by ID ({"id": "MDEwOlJlcG9zaXRvcnkxMTczMDM0Mg=="}).
	//
	// Note: ID is the GitHub GraphQL ID, not the GitHub database ID. eg: "curl https://api.github.com/repos/vuejs/vue | jq .node_id"
	"exclude": null,
	// Other example values:
	// - [{"forks":true}]
	// - [
	//     {
	//       "name": "owner/name"
	//     },
	//     {
	//       "id": "MDEwOlJlcG9zaXRvcnkxMTczMDM0Mg=="
	//     }
	//   ]
	// - [
	//     {
	//       "name": "vuejs/vue"
	//     },
	//     {
	//       "name": "php/php-src"
	//     },
	//     {
	//       "pattern": "^topsecretorg/.*"
	//     }
	//   ]

	// The type of Git URLs to use for cloning and fetching Git repositories on this GitHub instance.
	//
	// If "http", Sourcegraph will access GitHub repositories using Git URLs of the form http(s)://github.com/myteam/myproject.git (using https: if the GitHub instance uses HTTPS).
	//
	// If "ssh", Sourcegraph will access GitHub repositories using Git URLs of the form [email protected]:myteam/myproject.git. See the documentation for how to provide SSH private keys and known_hosts: https://docs.sourcegraph.com/admin/repo/auth#repositories-that-need-http-s-or-ssh-authentication.
	"gitURLType": "http",

	// The installation ID of the GitHub App.
	"githubAppInstallationID": null,

	// Deprecated and ignored field which will be removed entirely in the next release. GitHub repositories can no longer be enabled or disabled explicitly. Configure repositories to be mirrored via "repos", "exclude" and "repositoryQuery" instead.
	"initialRepositoryEnablement": null,

	// An array of organization names identifying GitHub organizations whose repositories should be mirrored on Sourcegraph.
	"orgs": null,
	// Other example values:
	// - ["name"]
	// - [
	//     "kubernetes",
	//     "golang",
	//     "facebook"
	//   ]

	// Whether the code host connection is in a pending state.
	"pending": false,

	// Rate limit applied when making background API requests to GitHub.
	"rateLimit": {
		"enabled": true,
		"requestsPerHour": 5000
	},

	// An array of repository "owner/name" strings specifying which GitHub or GitHub Enterprise repositories to mirror on Sourcegraph.
	"repos": null,
	// Other example values:
	// - ["owner/name"]
	// - [
	//     "kubernetes/kubernetes",
	//     "golang/go",
	//     "facebook/react"
	//   ]

	// The pattern used to generate the corresponding Sourcegraph repository name for a GitHub or GitHub Enterprise repository. In the pattern, the variable "{host}" is replaced with the GitHub host (such as github.example.com), and "{nameWithOwner}" is replaced with the GitHub repository's "owner/path" (such as "myorg/myrepo").
	//
	// For example, if your GitHub Enterprise URL is https://github.example.com and your Sourcegraph URL is https://src.example.com, then a repositoryPathPattern of "{host}/{nameWithOwner}" would mean that a GitHub repository at https://github.example.com/myorg/myrepo is available on Sourcegraph at https://src.example.com/github.example.com/myorg/myrepo.
	//
	// It is important that the Sourcegraph repository name generated with this pattern be unique to this code host. If different code hosts generate repository names that collide, Sourcegraph's behavior is undefined.
	"repositoryPathPattern": "{host}/{nameWithOwner}",

	// An array of strings specifying which GitHub or GitHub Enterprise repositories to mirror on Sourcegraph. The valid values are:
	//
	// - `public` mirrors all public repositories for GitHub Enterprise and is the equivalent of `none` for GitHub
	//
	// - `affiliated` mirrors all repositories affiliated with the configured token's user:
	// 	- Private repositories with read access
	// 	- Public repositories owned by the user or their orgs
	// 	- Public repositories with write access
	//
	// - `none` mirrors no repositories (except those specified in the `repos` configuration property or added manually)
	//
	// - All other values are executed as a GitHub advanced repository search as described at https://github.com/search/advanced. Example: to sync all repositories from the "sourcegraph" organization including forks the query would be "org:sourcegraph fork:true".
	//
	// If multiple values are provided, their results are unioned.
	//
	// If you need to narrow the set of mirrored repositories further (and don't want to enumerate it with a list or query set as above), create a new bot/machine user on GitHub or GitHub Enterprise that is only affiliated with the desired repositories.
	"repositoryQuery": [
		"none"
	],

	// A GitHub personal access token. Create one for GitHub.com at https://github.com/settings/tokens/new?description=Sourcegraph (for GitHub Enterprise, replace github.com with your instance's hostname). See https://docs.sourcegraph.com/admin/external_service/github#github-api-token-and-access for which scopes are required for which use cases.
	"token": null,

	// URL of a GitHub instance, such as https://github.com or https://github-enterprise.example.com.
	"url": null,
	// Other example values:
	// - "https://github.com"
	// - "https://github-enterprise.example.com"

	// An array of configurations defining existing GitHub webhooks that send updates back to Sourcegraph.
	"webhooks": null
	// Other example values:
	// - [
	//     {
	//       "org": "yourorgname",
	//       "secret": "webhook-secret"
	//     }
	//   ]
}

Default branch

Sourcegraph displays search results from the default branch of a repository when no revision: parameter is specified. If you'd like the search results to be displayed from another branch by default, you may change a repo's default branch on the github repo settings page. If this is not an option, consider using search contexts instead.

Troubleshooting

Hitting GitHub Search API rate limit with repositoryQuery

When Sourcegraph syncs repositories configured via repositoryQuery, it consumes GitHub API search rate limit, which is lower than the normal rate limit. The affiliated, public and none special values, however, trigger normal API requests instead of search API requests.

When the search rate limit quota is exhausted, an error like failed to list GitHub repositories for search: page=..., searchString=\"...\" can be found in logs. To work around this try reducing the frequency with which repository syncing happens by setting a higher value (in minutes) of repoListUpdateInterval in your Sourcegraph [site config] (https://docs.sourcegraph.com/admin/config/site_config).

repositoryQuery is the only repo syncing method that consumes GitHub search API quota, so if setting repoListUpdateInterval doesn't work consider switching your syncing method to use another option, like orgs, or using one of the special values described above.

"repositoryQuery": ["public"] does not return archived status of a repo

The repositoryQuery option "public" is valuable in that it allows sourcegraph to sync all public repositories, however, it does not return whether or not a repo is archived. This can result in archived repos appearing in normal search. You can see an example of what is returned by the GitHub API for a query to "public" here.

If you would like to sync all public repositories while omitting archived repos, consider generating a GitHub token with access to only public repositories, then use respositoryQuery with option affiliated and an exclude argument with option public as seen in the example below:

{
    "url": "https://github.example.com",
    "gitURLType": "http",
    "repositoryPathPattern": "devs/{nameWithOwner}",
    "repositoryQuery": [
        "affiliated"
    ],
    "token": "TOKEN_WITH_PUBLIC_ACCESS",
    "exclude": [
        {
            "archived": true
        }
    ]
}

repositoryQuery returns first 1000 results only

GitHub's Search API only returns the first 1000 results. Therefore a repositoryQuery (other than the three pre-defined options) needs to return a 1000 results or less otherwise Sourcegraph will not synchronize some repositories. To workaround this limitation you can split your query into multiple queries, each returning less than a 1000 results. For example if your query is org:Microsoft fork:no you can adjust your query to:

{
  // ...
  "repositoryQuery": [
    "org:Microsoft fork:no created:>=2019",
    "org:Microsoft fork:no created:2018",
    "org:Microsoft fork:no created:2016..2017",
    "org:Microsoft fork:no created:<2016"
  ]
}

If splitting by creation date does not work, try another field. See GitHub advanced search query for other fields you can try.

See Handle GitHub repositoryQuery that has more than 1000 results for ongoing work to address this limitation.