Deploying Sourcegraph executors

Executors provide a sandbox that can run resource-intensive or untrusted tasks on behalf of the Sourcegraph instance, such as:

• Automatically indexing a repository for precise code navigation
• Running batch changes

Requirements

Executors by default use KVM-based micro VMs powered by Firecracker in accordance with our sandboxing model to isolate jobs from each other and the host. This requires executors to be run on machines capable of running Linux KVM extensions. On the most popular cloud providers, this either means running executors on bare-metal machines (AWS) or machines capable of nested virtualization (GCP).

Optionally, executors can be run without using KVM-based isolation, which is less secure but might be easier to run on common machines.

Resource Recommendations

It is recommended to set the resources based on the number of jobs an instance will process in parallel.

A single job should have the following resources available.

• CPU: 4
• Memory: 12GB
• Disk: 20GB

So, if you expect an Executor instance to process up to 4 jobs in parallel, the recommended resources for the machine are

• CPU: 16
• Memory: 48GB
• Disk: 80GB

The above recommended resources can be changed to fit your constraints. See below for configuring resources for a Job.

_{Note: the smallest machine type on AWS that can support Executors with Firecracker is c5n.metal (72 vCPU and 192GB of Memory), but concurrency can be turned up for the additional cost.}

Job Configuration

The maximum number of Jobs an Executor instance can run in parallel is configured by the Environment Variable EXECUTOR_MAXIMUM_NUM_JOBS.

The CPU and Memory usage of an individual Job is configured by the Environment Variables EXECUTOR_JOB_NUM_CPUS and EXECUTOR_JOB_MEMORY.

See Environment Variables for additional Environment Variables.

_{Note: changing CPU and Memory for jobs will affect the overall requirements for an Executor instance.}

AWS

It is recommended to add the following Disk configuration in AWS.

• IOPS: Equal to the Disk Size (so if Disk is 100GB, then IOPS is 100)
• Throughput: 125MiB/s

Supported Infrastructures

• Operating System: Linux-based
• Architecture: AMD64

Firecracker Requirements

To run Executors with Firecracker enabled requires the machine to support Kernel-based Virtual Machine. See deploying Executors binary for additional information on configuring Linux Machines.

Cloud Providers

Machines on Cloud Providers have additional constraints for use with firecracker.

• AWS: machine type must be a metal instance (.metal)
• GCP: the instance must enable nested virtualization

Configure Sourcegraph

Executors must be run separately from your Sourcegraph instance.

Since they must still be able to reach the Sourcegraph instance in order to dequeue and perform work, requests between the Sourcegraph instance and the executors are authenticated via a shared secret.

Before starting any executors, generate an arbitrary secret string (with at least 20 characters) and set it as the executors.accessToken key in your Sourcegraph instance's site-config.

Executor installation

Once the shared secret is set in Sourcegraph, you can start setting up executors that can use that access token to talk to the Sourcegraph instance.

Supported installation types

Confirm executors are working

If executor instances boot correctly and can authenticate with the Sourcegraph frontend, they will show up in the Executors page under Site admin > Maintenance.

Using private registries

If you want to use docker images stored in a private registry that requires authentication, follow this section to configure it.

Depending on the executor runtime that is being used, different options exist for provisioning access to private container registries:

• Through a special secret called DOCKER_AUTH_CONFIG, set in executor secrets in Sourcegraph.
• Through the EXECUTOR_DOCKER_AUTH_CONFIG environment variable (also available as a variable in the terraform modules for executors).
• Through the config.json file in ~/.docker. If using executors with firecracker enabled (recommended) this option is not available.

When multiple of the above options are combined, executors will use them in the following order:

• If a DOCKER_AUTH_CONFIG executor secret is configured, that will be preferred. That is so that users can overwrite the credentials being used in their user-settings. This is the only option available in Sourcegraph Cloud.
• If the EXECUTOR_DOCKER_AUTH_CONFIG environment variable is set, this will be used as the next option.
• Finally, if neither of the above are set, executors will fall back to the config.json file in the user home directory of the user that is owning the executor process. NOTE: This is not available in the firecracker runtime, as the rootfs is not shared with the host.

The docker CLI supports three ways to use credentials:

• Using static credentials
• Using credential helpers
• Using credential stores

Credential helpers and credential stores are only available for use with the config.json configuration option, as they require additional infrastructural changes. Thus, those options are not available on Sourcegraph Cloud.

Using static credentials

The EXECUTOR_DOCKER_AUTH_CONFIG environment variable and the DOCKER_AUTH_CONFIG secret expect a docker config with only the necessary properties set for configuring authentication. The format of this config supports multiple registries to be configured and looks like this:

{
  "auths": {
    "myregistry.example.com[:port]": {
      "auth": "base64(username:password)"
    },
    "myregistry2.example.com[:port]": {
      "auth": "base64(username:password)"
    }
  }
}

You can either create this config yourself by hand, or let docker do it for you by running:

TMP_FILE="$(mktemp -d)" bash -c 'echo "<password>" | docker --config "${TMP_FILE}" login --username "<username>" --password-stdin "<registryurl>" && cat "${TMP_FILE}/config.json" && rm -rf "${TMP_FILE}"'

You can also run the following:

echo "username:password" | base64

and then paste the result of that into a JSON string like this:

{
  "auths": {
    "myregistry.example.com[:port]": {
      "auth": "<the value from above>"
    }
  }
}

For Google Container Registry, follow this guide for how to obtain long-lived static credentials.

Configuring the auth config for use in executors

Now that the config has been obtained, it can be used for the EXECUTOR_DOCKER_AUTH_CONFIG environment variable (and terraform variable docker_auth_config) or you can create an executor secret called DOCKER_AUTH_CONFIG. Global executor secrets will be available to every execution, while user and organization level executor secrets will only be available to the namespaces executions.

Using custom certificates with executors

By default, executors will search for certificates in the following files and directories:

Directory or file	Distribution
/etc/ssl/certs/ca-certificates.crt	Debian/Ubuntu/Gentoo etc.
/etc/pki/tls/certs/ca-bundle.crt	Fedora/RHEL 6
/etc/ssl/ca-bundle.pem	OpenSUSE
/etc/pki/tls/cacert.pem	OpenELEC
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem	CentOS/RHEL 7
/etc/ssl/cert.pem	Alpine Linux
/etc/ssl/certs	SLES10/SLES11
/etc/pki/tls/certs	Fedora/RHEL
/system/etc/security/cacerts	Android

If your environment makes use of custom certificates, you can add them to one of these locations in order for executors to pick them up.

Adding certificates to a binary deployment

After successfully deploying binaries, follow these steps:

1. Copy your certificates to /etc/ssl/certs.
2. If you are using systemd, run systemctl restart executor. If not, proceed to the next step.
3. Run executor run on the VM in order to restart the executor service.

Adding certificates with Firecracker

When running executors with the firecracker runtime, custom certificates need to be added in the container that is running within the Firecracker VM. To add custom certificates, you must create a new Docker image that contains the certificates. For example,

FROM upstream:tag

# Copy the certificates into the container
COPY customcert.crt /usr/local/share/ca-certificates/customcert.crt
# Update the certificate store
RUN chmod 644 /usr/local/share/ca-certificates/customcert.crt && update-ca-certificates
# ...

Code Intel

Once the custom image is built, you can configure the executor to use it by setting the codeIntelAutoIndexing.indexerMap to use the custom image. For example,

"codeIntelAutoIndexing.indexerMap": {
  "go": "myregistry.company.com/scip-go:custom"
}

Adding certificates to a Kubernetes deployment using manifests

First, add the certificate data as a secret in your preferred namespace:

SECRET_NAME=custom-certs
CERT_PATH=/path/to/cert.pem 
kubectl create secret generic $SECRET_NAME --from-file=customcert.crt=$CERT_PATH

Or as a declarative manifest:

apiVersion: v1
kind: Secret
metadata:
  name: custom-certs
data:
  customcert.crt: $(base64 -i /path/to/cert.pem)
type: Opaque

Next, mount the secret in the executor deployment. Add the following snippet to spec.template.spec.volumes of each executor deployment:

- name: custom-certs
  secret:
    secretName: custom-certs

Also add this snippet to spec.template.spec.containers.volumeMounts of each executor deployment (specifically, the executor container, in case you inject any sidecars):

- mountPath: /etc/ssl/certs
  name: custom-certs
  readOnly: true

Next, apply the updated YAML manifests. Once the executors have rolled out, they should be picking up your custom certificates.

Adding certificates to a Kubernetes deployment using Helm

You may follow the same instructions for the manifest deployment to set custom certificates.

Adding certificates to a Docker Compose deployment

First, ensure that the certificate file is present on the host machine. Next, add the volume to the executor compose file:

- '/path/to/certs:/etc/ssl/certs'

Next, restart the deployment with docker-compose down and docker-compose up -d.