Batch spec templating

Overview

Certain fields in a batch spec YAML support templating to create even more powerful and performant batch changes.

Templating in a batch spec uses the delimiters ${{ and }}. Inside the delimiters, template variables and template helper functions may be used to produce a text value.

Example batch spec

Here is an excerpt of a batch spec that uses templating:

on:
  - repositoriesMatchingQuery: lang:go fmt.Sprintf("%d", :[v]) patterntype:structural -file:vendor

steps:
  - run: comby -in-place 'fmt.Sprintf("%d", :[v])' 'strconv.Itoa(:[v])' ${{ join repository.search_result_paths " " }}
    #                                                                   ^ templating starts here
    container: comby/comby
  - run: goimports -w ${{ join previous_step.modified_files " " }}
    #                 ^ templating starts here
    container: unibeautify/goimports

Before executing the first run command, repository.search_result_paths will be replaced with the relative-to-root-dir file paths of each search result yielded by repositoriesMatchingQuery. By using the template helper function join, an argument list of whitespace-separated values is constructed.

The final run value, that will be executed, will look similar to this:

run: comby -in-place 'fmt.Sprintf("%d", :[v])' 'strconv.Itoa(:[v])' cmd/src/main.go internal/fmt/fmt.go

The result is that comby only search and replaces in those files, instead of having to search through the complete repository.

Before the second step is executed previous_step.modified_files will be replaced with the list of files that the previous comby step modified. It will look similar to this:

run: goimports -w cmd/src/main.go internal/fmt/fmt.go

See "Examples" for more examples of how to use and leverage templating in batch specs.

Fields with template support

Templating is supported in the following fields:

steps.run
steps.env values
steps.files values
steps.outputs.steps.outputs.<name>.value
steps.if

Additionally, with Sourcegraph 3.24 and Sourcegraph CLI 3.24 or later:

Template variables

Template variables are the names that are defined and accessible when using templating syntax in a given context.

Depending on the context in which templating is used, different variables are available.

For example: in the context of steps the template variable previous_step is available, but not in the context of changesetTemplate.

`steps` context

The following template variables are available in the fields under steps.

They are evaluated before the execution of each entry in steps, except for the step.* variables, which only contain values after the step has executed.

Template variable	Type	Description
`batch_change.name`	`string`	The `name` of the batch change, as set in the batch spec. Requires Sourcegraph CLI 3.26 or later.
`batch_change.description`	`string`	The `description` of the batch change, as set in the batch spec. Requires Sourcegraph CLI 3.26 or later.
`repository.search_result_paths`	`list of strings`	Unique list of file paths relative to the repository root directory in which the search results of the `repositoriesMatchingQuery`s have been found.
`repository.branch`	`string`	The target branch of the repository in which the step is being executed. Requires Sourcegraph 3.35 or later.
`repository.name`	`string`	Full name of the repository in which the step is being executed.
`previous_step.modified_files`	`list of strings`	List of files that have been modified by the previous step in `steps`. Empty list if no files have been modified.
`previous_step.added_files`	`list of strings`	List of files that have been added by the previous step in `steps`. Empty list if no files have been added.
`previous_step.deleted_files`	`list of strings`	List of files that have been deleted by the previous step in `steps`. Empty list if no files have been deleted.
`previous_step.stdout`	`string`	The complete output of the previous step on standard output.
`previous_step.stderr`	`string`	The complete output of the previous step on standard error.
`step.modified_files`	`list of strings`	Only in `steps.outputs`: List of files that have been modified by the just-executed step. Empty list if no files have been modified. Requires Sourcegraph 3.24 and Sourcegraph CLI 3.24 or later.
`step.added_files`	`list of strings`	Only in `steps.outputs`: List of files that have been added by the just-executed step. Empty list if no files have been added. Requires Sourcegraph 3.24 and Sourcegraph CLI 3.24 or later.
`step.deleted_files`	`list of strings`	Only in `steps.outputs`: List of files that have been deleted by the just-executed step. Empty list if no files have been deleted. Requires Sourcegraph 3.24 and Sourcegraph CLI 3.24 or later.
`step.stdout`	`string`	Only in `steps.outputs`: The complete output of the just-executed step on standard output. Requires Sourcegraph 3.24 and Sourcegraph CLI 3.24 or later.
`step.stderr`	`string`	Only in `steps.outputs`: The complete output of the just-executed step on standard error. Requires Sourcegraph 3.24 and Sourcegraph CLI 3.24 or later.
`steps.modified_files`	`list of strings`	List of files that have been modified by the `steps`. Empty list if no files have been modified. Requires Sourcegraph CLI 3.28 or later.
`steps.added_files`	`list of strings`	List of files that have been added by the `steps`. Empty list if no files have been added. Requires Sourcegraph CLI 3.28 or later.
`steps.deleted_files`	`list of strings`	List of files that have been deleted by the `steps`. Empty list if no files have been deleted. Requires Sourcegraph CLI 3.28 or later.
`steps.path`	`string`	Path (relative to the root of the directory, no leading `/` or `.`) in which the `steps` have been executed. Empty if no workspaces have been used and the `steps` were executed in the root of the repository. Requires Sourcegraph CLI 3.28 or later.

`changesetTemplate` context

The following template variables are available in the fields under changesetTemplate.

They are evaluated after the execution of all entries in steps.

Template variable	Type	Description
`batch_change.name`	`string`	The `name` of the batch change, as set in the batch spec. Requires Sourcegraph CLI 3.26 or later.
`batch_change.description`	`string`	The `description` of the batch change, as set in the batch spec. Requires Sourcegraph CLI 3.26 or later.
`repository.search_result_paths`	`list of strings`	Unique list of file paths relative to the repository root directory in which the search results of the `repositoriesMatchingQuery`s have been found.
`repository.branch`	`string`	The target branch of the repository in which the step is being executed. Requires Sourcegraph 3.35 or later.
`repository.name`	`string`	Full name of the repository in which the step is being executed.
`steps.modified_files`	`list of strings`	List of files that have been modified by the `steps`. Empty list if no files have been modified.
`steps.added_files`	`list of strings`	List of files that have been added by the `steps`. Empty list if no files have been added.
`steps.deleted_files`	`list of strings`	List of files that have been deleted by the `steps`. Empty list if no files have been deleted.
`steps.path`	`string`	Path (relative to the root of the directory, no leading `/` or `.`) in which the `steps` have been executed. Empty if no workspaces have been used and the `steps` were executed in the root of the repository. Requires Sourcegraph CLI 3.25 or later
`outputs.<name>`	depends on `outputs.<name>.format`, default: `string`	Value of an `output` set by `steps`. If the outputs.`outputs.<name>.format` is `yaml` or `json` and the `value` a data structure (i.e. array, object, ...), then subfields can be accessed too. See "Examples" below.

Template helper functions

${{ join repository.search_result_paths "\n" }} - joins the list of strings given as first argument with the separator as last argument.
${{ join_if "---" "a" "b" "" "d" }} - uses the first argument as separator to join the remaining arguments, ignoring blank strings.
${{ replace "a/b/c/d" "/" "-" }} - replaces occurrences of second argument in the first one with the last one.
${{ split repository.name "/" }} - splits the first argument into a list of strings at each occurrence of the last argument.
${{ matches repository.name "github.com/my-org/terra*" }} - matches the first argument against the glob pattern in the second argument, returning true/false.
${{ "${{ repository.name }}" }} - outputs the inner expression as a literal string, for example, to ignore the inner set of ${{ }}

The features of Go's text/template package are also available, including conditionals and loops, since it is the underlying templating engine.

Examples

Pass the exact list of search result file paths to a command:

steps:
  - run: comby -in-place -config /tmp/go-sprintf.toml -f ${{ join repository.search_result_paths "," }}
    container: comby/comby
    files:
      /tmp/go-sprintf.toml: |
        [sprintf_to_strconv]
        match='fmt.Sprintf("%d", :[v])'
        rewrite='strconv.Itoa(:[v])'

Run a command for each search result file path:

steps:
  - run: |
      for file in "${{repository.search_result_paths}}";
      do
        sed -i 's/mydockerhub-user/ci-dockerhub-user/g;' ${file}
      done      
    container: alpine:3

Format and fix files after a previous step modified them:

steps:
  - run: |
      find . -type f -name '*.go' -not -path "*/vendor/*" |\
      xargs sed -i 's/fmt.Println/log.Println/g'      
    container: alpine:3
  - run: goimports -w ${{ join previous_step.modified_files " " }}
    container: unibeautify/goimports

Use the steps.files combined with template variables to construct files inside the container:

steps:
  - run: |
      cat /tmp/search-results | while read file;
      do
        ruplacer --subvert whitelist allowlist --go ${file} || echo "nothing to replace";
        ruplacer --subvert blacklist denylist --go ${file} || echo "nothing to replace";
      done      
    container: ruplacer
    files:
      /tmp/search-results: ${{ join repository.search_result_paths "\n" }}

Put information in environment variables, based on the output of previous step steps.env also

steps:
  - run: echo $LINTER_ERRROS >> linter_errors.txt
    container: alpine:3
    env:
      LINTER_ERRORS: ${{ previous_step.stdout }}

If you need to escape the ${{ and }} delimiters you can simply render them as string literals:

steps:
  - run: cp /tmp/escaped.txt .
    container: alpine:3
    files:
      /tmp/escaped.txt: ${{ "${{" }} ${{ "}}" }}

Accessing the outputs set by steps in subsequent steps and the changesetTemplate:

steps:
  - run: echo "Hello there!"
    container: alpine:3
    outputs:
      myFriendlyMessage:
        value: "${{ step.stdout }}"
  - run: echo "We have access to the output here: ${{ outputs.myFriendlyMessage }}"
    container: alpine:3
    outputs:
      stepTwoOutput:
        otherMessage: "here too: ${{ outputs.myFriendlyMessage }}"

changesetTemplate:
  # [...]
  body: |
    The first step left us the following message: ${{ outputs.myFriendlyMessage }}
    The second step left this one: ${{ outputs.otherMessage }}

Using the steps.outputs.steps.outputs.<name>.format field, it's possible to parse the value of an output as JSON or YAML and access it as a data structure instead of just text:

steps:
  - run: cat .goreleaser.yml
    container: alpine:3
    outputs:
      goreleaserConfig:
        value: "${{ step.stdout }}"
        # The step's output is parsed as YAML, making it accessible as a YAML
        # object in the other templating fields.
        format: yaml
      goreleaserConfigExists:
        # We can use the power of Go's text/template engine to dynamically produce complex values
        value: "exists: ${{ gt (len step.stderr) 0 }}"
        format: yaml

changesetTemplate:
  # [...]

  # Since templating fields use Go's `text/template` and `goreleaserConfig` was
  # parsed as YAML we can iterate over every field:
  body: |
    This repository has a `gorelaserConfig`: ${{ outputs.goreleaserConfigExists.exists }}.

    The `goreleaser.yml` defines the following `before.hooks`:

    ${{ range $index, $hook := outputs.goreleaserConfig.before.hooks }}
    - `${{ $hook }}`
    ${{ end }}

Using the steps.if field to conditionally execute different steps in different repositories:

steps:
  # `if:` is true, step always executes.
  - if: true
    run: echo "name of repository is ${{ repository.name }}" >> message.txt
    container: alpine:3

  # `if:` checks for repository name. Only runs in github.com/sourcegraph/automation-testing
  - if: ${{ eq repository.name "github.com/sourcegraph/automation-testing" }}
    run: echo "hello from automation-testing" >> message.txt
    container: alpine:3

  # `if:` uses glob pattern to match repository name.
  - if: ${{ matches repository.name "*sourcegraph-testing*" }}
    run: echo "name contains sourcegraph-testing" >> message.txt
    container: alpine:3

  # Checks for go.mod existence and saves to outputs
  - run:  if [[ -f "go.mod" ]]; then echo "true"; else echo "false"; fi
    container: alpine:3
    outputs:
      goModExists:
        value: ${{ step.stdout }}

  # `if:` uses the just-set `outputs.goModExists` value as condition
  - if: ${{ outputs.goModExists }}
    run: go fmt ./...
    container: golang

  # `if:` checks for path, in case steps are executed in workspace.
  - if: ${{ eq steps.path "sub/directory/in/repo" }}
    run: echo "hello workspace" >> workspace.txt
    container: golang

Combine the template helper functions with the helper functions built into Go's text/template library:

changesetTemplate:
  # [...]
  body: |
    The host of the repository: ${{ index (split repository.name "/") 0 }}
    The org of the repository: ${{ index (split repository.name "/") 1 }}