Running in CI

Once you have your tests set up and running locally, you can run them in a CI environment via the same npx autoblocks testing exec command. Test runs in a CI environment can be viewed on the test suites page.

Prerequisites

Make sure you've followed our quick start guide and have your tests running locally.

Setup

Create secrets

You will need to add your Autoblocks API key as a secret in your GitHub repository. Make sure to include any other secrets your tests depend on, such as your OpenAI API key.

You can get your Autoblocks API key from the settings page and follow the instructions here for adding a repository secret.

While you used your local testing API key for running tests from your local machine, you should use your organization's API key (labeled "API Key") for running tests in CI:

Ensure your commit email address is configured

We associate test runs on feature branches with your Autoblocks user based on the commit author's email address. Make sure your commit email address matches your Autoblocks user email address.

You can update your commit address globally:

git config --global user.email "myname@mycompany.com"

Or within a specific repository:

git config user.email "myname@mycompany.com"

Check your existing email address:

git config --global user.email

Or for a particular repository:

git config user.email

Examples

Run on every push

Run your tests on every push to your repository:

name: Autoblocks Tests

on: push

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      # Set up your environment like you would for your
      # unit or integration tests
      ...

      # Run the Autoblocks tests
      - name: Run Autoblocks tests
        run: npx autoblocks testing exec -- <cmd>
        env:
          # This is created automatically by GitHub
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # These were added by you to your repository's secrets
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          AUTOBLOCKS_API_KEY: ${{ secrets.AUTOBLOCKS_API_KEY }}

Why do you need to set the GITHUB_TOKEN as an environment variable?

The CLI uses the automated GITHUB_TOKEN secret to fetch metadata about the run that is not directly available within the run's context.

The GITHUB_TOKEN is created for you automatically by GitHub and is available to use in your workflow file. You do not need to add it to your repository's secrets. Its scope is determined by your workflow's permissions.

Run only when a pull request is opened or updated

Run your tests when a pull request is opened or updated:

name: Autoblocks Tests

on: pull_request

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest
    
    ...

Run on a schedule

Run your tests on a schedule:

name: Autoblocks Tests

on:
  schedule: # Run every day at ~7:17am PST.
    - cron: '17 15 * * *'

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest
    
    ...

Schedule your workflow to run at a minute that isn't 0; this is recommended by the GitHub team to avoid overload and reduce the risk your job is dropped.

Run manually

Run your tests manually via a workflow_dispatch event:

name: Autoblocks Tests

on: workflow_dispatch

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest
    
    ...

See the GitHub documentation on manually running a workflow.

Run only on certain commits

This workflow will run your tests only when a commit contains a certain string. For example, you can run your tests only when a commit message contains [autoblocks]:

name: Autoblocks Tests

# This workflow will run on every push, but the job will not run unless
# the string "[autoblocks]" is somewhere in the commit message
on: push

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    if: contains(github.event.head_commit.message, '[autoblocks]')
    
    ...

Or, if you are triggering on pull requests:

name: Autoblocks Tests

# This workflow will run on every pull request update, but the job will not run unless
# the string "[autoblocks]" is somewhere in the pull request title
on: pull_request

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    if: contains(github.event.pull_request.title, '[autoblocks]')
    
    ...

Run in multiple scenarios

You can use a combination of all of the above to run your tests in multiple scenarios:

name: Autoblocks Tests

on:
  # Run on every push when the commit message contains "[autoblocks]".
  # See below for the if condition that enforces this.
  push:

  # Run manually.
  workflow_dispatch:

  # Run every day at ~7:17am PST.
  schedule: 
    - cron: '17 15 * * *'

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    # For push events, check the commit message contains "[autoblocks]".
    if: github.event_name != 'push' || contains(github.event.head_commit.message, '[autoblocks]')
    
    ...

More examples

Check out some end to end examples in our examples repository:

Python:
- Project
- Workflow
TypeScript:
- Project
- Workflow

Slack Notifications

You can receive Slack notifications with test results by setting the AUTOBLOCKS_SLACK_WEBHOOK_URL environment variable. To get set up:

Create a webhook URL for the channel where you want results to be posted by following Slack's instructions on setting up incoming webhooks
Add this webhook URL to your repository as a secret called AUTOBLOCKS_SLACK_WEBHOOK_URL in addition to the secrets you already added in the setup section

Then, set the environment variable in your workflow:

name: Autoblocks Tests

on: push

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    steps:
      ...

      - name: Run Autoblocks tests
        run: npx autoblocks testing exec -- <cmd>
        env:
          # This is created automatically by GitHub
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # These were added by you to your repository's secrets
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          AUTOBLOCKS_API_KEY: ${{ secrets.AUTOBLOCKS_API_KEY }}
          # Posts a message to a Slack channel with the test results
          AUTOBLOCKS_SLACK_WEBHOOK_URL: ${{ secrets.AUTOBLOCKS_SLACK_WEBHOOK_URL }}

If you have multiple triggers for your workflow and only want to receive Slack notifications for some of them, you can add a step that only sets the environment variable for certain event types:

name: Autoblocks Tests

on:
  # This workflow has multiple triggers
  push:
  workflow_dispatch:
  schedule: 
    - cron: '17 15 * * *'

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    steps:
      ...

      # But we only want to send Slack notifications for scheduled runs
      - name: Set Autoblocks Slack webhook URL
        if: github.event_name == 'schedule'
        run: echo "AUTOBLOCKS_SLACK_WEBHOOK_URL=${{ secrets.AUTOBLOCKS_SLACK_WEBHOOK_URL }}" >> $GITHUB_ENV
      
      - name: Run Autoblocks tests
        run: npx autoblocks testing exec -- <cmd>
        env:
          # This is created automatically by GitHub
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # These were added by you to your repository's secrets
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          AUTOBLOCKS_API_KEY: ${{ secrets.AUTOBLOCKS_API_KEY }}
          # The AUTOBLOCKS_SLACK_WEBHOOK_URL has already been set
          # above, so it doesn't need to be set here

GitHub Comments

A summary of test results is posted automatically as a comment on the pull request associated with the commit that triggered the test run. If a commit is pushed but a pull request hasn't been opened yet, the summary is posted on the commit instead.

Autoblocks updates the same comment with the latest test results each time the workflow runs. This prevents the comment section from becoming cluttered.

GitHub Actions Permissions (Advanced)

If your GitHub organization has set the default permissions for GITHUB_TOKEN to be restricted, you will need to explicitly give the workflow permission to post comments on pull requests or commits.

Commenting on a pull request from a pull_request event:

name: Autoblocks Tests

on: pull_request

permissions:
  # Allow commenting on pull requests
  issues: write

Commenting on a pull request from a push event:

name: Autoblocks Tests

on: push

permissions:
  # Allow commenting on pull requests
  issues: write
  # Allow getting associated pull requests when event is type `push`
  pull-requests: read

Other CI Providers

Using a CI provider other than GitHub Actions? Contact us at support@autoblocks.ai and we'll help get you set up.