Running in CI

Once you have your tests set up and running locally, you can run them in a CI environment via the same npx autoblocks testing exec command. Test runs in a CI environment can be viewed on the test suites page.

Prerequisites

Make sure you've followed our quick start guide and have your tests running locally.

Setup

Create secrets

You will need to add your Autoblocks API key as a secret in your GitHub repository. Make sure to include any other secrets your tests depend on, such as your OpenAI API key.

You can get your Autoblocks API key from the settings page and follow the instructions here for adding a repository secret.

Ensure your commit email address is configured

We associate test runs on feature branches with your Autoblocks user based on the commit author's email address. Make sure your commit email address matches your Autoblocks user email address.

You can update your commit address globally:

git config --global user.email "myname@mycompany.com"

Or within a specific repository:

git config user.email "myname@mycompany.com"

Examples

Run on every push

Run your tests on every push to your repository:

name: Autoblocks Tests

on: push

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5

      - name: Install Poetry
        run: curl -sSL https://install.python-poetry.org | python3 -

      - name: Install dependencies
        run: poetry install

      - name: Run Autoblocks tests
        run: npx autoblocks testing exec -- poetry run start
        env:
          # This is created automatically by GitHub
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # These were added by you to your repository's secrets
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          AUTOBLOCKS_API_KEY: ${{ secrets.AUTOBLOCKS_API_KEY }}

Run only when a pull request is opened or updated

Run your tests when a pull request is opened or updated:

name: Autoblocks Tests

on: pull_request

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest
    
    ...

Run on a schedule

Run your tests on a schedule:

name: Autoblocks Tests

on:
  schedule: # Run every day at ~7:17am PST.
    - cron: '17 15 * * *'

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest
    
    ...

Run manually

Run your tests manually via a workflow_dispatch event:

name: Autoblocks Tests

on: workflow_dispatch

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest
    
    ...

See the GitHub documentation on manually running a workflow.

Run only on certain commits

This workflow will run your tests only when a commit contains a certain string. For example, you can run your tests only when a commit message contains [autoblocks]:

name: Autoblocks Tests

# This workflow will run on every push, but the job will not run unless
# the string "[autoblocks]" is somewhere in the commit message
on: push

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    if: contains(github.event.head_commit.message, '[autoblocks]')
    
    ...

Or, if you are triggering on pull requests:

name: Autoblocks Tests

# This workflow will run on every pull request update, but the job will not run unless
# the string "[autoblocks]" is somewhere in the pull request title
on: pull_request

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    if: contains(github.event.pull_request.title, '[autoblocks]')
    
    ...

Run in multiple scenarios

You can use a combination of all of the above to run your tests in multiple scenarios:

name: Autoblocks Tests

on:
  # Run on every push when the commit message contains "[autoblocks]".
  # See below for the if condition that enforces this.
  push:

  # Run manually.
  workflow_dispatch:

  # Run every day at ~7:17am PST.
  schedule: 
    - cron: '17 15 * * *'

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    # For push events, check the commit message contains "[autoblocks]".
    if: github.event_name != 'push' || contains(github.event.head_commit.message, '[autoblocks]')
    
    ...

More examples

Check out some end to end examples in our examples repository:

Slack Notifications

You can receive Slack notifications with test results by setting the AUTOBLOCKS_SLACK_WEBHOOK_URL environment variable. Create a webhook URL for the channel where you want results to be posted by following Slack's instructions on setting up incoming webhooks. Add this webhook URL to your repository as a secret called AUTOBLOCKS_SLACK_WEBHOOK_URL in addition to the secrets you already added in the setup section.

Then, set the environment variable in your workflow:

name: Autoblocks Tests

on: push

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    steps:
      ...

      - name: Run Autoblocks tests
        run: npx autoblocks testing exec -- poetry run start
        env:
          # This is created automatically by GitHub
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # These were added by you to your repository's secrets
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          AUTOBLOCKS_API_KEY: ${{ secrets.AUTOBLOCKS_API_KEY }}
          # Posts a message to a Slack channel with the test results
          AUTOBLOCKS_SLACK_WEBHOOK_URL: ${{ secrets.AUTOBLOCKS_SLACK_WEBHOOK_URL }}

If you have multiple triggers for your workflow and only want to receive Slack notifications for some of them, you can add a step that only sets the environment variable for certain event types:

name: Autoblocks Tests

on:
  # This workflow has multiple triggers
  push:
  workflow_dispatch:
  schedule: 
    - cron: '17 15 * * *'

jobs:
  autoblocks-tests:
    runs-on: ubuntu-latest

    steps:
      ...

      # But we only want to send Slack notifications for scheduled runs
      - name: Set Autoblocks Slack webhook URL
        if: github.event_name == 'schedule'
        run: echo "AUTOBLOCKS_SLACK_WEBHOOK_URL=${{ secrets.AUTOBLOCKS_SLACK_WEBHOOK_URL }}" >> $GITHUB_ENV
      
      - name: Run Autoblocks tests
        run: npx autoblocks testing exec -- poetry run start
        env:
          # This is created automatically by GitHub
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # These were added by you to your repository's secrets
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          AUTOBLOCKS_API_KEY: ${{ secrets.AUTOBLOCKS_API_KEY }}
          # The AUTOBLOCKS_SLACK_WEBHOOK_URL has already been set
          # above, so it doesn't need to be set here

Other CI Providers

Using a CI provider other than GitHub Actions? Contact us at support@autoblocks.ai and we'll help get you set up.