Prerequisites
Make sure you’ve followed our quick start guide and have your tests running locally.
Once you have your tests set up and running locally,
you can run them in a CI environment via the same npx autoblocks testing exec command.
Setup
Create secrets
You will need to add your Autoblocks API key as a secret in your GitHub repository.
Make sure to include any other secrets your tests depend on, such as your OpenAI API key.
You can get your Autoblocks API key from the settings
page and follow the instructions here
for adding a repository secret.
We associate test runs on feature branches with your Autoblocks user based on the commit author’s email address.
Make sure your commit email address matches your Autoblocks user email address.
You can update your commit address globally:
git config --global user.email "myname@mycompany.com"
git config user.email "myname@mycompany.com"
Examples
Run on every push
Run your tests on every push to your repository:
name: Autoblocks Tests
on: push
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
# Set up your environment like you would for your
# unit or integration tests
...
# Run the Autoblocks tests
- name: Run Autoblocks tests
run: npx autoblocks testing exec -- <cmd>
env:
# This is created automatically by GitHub
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# These were added by you to your repository's secrets
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
AUTOBLOCKS_V2_API_KEY: ${{ secrets.AUTOBLOCKS_V2_API_KEY }}
Why do you need to set the GITHUB_TOKEN as an environment variable?The CLI uses the automated GITHUB_TOKEN secret
to fetch metadata about the run that is not directly available within the run’s context.The GITHUB_TOKEN is created for you automatically by GitHub and is available to use in your workflow file.
You do not need to add it to your repository’s secrets.
Its scope is determined by your workflow’s permissions. Run only when a pull request is opened or updated
Run your tests when a pull request is opened or updated:
name: Autoblocks Tests
on: pull_request
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
...
Run on a schedule
Run your tests on a schedule:
name: Autoblocks Tests
on:
schedule: # Run every day at ~7:17am PST.
- cron: '17 15 * * *'
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
...
Run manually
Run your tests manually via a workflow_dispatch event:
name: Autoblocks Tests
on: workflow_dispatch
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
...
Run only on certain commits
This workflow will run your tests only when a commit contains a certain string.
For example, you can run your tests only when a commit message contains [autoblocks]:
name: Autoblocks Tests
# This workflow will run on every push, but the job will not run unless
# the string "[autoblocks]" is somewhere in the commit message
on: push
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
if: contains(github.event.head_commit.message, '[autoblocks]')
...
name: Autoblocks Tests
# This workflow will run on every pull request update, but the job will not run unless
# the string "[autoblocks]" is somewhere in the pull request title
on: pull_request
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
if: contains(github.event.pull_request.title, '[autoblocks]')
...
Run in multiple scenarios
You can use a combination of all of the above to run your tests in multiple scenarios:
name: Autoblocks Tests
on:
# Run on every push when the commit message contains "[autoblocks]".
# See below for the if condition that enforces this.
push:
# Run manually.
workflow_dispatch:
# Run every day at ~7:17am PST.
schedule:
- cron: '17 15 * * *'
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
# For push events, check the commit message contains "[autoblocks]".
if: github.event_name != 'push' || contains(github.event.head_commit.message, '[autoblocks]')
...
More examples
Check out some end to end examples in our examples repository:
Slack Notifications
You can receive Slack notifications with test results by setting the AUTOBLOCKS_SLACK_WEBHOOK_URL environment variable. To get set up:
- Create a webhook URL for the channel where you want results to be posted by following Slack’s instructions on setting up incoming webhooks
- Add this webhook URL to your repository as a secret called
AUTOBLOCKS_SLACK_WEBHOOK_URL
in addition to the secrets you already added in the setup section
Then, set the environment variable in your workflow:
name: Autoblocks Tests
on: push
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
steps:
...
- name: Run Autoblocks tests
run: npx autoblocks testing exec -- <cmd>
env:
# This is created automatically by GitHub
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# These were added by you to your repository's secrets
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
AUTOBLOCKS_V2_API_KEY: ${{ secrets.AUTOBLOCKS_V2_API_KEY }}
# Posts a message to a Slack channel with the test results
AUTOBLOCKS_SLACK_WEBHOOK_URL: ${{ secrets.AUTOBLOCKS_SLACK_WEBHOOK_URL }}
name: Autoblocks Tests
on:
# This workflow has multiple triggers
push:
workflow_dispatch:
schedule:
- cron: '17 15 * * *'
jobs:
autoblocks-tests:
runs-on: ubuntu-latest
steps:
...
# But we only want to send Slack notifications for scheduled runs
- name: Set Autoblocks Slack webhook URL
if: github.event_name == 'schedule'
run: echo "AUTOBLOCKS_SLACK_WEBHOOK_URL=${{ secrets.AUTOBLOCKS_SLACK_WEBHOOK_URL }}" >> $GITHUB_ENV
- name: Run Autoblocks tests
run: npx autoblocks testing exec -- <cmd>
env:
# This is created automatically by GitHub
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# These were added by you to your repository's secrets
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
AUTOBLOCKS_V2_API_KEY: ${{ secrets.AUTOBLOCKS_V2_API_KEY }}
# The AUTOBLOCKS_SLACK_WEBHOOK_URL has already been set
# above, so it doesn't need to be set here
Autoblocks updates the same comment with the latest test results each time the workflow runs.
This prevents the comment section from becoming cluttered.
If you would like to disable comments, you can set the AUTOBLOCKS_DISABLE_GITHUB_COMMENTS environment variable to 1.
GitHub Actions Permissions (Advanced)
If your GitHub organization has set the default permissions
for GITHUB_TOKEN to be restricted, you will need to explicitly give the workflow permission to post comments on pull requests or commits.
- Commenting on a pull request from a
pull_request event:
name: Autoblocks Tests
on: pull_request
permissions:
# Allow commenting on pull requests
issues: write
- Commenting on a pull request from a
push event:
name: Autoblocks Tests
on: push
permissions:
# Allow commenting on pull requests
issues: write
# Allow getting associated pull requests when event is type `push`
pull-requests: read
Other CI Providers
Using a CI provider other than GitHub Actions? Contact us at support@autoblocks.ai and we’ll help get you set up.
