Autoblocks

Empower Non-Technical Users to Test and Evaluate

Our Testing SDKs provide engineers with best-in-class tools for a rapid iterate → test → evaluate feedback loop as they make changes to a codebase. At the same time, non-technical stakeholders often have valuable insights that can be difficult to capture and integrate into the development process. Directly including these folks—your designers, product managers, CEOs, and anyone else who isn't writing code—in the development and testing process of your AI product will be crucial to its success.

Prerequisites

There are a few requirements to get started:

  1. Setup test suites
  2. Setup test suites to run in CI
  3. Setup prompt management

With these in place, you can empower anyone on your team to make changes to a prompt and then run the same tests a developer would against those changes.

Update your GitHub Actions workflow

The workflow you created in the CI setup guide should be updated to run on the workflow_dispatch event. It requires an input called autoblocks-overrides:

name: Autoblocks Tests

on:
  pull_request:
  workflow_dispatch:
    inputs:
      autoblocks-overrides:
        type: string
        description: The prompt snapshot(s) that triggered this workflow
        required: false

You will never need to set this input yourself; it is used by our testing exec CLI command under the hood to notify the SDKs that they should override their configuration.

Install the GitHub App

Install our GitHub App to the repository where the above GitHub Actions workflow resides.

Invite team members to Autoblocks

Make sure you've invited any team members that want to participate in the testing and evaluation process to your Autoblocks organization.

Your GitHub Actions workflow will be triggered by our GitHub App and not by an individual GitHub user. This means you don't need to spend GitHub seats on team members who will only be running tests.

Run a test from the UI

Running tests from the UI is centered around prompts. With Autoblocks prompt management integrated into your codebase, anyone can make changes to a prompt and its parameters in the Autoblocks platform and then run the same tests a developer would against those changes.

Find the prompt you want to test

Navigate to the prompts page and click on the edit button for the prompt you want to test.

Make changes to the prompt

Make changes to the prompt's parameters or templates on the Edit tab.

Test the prompt

Move to the Test tab, preview your changes, and click the Test Prompt button.

View your test run

You can find your UI-triggered test runs on the test suites page.