Dataset & Test Case Management

Autoblocks enables you to manage test cases the way you would like to; in code, managed through our web application, or a hybrid approach with both. This gives flexibility to the developer about how and when they want to integrate with Autoblocks.

Schema management

When creating a dataset for the first time, you will be prompted to build a schema. A schema is the list of properties that each item in the dataset will have. Autoblocks will automatically version the schema every time you update it and prevent you from making any breaking changes.

Dataset splits

Splits are a way to divide your dataset into smaller, more manageable pieces. This is useful for creating a subset of your dataset to use for testing different scenarios.

Using datasets in code

After you have set up a dataset, you can use the Autoblocks API clint to fetch the items.

For more information about the Autoblocks API client, view the JavaScript or Python reference.

Generate types (TypeScript only)

When using the TypeScript SDK, you can generate types for your datasets based on the schema definition. This allows you to have strongly typed data in your application.

In order to generate types, you need to set your Autoblocks API key from the settings page as an environment variable:

export AUTOBLOCKS_API_KEY=...

Then, add the datasets generate command to your package.json scripts:

"scripts": {
  "gen": "datasets generate"
}

You will need to run this script any time you make a schema change to a dataset.

Make sure to generate the types in your CI/CD pipeline before running type checking on your application.

"scripts": {
  "gen": "datasets generate",
  "type-check": "npm run gen && tsc --noEmit"
}

Retrieve dataset items for the latest revision in a schema version

import { AutoblocksAPIClient } from '@autoblocks/client';

const client = new AutoblocksAPIClient({
  apiKey: process.env.AUTOBLOCKS_API_KEY,
});

const main = async () => {
  const dataset = await client.getDataset({
    name: 'My Dataset',
    schemaVersion: '1',
  });

  console.log(dataset);
};

main();

Retrieve dataset items for a specific revision in a schema version

You can also pin the revision id of the dataset if you want to always require a code change when the dataset is updated.

import { AutoblocksAPIClient } from '@autoblocks/client';

const client = new AutoblocksAPIClient({
  apiKey: process.env.AUTOBLOCKS_API_KEY,
});

const main = async () => {
  const dataset = await client.getDataset({
    name: 'My Dataset',
    schemaVersion: '1',
    revisionId: '123',
  });

  console.log(dataset);
};

main();

Filter by splits

You can also filter the dataset by splits. This is useful if you want to get a subset of the dataset for testing.

import { AutoblocksAPIClient } from '@autoblocks/client';

const client = new AutoblocksAPIClient({
  apiKey: process.env.AUTOBLOCKS_API_KEY,
});

const main = async () => {
  const dataset = await client.getDataset({
    name: 'My Dataset',
    schemaVersion: '1',
    splits: ['split-1'],
  });

  console.log(dataset);
};

main();

Associate test cases with dataset items

You can associate test cases with dataset items to associate test results with the dataset items in the Autoblocks UI.

import { AutoblocksAPIClient } from '@autoblocks/client';
import { runTestSuite } from '@autoblocks/client/testing';

const client = new AutoblocksAPIClient({
  apiKey: process.env.AUTOBLOCKS_API_KEY,
});

interface TestCase {
  input: string;
  datasetItemId: string;
}

const dataset = await client.getDataset({
  name: 'My Dataset',
  schemaVersion: '1',
});

runTestSuite<TestCase, string>({
  id: 'my-test-suite',
  testCases: dataset.items.map((item) => ({
    datasetItemId: item.id,
    ...item.data,
  }))
  testCaseHash: ['input'],
  fn: ({ testCase }) => testCase.input, // Replace with your LLM call
  evaluators: [], // Replace with your evaluators
});