Test Case Management

Autoblocks enables you to manage test cases the way you would like to; in code, managed through our web application, or a hybrid approach with both. This gives flexibility to the developer about how and when they want to integrate with Autoblocks.

Prerequisites

In order to take full advantage of what Autoblocks has to offer for managing test cases, you will need to get started by setting up and running your first test. This will have you configure a simple test through code, then enable you to quickly iterate by moving test cases to a managed setup. Click here to get started testing.

Picking a test suite

Now that you have a test suite set up, you are ready to start managing test cases in the UI. To do so, visit the Test Cases page, where you will see a list of test suites that you can create test cases for. Click the one you created when running your first test; this is where you can set up test cases related to your suite.

Managing test cases

Autoblocks provides a simple JSON editor for entering the content of your test case.

Autoblocks validates that a test case is valid JSON. When retrieving managed test cases in code, you will need to map the managed test cases to your test case class.

Generate test cases from production data

To start generating test cases from production data, you need an existing test suite. If you have not created a test suite yet, follow the prerequisites to do so.

Once your test suite exists, create a saved view that filters down to the events that are relevant to what you are testing. Open the saved view from the side bar, and you will now see a button to generate test cases.

Autoblocks requires you to specify a test suite and a structured list of mappings to generate test cases. These mappings delineate an event key for extracting a specific value and a corresponding test case key where this value is stored upon the generation of a new test case. It is suggested that you have your saved view open in another window so you can easily review what properties you want to map.

If there are existing test cases, you must include mappings for each existing test case key.

When selecting an event key, most often you will use the properties dropdown. However, event properties that are represented as arrays will not be present. In this case, you can use a manual input to type in the necessary mapping. For example messages[0].content.

After selecting your test suite and setting up your mappings, you will see a table of suggested test cases. Select the ones you want to add, and click "Create test cases" to add them to your test suite.

Using managed test cases

After you have set up managed test cases for a test suite, you can use the API client to fetch them.

For more information about the Autoblocks API client, view the JavaScript or Python reference.

  import { runTestSuite } from '@autoblocks/client/testing';
  import { AutoblocksAPIClient } from '@autoblocks/client'

  // Function to test
  function testFn(args: { testCase: TestCase }) {
    return genFlashcardsFromNotes(args.testCase.notes)
  }

  // Instantiate API client
  const client = new AutoblocksAPIClient();

  // Fetch test cases
  const { testCases: managedTestCases } = await client.getTestCases<TestCase>({
    testSuiteId: 'flashcard-generator',
  });

  // Build test cases array
  const testCases: TestCase[] = managedTestCases.map(
    (testCase) => testCase.body
  );

  // Run test suite with managed test cases
  await runTestSuite<TestCase, Flashcard[]>({
    id: 'flashcard-generator',
    testCases: testCases,
    testCaseHash: ['notes'],
    evaluators: [new IsSupportedByNotes()],
    fn: testFn,
    maxTestCaseConcurrency: 5,
  });

Hybrid approach using managed and in-code test cases

You can also choose to take a hybrid approach, using both managed and in-code test cases. This may be useful if your developers want to be able to manage their test cases in code, while still allowing other team members to experiment by adding and adjusting managed test cases in Autoblocks.

To uniquely identify test cases, Autoblocks has a function that uses property keys to generate a hash. All test cases have their hashes evaluated the same way.

import { AutoblocksAPIClient } from '@autoblocks/client'

// Instantiate API client
const client = new AutoblocksAPIClient();

// Test cases that are defined in code. Can be defined anywhere
const inCodeTestCases = [
  {
    notes: "The United States was founded in 1776."
  },
  {
    notes: "Autoblocks is an AI ops company focused on helping product teams build with AI."
  }
];

// Fetch managed test cases
const { testCases: managedTestCases } = await client.getTestCases<TestCase>({
  testSuiteId: 'flashcard-generator',
});

// Build test cases array
const testCases: TestCase[] = [
  ...managedTestCases.map((testCase) => testCase.body)
  ...inCodeTestCases
];

Test results can then be viewed in the test run comparison UI.

Learn more about test run comparison