​
Key Features

Autoblocks Testing provides a powerful framework for testing your LLM applications. It enables you to declaratively define tests and execute them either locally or in a CI/CD pipeline.

​
Declarative Test Definition

Define your tests using a simple, declarative API that works with both TypeScript and Python. Your tests can exist as standalone scripts or be integrated into your existing test framework.

​
Flexible Test Cases

Create test cases that match your application’s needs. Test cases can contain any properties necessary to run your tests and make assertions on the output.

​
Powerful Evaluators

Build custom evaluators to assess your test outputs. Evaluators can:

  • Score outputs on a scale from 0 to 1
  • Define pass/fail thresholds
  • Include metadata for better debugging
  • Support both synchronous and asynchronous evaluation
  • Handle concurrent execution with configurable limits

​
Local and CI/CD Support

Run your tests:

  • Locally during development
  • In your CI/CD pipeline for automated testing
  • With progress tracking and real-time results

​
Rich Results Visualization

View detailed test results in the Autoblocks platform, including:

  • Test suite progress
  • Individual test case results
  • Evaluation scores and metadata
  • Failure analysis

​
Getting Started

Choose your preferred language to begin:

​
Core Concepts

​
Test Cases

Test cases define the inputs and expected outputs for your tests. They can be simple or complex, depending on your needs.

​
Evaluators

Evaluators assess the output of your tests and determine if they pass or fail. They can:

  • Score outputs numerically
  • Define pass/fail thresholds
  • Include metadata for debugging
  • Support both sync and async evaluation

​
Test Suites

Test suites bring together your test cases and evaluators to run comprehensive tests on your application.

​
Next Steps