​
Python SDK Reference

​
run_test_suite

The main entrypoint into the testing framework.

namerequiredtypedescription
idtruestringA unique ID for the test suite. This will be displayed in the Autoblocks platform and should remain the same for the lifetime of a test suite.
test_casestruelist[BaseTestCase]A list of instances that subclass BaseTestCase. These are typically dataclasses and can be any schema that facilitates testing your application. They will be passed directly to fn and will also be made available to your evaluators. BaseTestCase is an abstract base class that requires you to implement the hash function. See Test case hashing for more information.
evaluatorstruelist[BaseTestEvaluator]A list of instances that subclass BaseTestEvaluator.
fntrueCallable[[BaseTestCase], Any]The function you are testing. Its only argument is an instance of a test case. This function can be synchronous or asynchronous and can return any type.
max_test_case_concurrencyfalseintThe maximum number of test cases that can be running concurrently through fn. Useful to avoid rate limiting from external services, such as an LLM provider.

​
BaseTestCase

An abstract base class that you can subclass to create your own test cases.

namerequiredtypedescription
hashtrueCallable[[], str]A method that returns a string that uniquely identifies the test case for its lifetime. See Test case hashing for more information.

​
BaseTestEvaluator

An abstract base class that you can subclass to create your own evaluators.

namerequiredtypedescription
idtruestringA unique identifier for the evaluator.
max_concurrencyfalsenumberThe maximum number of concurrent calls to evaluate_test_case allowed for the evaluator. Useful to avoid rate limiting from external services, such as an LLM provider.
evaluate_test_casetrueCallable[[BaseTestCase, Any], Optional[Evaluation]]Creates an evaluation on a test case and its output. This method can be synchronous or asynchronous.

​
Evaluation

A class that represents the result of an evaluation.

namerequiredtypedescription
scoretruenumberA number between 0 and 1 that represents the score of the evaluation.
thresholdfalseThresholdAn optional Threshold that describes the range the score must be in in order to be considered passing. If no threshold is attached, the score is reported and the pass / fail status is undefined.
metadatafalseobjectKey-value pairs that provide additional context about the evaluation. This is typically used to explain why an evaluation failed. Attached metadata is surfaced in the test run comparison UI.

​
Threshold

A class that defines the passing criteria for an evaluation.

namerequiredtypedescription
ltfalsenumberThe score must be less than this number in order to be considered passing.
ltefalsenumberThe score must be less than or equal to this number in order to be considered passing.
gtfalsenumberThe score must be greater than this number in order to be considered passing.
gtefalsenumberThe score must be greater than or equal to this number in order to be considered passing.

​
Example

from dataclasses import dataclass
from autoblocks.testing.models import BaseTestCase
from autoblocks.testing.models import BaseTestEvaluator
from autoblocks.testing.models import Evaluation
from autoblocks.testing.models import Threshold
from autoblocks.testing.run import run_test_suite
from autoblocks.testing.util import md5

@dataclass
class MyTestCase(BaseTestCase):
    input: str
    expected_substrings: list[str]

    def hash(self) -> str:
        return md5(self.input)

class HasAllSubstrings(BaseTestEvaluator):
    id = "has-all-substrings"

    def evaluate_test_case(self, test_case: MyTestCase, output: str) -> Evaluation:
        score = 1.0
        for substring in test_case.expected_substrings:
            if substring not in output:
                score = 0.0
                break

        return Evaluation(
            score=score,
            threshold=Threshold(gte=1.0),
            metadata={
                "reason": "Output must contain all expected substrings",
            },
        )

run_test_suite(
    id="my-test-suite",
    test_cases=[
        MyTestCase(
            input="hello world",
            expected_substrings=["hello", "world"],
        )
    ],
    fn=lambda test_case: test_case.input,
    evaluators=[HasAllSubstrings()],
)