> ## Documentation Index
> Fetch the complete documentation index at: https://docs.autoblocks.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Python SDK Reference

> Technical reference for the Autoblocks Python SDK testing functionality.

# Python SDK Reference

## `run_test_suite`

The main entrypoint into the testing framework.

| name                        | required | type                            | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| --------------------------- | -------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`                        | true     | `string`                        | A unique ID for the test suite. This will be displayed in the Autoblocks platform and should remain the same for the lifetime of a test suite.                                                                                                                                                                                                                                                                                                                             |
| `test_cases`                | true     | `list[BaseTestCase]`            | A list of instances that subclass `BaseTestCase`. These are typically [dataclasses](https://docs.python.org/3/library/dataclasses.html) and can be any schema that facilitates testing your application. They will be passed directly to `fn` and will also be made available to your evaluators. `BaseTestCase` is an abstract base class that requires you to implement the `hash` function. See [Test case hashing](#reference-test-case-hashing) for more information. |
| `evaluators`                | true     | `list[BaseTestEvaluator]`       | A list of instances that subclass [`BaseTestEvaluator`](#reference-base-test-evaluator).                                                                                                                                                                                                                                                                                                                                                                                   |
| `fn`                        | true     | `Callable[[BaseTestCase], Any]` | The function you are testing. Its only argument is an instance of a test case. This function can be synchronous or asynchronous and can return any type.                                                                                                                                                                                                                                                                                                                   |
| `max_test_case_concurrency` | false    | `int`                           | The maximum number of test cases that can be running concurrently through `fn`. Useful to avoid rate limiting from external services, such as an LLM provider.                                                                                                                                                                                                                                                                                                             |

## `BaseTestCase`

An abstract base class that you can subclass to create your own test cases.

| name   | required | type                | description                                                                                                                                                         |
| ------ | -------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hash` | true     | `Callable[[], str]` | A method that returns a string that uniquely identifies the test case for its lifetime. See [Test case hashing](#reference-test-case-hashing) for more information. |

## `BaseTestEvaluator`

An abstract base class that you can subclass to create your own evaluators.

| name                 | required | type                                                  | description                                                                                                                                                              |
| -------------------- | -------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `id`                 | true     | `string`                                              | A unique identifier for the evaluator.                                                                                                                                   |
| `max_concurrency`    | false    | `number`                                              | The maximum number of concurrent calls to `evaluate_test_case` allowed for the evaluator. Useful to avoid rate limiting from external services, such as an LLM provider. |
| `evaluate_test_case` | true     | `Callable[[BaseTestCase, Any], Optional[Evaluation]]` | Creates an evaluation on a test case and its output. This method can be synchronous or asynchronous.                                                                     |

## `Evaluation`

A class that represents the result of an evaluation.

| name        | required | type        | description                                                                                                                                                                                                                    |
| ----------- | -------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `score`     | true     | `number`    | A number between 0 and 1 that represents the score of the evaluation.                                                                                                                                                          |
| `threshold` | false    | `Threshold` | An optional [`Threshold`](#reference-threshold) that describes the range the score must be in in order to be considered passing. If no threshold is attached, the score is reported and the pass / fail status is undefined.   |
| `metadata`  | false    | `object`    | Key-value pairs that provide additional context about the evaluation. This is typically used to explain why an evaluation failed. Attached metadata is surfaced in the [test run comparison](/testing/test-run-comparison) UI. |

## `Threshold`

A class that defines the passing criteria for an evaluation.

| name  | required | type     | description                                                                                   |
| ----- | -------- | -------- | --------------------------------------------------------------------------------------------- |
| `lt`  | false    | `number` | The score must be **less than** this number in order to be considered passing.                |
| `lte` | false    | `number` | The score must be **less than or equal** to this number in order to be considered passing.    |
| `gt`  | false    | `number` | The score must be **greater than** this number in order to be considered passing.             |
| `gte` | false    | `number` | The score must be **greater than or equal** to this number in order to be considered passing. |

## Example

<CodeGroup>
  ```python
  from dataclasses import dataclass
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.models import BaseTestEvaluator
  from autoblocks.testing.models import Evaluation
  from autoblocks.testing.models import Threshold
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class MyTestCase(BaseTestCase):
      input: str
      expected_substrings: list[str]

      def hash(self) -> str:
          return md5(self.input)

  class HasAllSubstrings(BaseTestEvaluator):
      id = "has-all-substrings"

      def evaluate_test_case(self, test_case: MyTestCase, output: str) -> Evaluation:
          score = 1.0
          for substring in test_case.expected_substrings:
              if substring not in output:
                  score = 0.0
                  break

          return Evaluation(
              score=score,
              threshold=Threshold(gte=1.0),
              metadata={
                  "reason": "Output must contain all expected substrings",
              },
          )

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          MyTestCase(
              input="hello world",
              expected_substrings=["hello", "world"],
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[HasAllSubstrings()],
  )
  ```
</CodeGroup>