> ## Documentation Index
> Fetch the complete documentation index at: https://docs.autoblocks.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Out of Box Evaluators

> Learn about the built-in evaluators available in Autoblocks.

Autoblocks provides a set of evaluators that can be used out of the box. These evaluators are designed to be easily integrated into your test suite and can help you get started with testing your AI-powered applications.

Each evaluator below lists the custom properties and methods that need to be implemented to use the evaluator in your test suite.
You must set the `id` property, which is a unique identifier for the evaluator.

All of the code snippets can be run by following the instructions in the [Quick Start](/v2/guides/testing/overview.mdx) guide.

**Ragas**

* [LLM Context Precision With Reference](#out-of-box-evaluators-ragas)
* [Non LLM Context Precision With Reference](#out-of-box-evaluators-ragas)
* [LLM Context Recall](#out-of-box-evaluators-ragas)
* [Non LLM Context Recall](#out-of-box-evaluators-ragas)
* [Context Entities Recall](#out-of-box-evaluators-ragas)
* [Noise Sensitivity](#out-of-box-evaluators-ragas)
* [Response Relevancy](#out-of-box-evaluators-ragas)
* [Faithfulness](#out-of-box-evaluators-ragas)
* [Factual Correctness](#out-of-box-evaluators-ragas)
* [Semantic Similarity](#out-of-box-evaluators-semantic-similarity)

## Logic Based

### Is Equals

The `IsEquals` evaluator checks if the expected output equals the actual output.

Scores 1 if equal, 0 otherwise.

| Name               | Required | Type                            | Description                                    |
| ------------------ | -------- | ------------------------------- | ---------------------------------------------- |
| test\_case\_mapper | Yes      | Callable\[\[BaseTestCase], str] | Map your test case to a string for comparison. |
| output\_mapper     | Yes      | Callable\[\[OutputType], str]   | Map your output to a string for comparison.    |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseIsEquals
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str
      expected_output: str

      def hash(self) -> str:
          return md5(self.input)

  class IsEquals(BaseIsEquals[TestCase, str]):
      id = "is-equals"

      def test_case_mapper(self, test_case: TestCase) -> str:
          return test_case.expected_output

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="hello world",
              expected_output="hello world",
          ),
          TestCase(
              input="hi world",
              expected_output="hello world",
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[IsEquals()],
  )
  ```

  ```typescript TypeScript
  import { BaseIsEquals } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
    expectedOutput: string;
  }

  class IsEquals extends BaseIsEquals<TestCase, string> {
    id = 'is-equals';

    outputMapper(args: { output: string }) {
      return args.output;
    }

    testCaseMapper(args: { testCase: TestCase }) {
      return args.testCase.expectedOutput;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'hello world',
        expectedOutput: 'hello world',
      },
      {
        input: 'hi world',
        expectedOutput: 'hello world',
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new IsEquals()],
  });
  ```
</CodeGroup>

### Is Valid JSON

The `IsValidJSON` evaluator checks if the output is valid JSON.

Scores 1 if it is valid, 0 otherwise.

| Name           | Required | Type                          | Description                                                         |
| -------------- | -------- | ----------------------------- | ------------------------------------------------------------------- |
| output\_mapper | Yes      | Callable\[\[OutputType], str] | Map your output to the string that you want to check is valid JSON. |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseIsValidJSON
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str

      def hash(self) -> str:
          return md5(self.input)

  class IsValidJSON(BaseIsValidJSON[TestCase, str]):
      id = "is-valid-json"

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="hello world",
          ),
          TestCase(
              input='{"hello": "world"}'
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[IsValidJSON()],
  )
  ```

  ```typescript TypeScript
  import { BaseIsValidJSON } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
  }

  class IsValidJSON extends BaseIsValidJSON<TestCase, string> {
    id = 'is-valid-json';

    outputMapper(args: { output: string }) {
      return args.output;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'hello world',
      },
      {
        input: '{"hello": "world"}'
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new IsValidJSON()],
  });
  ```
</CodeGroup>

### Has All Substrings

The `HasAllSubstrings` evaluator checks if the output contains all the expected substrings.

Scores 1 if all substrings are present, 0 otherwise.

| Name               | Required | Type                                   | Description                                                         |
| ------------------ | -------- | -------------------------------------- | ------------------------------------------------------------------- |
| test\_case\_mapper | Yes      | Callable\[\[BaseTestCase], list\[str]] | Map your test case to a list of strings to check for in the output. |
| output\_mapper     | Yes      | Callable\[\[OutputType], str]          | Map your output to a string for comparison.                         |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseHasAllSubstrings
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str
      expected_substrings: list[str]

      def hash(self) -> str:
          return md5(self.input)

  class HasAllSubstrings(BaseHasAllSubstrings[TestCase, str]):
      id = "has-all-substrings"

      def test_case_mapper(self, test_case: TestCase) -> list[str]:
          return test_case.expected_substrings

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="hello world",
              expected_substrings=["hello", "world"],
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[HasAllSubstrings()],
  )
  ```

  ```typescript TypeScript
  import { BaseHasAllSubstrings } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
    expectedSubstrings: string[];
  }

  class HasAllSubstrings extends BaseHasAllSubstrings<TestCase, string> {
    id = 'has-all-substrings';

    outputMapper(args: { output: string }) {
      return args.output;
    }

    testCaseMapper(args: { testCase: TestCase }) {
      return args.testCase.expectedSubstrings;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'hello world',
        expectedSubstrings: ['hello', 'world'],
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new HasAllSubstrings()],
  });
  ```
</CodeGroup>

### Assertions (Rubric/Rules)

The `Assertions` evaluator enables you to define a set of assertions or rules that your output must satisfy.

<Note>
  Individual assertions can be marked as not required, and if they are not met, the evaluator will still pass.
</Note>

| Name                 | Required | Type                                                         | Description                                      |
| -------------------- | -------- | ------------------------------------------------------------ | ------------------------------------------------ |
| evaluate\_assertions | Yes      | Callable\[\[BaseTestCase, Any], Optional\[List\[Assertion]]] | Implement your logic to evaluate the assertions. |

<CodeGroup>
  ```python Python
  from typing import Optional
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseAssertions
  from autoblocks.testing.models import Assertion
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCaseCriterion:
      criterion: str
      required: bool


  @dataclass
  class TestCase(BaseTestCase):
      input: str
      assertions: Optional[list[TestCaseCriterion]] = None

      def hash(self) -> str:
          return md5(self.input)


  class AssertionsEvaluator(BaseAssertions[TestCase, str]):
      id = "assertions"

      def evaluate_assertions(self, test_case: TestCase, output: str) -> list[Assertion]:
          if test_case.assertions is None:
              return []
          result = []
          for assertion in test_case.assertions:
              result.append(
                  Assertion(
                      criterion=assertion.criterion,
                      passed=assertion.criterion in output,
                      required=assertion.required,
                  )
              )
          return result


  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="hello world",
              assertions=[
                  TestCaseCriterion(criterion="hello", required=True),
                  TestCaseCriterion(criterion="world", required=True),
                  TestCaseCriterion(criterion="hi", required=False),
              ],
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[AssertionsEvaluator()],
  )
  ```

  ```typescript TypeScript
  import {
    runTestSuite,
    BaseAssertions,
    Assertion,
  } from '@autoblocks/client/testing';

  interface TestCaseCriterion {
    criterion: string;
    required: boolean;
  }

  interface TestCase {
    input: string;
    assertions: TestCaseCriterion[];
  }

  class AssertionsEvaluator extends BaseAssertions<TestCase, string> {
    id = 'assertions';

    evaluateAssertions(args: {
      testCase: TestCase;
      output: string;
    }): Assertion[] {
      return args.testCase.assertions.map((assertion) => {
        return {
          criterion: assertion.criterion,
          passed: args.output.includes(assertion.criterion),
          required: assertion.required,
        };
      });
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'hello world',
        assertions: [
          {
            criterion: 'hello',
            required: true,
          },
          {
            criterion: 'world',
            required: true,
          },
          {
            criterion: 'hi',
            required: false,
          },
        ],
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new AssertionsEvaluator()],
  });
  ```
</CodeGroup>

## LLM Judges

### Custom LLM Judge

The `CustomLLMJudge` evaluator enables you to define custom evaluation criteria using an LLM judge.

| Name                    | Required | Type                                  | Description                                                              |
| ----------------------- | -------- | ------------------------------------- | ------------------------------------------------------------------------ |
| output\_mapper          | Yes      | Callable\[\[OutputType], str]         | Map your output to the string that you want to evaluate.                 |
| model                   | No       | str                                   | The OpenAI model to use. Defaults to "gpt-4o".                           |
| num\_overrides          | No       | int                                   | Number of recent evaluation overrides to use as examples. Defaults to 0. |
| example\_output\_mapper | No       | Callable\[\[EvaluationOverride], str] | Map an EvaluationOverride to a string representation of the output.      |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseCustomLLMJudge
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str

      def hash(self) -> str:
          return md5(self.input)

  class CustomLLMJudge(BaseCustomLLMJudge[TestCase, str]):
      id = "custom-llm-judge"

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="Hello, how are you?",
          ),
          TestCase(
              input="I hate you!"
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[CustomLLMJudge()],
  )
  ```

  ```typescript TypeScript
  import { BaseCustomLLMJudge } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
  }

  class CustomLLMJudge extends BaseCustomLLMJudge<TestCase, string> {
    id = 'custom-llm-judge';

    outputMapper(args: { output: string }) {
      return args.output;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'Hello, how are you?',
      },
      {
        input: 'I hate you!'
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new CustomLLMJudge()],
  });
  ```
</CodeGroup>

### Accuracy

The `Accuracy` evaluator checks if the output is accurate compared to an expected output.

Scores 1 if accurate, 0.5 if somewhat accurate, 0 if inaccurate.

| Name                    | Required | Type                                  | Description                                                              |
| ----------------------- | -------- | ------------------------------------- | ------------------------------------------------------------------------ |
| output\_mapper          | Yes      | Callable\[\[OutputType], str]         | Map your output to the string that you want to check for accuracy.       |
| model                   | No       | str                                   | The OpenAI model to use. Defaults to "gpt-4o".                           |
| num\_overrides          | No       | int                                   | Number of recent evaluation overrides to use as examples. Defaults to 0. |
| example\_output\_mapper | No       | Callable\[\[EvaluationOverride], str] | Map an EvaluationOverride to a string representation of the output.      |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseAccuracy
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str
      expected_output: str

      def hash(self) -> str:
          return md5(self.input)

  class Accuracy(BaseAccuracy[TestCase, str]):
      id = "accuracy"

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="What is the capital of France?",
              expected_output="The capital of France is Paris.",
          ),
          TestCase(
              input="What is the capital of France?",
              expected_output="Paris is the capital of France.",
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[Accuracy()],
  )
  ```

  ```typescript TypeScript
  import { BaseAccuracy } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
    expectedOutput: string;
  }

  class Accuracy extends BaseAccuracy<TestCase, string> {
    id = 'accuracy';

    outputMapper(args: { output: string }) {
      return args.output;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'What is the capital of France?',
        expectedOutput: 'The capital of France is Paris.',
      },
      {
        input: 'What is the capital of France?',
        expectedOutput: 'Paris is the capital of France.',
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new Accuracy()],
  });
  ```
</CodeGroup>

### NSFW

The `NSFW` evaluator checks if the output is safe for work.

Scores 1 if safe, 0 otherwise.

| Name                    | Required | Type                                  | Description                                                              |
| ----------------------- | -------- | ------------------------------------- | ------------------------------------------------------------------------ |
| output\_mapper          | Yes      | Callable\[\[OutputType], str]         | Map your output to the string that you want to check for NSFW content.   |
| model                   | No       | str                                   | The OpenAI model to use. Defaults to "gpt-4o".                           |
| num\_overrides          | No       | int                                   | Number of recent evaluation overrides to use as examples. Defaults to 0. |
| example\_output\_mapper | No       | Callable\[\[EvaluationOverride], str] | Map an EvaluationOverride to a string representation of the output.      |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseNSFW
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str

      def hash(self) -> str:
          return md5(self.input)

  class NSFW(BaseNSFW[TestCase, str]):
      id = "nsfw"

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="Hello, how are you?",
          ),
          TestCase(
              input="Explicit content here"
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[NSFW()],
  )
  ```

  ```typescript TypeScript
  import { BaseNSFW } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
  }

  class NSFW extends BaseNSFW<TestCase, string> {
    id = 'nsfw';

    outputMapper(args: { output: string }) {
      return args.output;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'Hello, how are you?',
      },
      {
        input: 'Explicit content here'
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new NSFW()],
  });
  ```
</CodeGroup>

### Toxicity

The `Toxicity` evaluator checks if the output is not toxic.

Scores 1 if it is not toxic, 0 otherwise.

| Name                    | Required | Type                                  | Description                                                              |
| ----------------------- | -------- | ------------------------------------- | ------------------------------------------------------------------------ |
| output\_mapper          | Yes      | Callable\[\[OutputType], str]         | Map your output to the string that you want to check for toxicity.       |
| model                   | No       | str                                   | The OpenAI model to use. Defaults to "gpt-4o".                           |
| num\_overrides          | No       | int                                   | Number of recent evaluation overrides to use as examples. Defaults to 0. |
| example\_output\_mapper | No       | Callable\[\[EvaluationOverride], str] | Map an EvaluationOverride to a string representation of the output.      |

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from autoblocks.testing.evaluators import BaseToxicity
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      input: str

      def hash(self) -> str:
          return md5(self.input)

  class Toxicity(BaseToxicity[TestCase, str]):
      id = "toxicity"

      def output_mapper(self, output: str) -> str:
          return output

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              input="Hello, how are you?",
          ),
          TestCase(
              input="I hate you!"
          )
      ],
      fn=lambda test_case: test_case.input,
      evaluators=[Toxicity()],
  )
  ```

  ```typescript TypeScript
  import { BaseToxicity } from '@autoblocks/client/testing';
  import { runTestSuite } from '@autoblocks/client/testing/v2';

  interface TestCase {
    input: string;
  }

  class Toxicity extends BaseToxicity<TestCase, string> {
    id = 'toxicity';

    outputMapper(args: { output: string }) {
      return args.output;
    }
  }

  runTestSuite<TestCase, string>({
    id: 'my-test-suite',
    testCases: [
      {
        input: 'Hello, how are you?',
      },
      {
        input: 'I hate you!'
      }
    ],
    testCaseHash: ['input'],
    fn: ({ testCase }) => testCase.input,
    evaluators: [new Toxicity()],
  });
  ```
</CodeGroup>

## Ragas

[Ragas](https://docs.ragas.io/en/stable/) is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines.
We have built wrappers around the metrics to make integration with Autoblocks seamless.

**Available Ragas evaluators:**

* [`BaseRagasLLMContextPrecisionWithReference`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision) uses a LLM to measure the proportion of relevant chunks in the retrieved\_contexts.
* [`BaseRagasNonLLMContextPrecisionWithReference`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_entities_recall) measures the proportion of relevant chunks in the retrieved\_contexts without using a LLM.
* [`BaseRagasLLMContextRecall`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/#llm-based-context-recall) evaluates the extent to which the retrieved context aligns with the annotated answer, treated as the ground truth.
* [`BaseRagasNonLLMContextRecall`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall) uses non llm string comparison metrics to identify if a retrieved context is relevant or not.
* [`BaseRagasContextEntitiesRecall`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_entities_recall)  evaluates the measure of recall of the retrieved context, based on the number of entities present in both ground\_truths and contexts relative to the number of entities present in the ground\_truths alone.
* [`BaseRagasNoiseSensitivity`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity) measures how often a system makes errors by providing incorrect responses when utilizing either relevant or irrelevant retrieved documents.
* [`BaseRagasResponseRelevancy`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance) focuses on assessing how pertinent the generated answer is to the given prompt.
* [`BaseRagasFaithfulness`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness) measures the factual consistency of the generated answer against the given context.
* [`BaseRagasFactualCorrectness`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness) compares and evaluates the factual accuracy of the generated response with the reference. This metric is used to determine the extent to which the generated response aligns with the reference.
* [`BaseRagasSemanticSimilarity`](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/semantic_similarity) measures the semantic resemblance between the generated answer and the ground truth.

<Note>
  The Ragas evaluators are only available in the Python SDK. You must install Ragas (`pip install ragas`) before using these evaluators.
  Our wrappers require at least version `0.2.*` of Ragas.
</Note>

| Name                        | Required | Type                                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| --------------------------- | -------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| id                          | Yes      | str                                         | The unique identifier for the evaluator.                                                                                                                                                                                                                                                                                                                                                                                                                     |
| threshold                   | No       | Threshold                                   | The [threshold](/testing/sdk-reference#threshold) for the evaluation used to determine pass/fail.                                                                                                                                                                                                                                                                                                                                                            |
| llm                         | No       | Any                                         | Custom LLM for the evaluation. Required for any Ragas evaluator that uses a LLM. Read More: [https://docs.ragas.io/en/stable/howtos/customizations/customize\_models/](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/)                                                                                                                                                                                                              |
| embeddings                  | No       | Any                                         | Custom embeddings model for the evaluation. Required for any Ragas evaluator that uses embeddings. Read More: [https://docs.ragas.io/en/stable/howtos/customizations/customize\_models/](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/)                                                                                                                                                                                            |
| mode                        | No       | str                                         | Only applicable for the `BaseRagasFactualCorrectness` evaluator. Read More: [https://docs.ragas.io/en/stable/concepts/metrics/available\_metrics/factual\_correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness)                                                                                                                                                                                               |
| atomicity                   | No       | str                                         | Only applicable for the `BaseRagasFactualCorrectness` evaluator. Read More: [https://docs.ragas.io/en/stable/concepts/metrics/available\_metrics/factual\_correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness)                                                                                                                                                                                               |
| focus                       | No       | str                                         | Only applicable for the `BaseRagasNoiseSensitivity` and `BaseRagasFaithfulness` evaluator. Read More: [https://docs.ragas.io/en/stable/concepts/metrics/available\_metrics/noise\_sensitivity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity) and [https://docs.ragas.io/en/stable/concepts/metrics/available\_metrics/faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness) |
| user\_input\_mapper         | No       | Callable\[\[TestCaseType, OutputType], str] | Map your test case or output to the user input passed to Ragas.                                                                                                                                                                                                                                                                                                                                                                                              |
| response\_mapper            | No       | Callable\[OutputType], str]                 | Map your output to the response passed to Ragas.                                                                                                                                                                                                                                                                                                                                                                                                             |
| reference\_mapper           | No       | Callable\[\[TestCaseType], str]             | Map your test case to the reference passed to Ragas.                                                                                                                                                                                                                                                                                                                                                                                                         |
| retrieved\_contexts\_mapper | No       | Callable\[\[TestCaseType, OutputType], str] | Map your test case and output to the retrieved contexts passed to Ragas.                                                                                                                                                                                                                                                                                                                                                                                     |
| reference\_contexts\_mapper | No       | Callable\[\[TestCaseType], str]             | Map your test case to the reference contexts passed to Ragas.                                                                                                                                                                                                                                                                                                                                                                                                |

<Note>
  Individual Ragas evaluators require different parameters.
  You can find sample implementations for each of the Ragas evaluators [here](https://github.com/autoblocksai/python-sdk/blob/main/tests/autoblocks/test_ragas_evaluators.py).
</Note>

<CodeGroup>
  ```python Python
  from dataclasses import dataclass

  from langchain_openai import ChatOpenAI
  from langchain_openai import OpenAIEmbeddings
  from ragas.embeddings import LangchainEmbeddingsWrapper  # type: ignore[import-untyped]
  from ragas.llms import LangchainLLMWrapper  # type: ignore[import-untyped]

  from autoblocks.testing.evaluators import BaseRagasResponseRelevancy
  from autoblocks.testing.models import BaseTestCase
  from autoblocks.testing.models import Threshold
  from autoblocks.testing.run import run_test_suite
  from autoblocks.testing.util import md5

  @dataclass
  class TestCase(BaseTestCase):
      question: str
      expected_answer: str

      def hash(self) -> str:
          return md5(self.question)
      
  @dataclass
  class Output:
      answer: str
      contexts: list[str]

  # You can use any of the Ragas evaluators listed here:
  # https://docs.autoblocks.ai/testing/offline-evaluations#out-of-box-evaluators-ragas
  class ResponseRelevancy(BaseRagasResponseRelevancy[TestCase, Output]):
      id = "response-relevancy"
      threshold = Threshold(gte=1)
      llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
      embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

      def user_input_mapper(self, test_case: TestCase, output: Output) -> str:
          return test_case.question

      def response_mapper(self, output: Output) -> str:
          return output.answer

      def retrieved_contexts_mapper(self, test_case: TestCase, output: Output) -> list[str]:
          return output.contexts

  run_test_suite(
      id="my-test-suite",
      test_cases=[
          TestCase(
              question="How tall is the Eiffel Tower?",
              expected_answer="300 meters"
          )
      ],
      fn=lambda test_case: Output(
          answer="300 meters",
          contexts=["The Eiffel tower stands 300 meters tall."],
      ),
      evaluators=[ResponseRelevancy()],
  )
  ```
</CodeGroup>