> ## Documentation Index
> Fetch the complete documentation index at: https://docs.autoblocks.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluators

> Learn how to create and use LLM evaluators to assess your AI agent's performance

## Prerequisites

Before using evaluators, you must configure your OpenAI API key in the settings. This key is required for the LLM-based evaluation functionality.

## LLM as a Judge

### What is an LLM Evaluator?

An LLM evaluator uses a large language model to assess your agent's performance by analyzing conversation transcripts. The evaluator:

* Reviews the entire conversation
* Evaluates against specified criteria
* Provides a pass/fail result
* Explains the reasoning behind its decision

### Creating an Evaluator

#### Define Success Criteria

Clearly specify what constitutes a successful interaction. For example:

* "The agent should confirm the appointment date and time"
* "The agent must verify the caller's name"
* "The agent should handle interruptions politely"
* "The agent must not share sensitive information"

#### Example Evaluation Criteria

```json
{
    "name": "Appointment Confirmation Check",
    "criteria": [
        "Agent must confirm the appointment date",
        "Agent must verify patient identity",
        "Agent should maintain professional tone",
        "Agent must handle any scheduling conflicts appropriately"
    ]
}
```

### Best Practices

#### Writing Effective Criteria

1. **Be Specific**
   * Use clear, measurable objectives
   * Avoid ambiguous language
   * Include specific requirements

2. **Focus on Key Behaviors**
   * Identify critical success factors
   * Prioritize important interactions
   * Define must-have elements

3. **Consider Edge Cases**
   * Include criteria for handling interruptions
   * Address potential misunderstandings
   * Cover error scenarios

#### Example Scenarios

**Basic Appointment Confirmation**

```json
{
    "name": "Basic Confirmation",
    "criteria": [
        "Verify appointment date and time",
        "Confirm patient name",
        "End call professionally"
    ]
}
```

**Complex Medical Scheduling**

```json
{
    "name": "Medical Scheduling",
    "criteria": [
        "Verify patient identity securely",
        "Confirm appointment type and duration",
        "Check insurance information",
        "Handle scheduling conflicts",
        "Provide preparation instructions"
    ]
}
```

### Understanding Results

#### Evaluation Output

The LLM evaluator provides:

* A pass/fail status
* A reason explaining the decision

#### Example Output

```json
{
    "status": "fail",
    "reason": "The agent failed to verify the patient's identity before sharing appointment details"
}
```

### Tips for Success

1. **Iterate on Criteria**
   * Start with basic requirements
   * Test with different scenarios
   * Refine based on results

2. **Balance Strictness**
   * Set reasonable expectations
   * Account for natural conversation flow
   * Consider multiple valid approaches

3. **Review and Adjust**
   * Monitor evaluation results
   * Identify patterns in failures
   * Update criteria as needed

## Webhook Evaluators

### What is a Webhook Evaluator?

A webhook evaluator allows you to implement custom evaluation logic by hosting your own evaluation endpoint. This gives you complete control over the evaluation process and allows for complex, domain-specific evaluation criteria.

### Webhook Payload Structure

The webhook will receive a JSON payload containing:

* Input details about the scenario, persona, and data fields
* Output containing the conversation transcript

Example payload:

```json [expandable]
{
  "input": {
    "scenarioName": "Patient Appointment Confirmation",
    "scenarioDescription": "Test how well the agent confirms patient identity and appointment details",
    "personaName": "Impatient Caller",
    "personaDescription": "In a hurry, frequently interrupts, and expresses urgency throughout the call.",
    "edgeCases": [],
    "dataFields": [
      {
        "name": "firstName",
        "description": "Patient's first name",
        "example": "John"
      },
      {
        "name": "lastName",
        "description": "Patient's last name",
        "example": "Doe"
      },
      {
        "name": "dateOfBirth",
        "description": "Patient's date of birth",
        "example": "1990-01-01"
      },
      {
        "name": "appointmentTime",
        "description": "Scheduled appointment time",
        "example": "2024-03-27T14:30:00Z"
      }
    ]
  },
  "output": {
    "messages": [
      {
        "id": "item_BPE8uM8EiTE44GBo6Eewd",
        "timestamp": "2025-04-22T20:04:25.051Z",
        "role": "user",
        "roleLabel": "Your Agent",
        "content": "Hello, how are you?"
      },
      // ... conversation messages ...
    ]
  }
}
```

### Implementing a Webhook Evaluator

You can host your webhook evaluator using services like [Val Town](https://www.val.town/x/autoblocks/Autoblocks_Webhook_Evaluator), which provides a simple way to deploy and run JavaScript functions as webhooks.

Example implementation using Val Town:

```javascript
// Example webhook evaluator hosted on Val Town
// https://www.val.town/x/autoblocks/Autoblocks_Webhook_Evaluator

import { Evaluation } from "npm:@autoblocks/client/testing";

export default async function httpHandler(request: Request): Promise<Response> {
  if (request.method !== "POST") {
    return Response.json({ message: "Invalid method." }, {
      status: 400,
    });
  }
  try {
    const body = await request.json();

    // Analyze the messages in the body
    // Create an evaluation object

    const response: Evaluation = {
      // Add your score between 0 and 1
      score: 1,
      // Use threshold to detrmine if the evaluation passed or failed.
      threshold: {
        gte: 1,
      },
      metadata: {
        reason: "Add in your reason here.",
      },
    };
    return Response.json(response);
  } catch (e) {
    return Response.json({ message: "Could not evaluate." }, {
      status: 500,
    });
  }
}
```

### Using SDKs for Types

You can use our official SDKs to ensure correct types for the return value:

* [JavaScript SDK](https://github.com/autoblocksai/javascript-sdk) - Use the `Evaluation` class in the `@autoblocks/client/testing` package
* [Python SDK](https://github.com/autoblocksai/python-sdk) - Use the `Evaluation` dataclass in the `autoblocks.testing.models` package

<CodeGroup>
  ```typescript TypeScript
  import type { Evaluation } from "@autoblocks/client/testing";

  // implement your evaluation logic here

  const evaluation: Evaluation = {
    score: 1,
    threshold: {
      gte: 1,
    },
  };
  ```

  ```python Python
  from autoblocks.testing.models import Evaluation
  from autoblocks.testing.models import Threshold

  # implement your evaluation logic here

  evaluation = Evaluation(
    score=1,
    threshold=Threshold(gte=1),
  )
  ```
</CodeGroup>

### Best Practices

1. **Error Handling**
   * Implement proper error handling
   * Return meaningful error messages
   * Log evaluation failures

2. **Performance**
   * Keep evaluation logic efficient
   * Handle timeouts appropriately
   * Cache expensive computations

3. **Testing**
   * Test with various scenarios
   * Verify edge cases
   * Monitor evaluation consistency

4. **Security**
   * Secure your endpoint with authentication headers
   * Validate incoming requests
   * Use environment variables for sensitive data