Evaluators

​

Prerequisites

Before using evaluators, you must configure your OpenAI API key in the settings. This key is required for the LLM-based evaluation functionality.

​

LLM as a Judge

​

What is an LLM Evaluator?

An LLM evaluator uses a large language model to assess your agent’s performance by analyzing conversation transcripts. The evaluator:

• Reviews the entire conversation
• Evaluates against specified criteria
• Provides a pass/fail result
• Explains the reasoning behind its decision

​

Creating an Evaluator

​

Define Success Criteria

Clearly specify what constitutes a successful interaction. For example:

• “The agent should confirm the appointment date and time”
• “The agent must verify the caller’s name”
• “The agent should handle interruptions politely”
• “The agent must not share sensitive information”

​

Example Evaluation Criteria

{
    "name": "Appointment Confirmation Check",
    "criteria": [
        "Agent must confirm the appointment date",
        "Agent must verify patient identity",
        "Agent should maintain professional tone",
        "Agent must handle any scheduling conflicts appropriately"
    ]
}

​

Best Practices

​

Writing Effective Criteria

1. Be Specific

   • Use clear, measurable objectives
   • Avoid ambiguous language
   • Include specific requirements

2. Focus on Key Behaviors

   • Identify critical success factors
   • Prioritize important interactions
   • Define must-have elements

3. Consider Edge Cases

   • Include criteria for handling interruptions
   • Address potential misunderstandings
   • Cover error scenarios

​

Example Scenarios

Basic Appointment Confirmation

{
    "name": "Basic Confirmation",
    "criteria": [
        "Verify appointment date and time",
        "Confirm patient name",
        "End call professionally"
    ]
}

Complex Medical Scheduling

{
    "name": "Medical Scheduling",
    "criteria": [
        "Verify patient identity securely",
        "Confirm appointment type and duration",
        "Check insurance information",
        "Handle scheduling conflicts",
        "Provide preparation instructions"
    ]
}

​

Understanding Results

​

Evaluation Output

The LLM evaluator provides:

• A pass/fail status
• A reason explaining the decision

​

Example Output

{
    "status": "fail",
    "reason": "The agent failed to verify the patient's identity before sharing appointment details"
}

​

Tips for Success

1. Iterate on Criteria

   • Start with basic requirements
   • Test with different scenarios
   • Refine based on results

2. Balance Strictness

   • Set reasonable expectations
   • Account for natural conversation flow
   • Consider multiple valid approaches

3. Review and Adjust

   • Monitor evaluation results
   • Identify patterns in failures
   • Update criteria as needed

​

Webhook Evaluators

​

What is a Webhook Evaluator?

A webhook evaluator allows you to implement custom evaluation logic by hosting your own evaluation endpoint. This gives you complete control over the evaluation process and allows for complex, domain-specific evaluation criteria.

​

Webhook Payload Structure

The webhook will receive a JSON payload containing:

• Input details about the scenario, persona, and data fields
• Output containing the conversation transcript

Example payload:

{
  "input": {
    "scenarioName": "Patient Appointment Confirmation",
    "scenarioDescription": "Test how well the agent confirms patient identity and appointment details",
    "personaName": "Impatient Caller",
    "personaDescription": "In a hurry, frequently interrupts, and expresses urgency throughout the call.",
    "edgeCases": [],
    "dataFields": [
      {
        "name": "firstName",
        "description": "Patient's first name",
        "example": "John"
      },
      {
        "name": "lastName",
        "description": "Patient's last name",
        "example": "Doe"
      },
      {
        "name": "dateOfBirth",
        "description": "Patient's date of birth",
        "example": "1990-01-01"
      },
      {
        "name": "appointmentTime",
        "description": "Scheduled appointment time",
        "example": "2024-03-27T14:30:00Z"
      }
    ]
  },
  "output": {
    "messages": [
      {
        "id": "item_BPE8uM8EiTE44GBo6Eewd",
        "timestamp": "2025-04-22T20:04:25.051Z",
        "role": "user",
        "roleLabel": "Your Agent",
        "content": "Hello, how are you?"
      },
      // ... conversation messages ...
    ]
  }
}

​

Implementing a Webhook Evaluator

You can host your webhook evaluator using services like Val Town, which provides a simple way to deploy and run JavaScript functions as webhooks.

Example implementation using Val Town:

// Example webhook evaluator hosted on Val Town
// https://www.val.town/x/autoblocks/Autoblocks_Webhook_Evaluator

import { Evaluation } from "npm:@autoblocks/client/testing";

export default async function httpHandler(request: Request): Promise<Response> {
  if (request.method !== "POST") {
    return Response.json({ message: "Invalid method." }, {
      status: 400,
    });
  }
  try {
    const body = await request.json();

    // Analyze the messages in the body
    // Create an evaluation object

    const response: Evaluation = {
      // Add your score between 0 and 1
      score: 1,
      // Use threshold to detrmine if the evaluation passed or failed.
      threshold: {
        gte: 1,
      },
      metadata: {
        reason: "Add in your reason here.",
      },
    };
    return Response.json(response);
  } catch (e) {
    return Response.json({ message: "Could not evaluate." }, {
      status: 500,
    });
  }
}

​

Using SDKs for Types

You can use our official SDKs to ensure correct types for the return value:

• JavaScript SDK - Use the Evaluation class in the @autoblocks/client/testing package
• Python SDK - Use the Evaluation dataclass in the autoblocks.testing.models package

import type { Evaluation } from "@autoblocks/client/testing";

// implement your evaluation logic here

const evaluation: Evaluation = {
  score: 1,
  threshold: {
    gte: 1,
  },
};

​

Best Practices

1. Error Handling

   • Implement proper error handling
   • Return meaningful error messages
   • Log evaluation failures

2. Performance

   • Keep evaluation logic efficient
   • Handle timeouts appropriately
   • Cache expensive computations

3. Testing

   • Test with various scenarios
   • Verify edge cases
   • Monitor evaluation consistency

4. Security

   • Secure your endpoint with authentication headers
   • Validate incoming requests
   • Use environment variables for sensitive data