​
Evaluators Overview

Autoblocks Evaluators provide a comprehensive system for assessing the quality and performance of your AI applications. They enable you to define custom evaluation criteria and integrate them seamlessly into your development workflow.

​
Key Features

​
Flexible Evaluation Types

  • Rule-based evaluators for simple checks
  • LLM-based evaluators for complex assessments
  • Webhook evaluators for custom logic
  • Out-of-box evaluators for common use cases

​
Integration Options

  • TypeScript and Python SDK support
  • UI-based evaluator creation
  • CLI integration
  • CI/CD pipeline support

​
Rich Evaluation Capabilities

  • Custom scoring logic
  • Threshold-based pass/fail
  • Detailed evaluation metadata
  • Evaluation history tracking

​
Getting Started

Choose your preferred language to begin:

​
Core Concepts

​
Evaluator Types

Different approaches to evaluation:
  • Rule-based evaluators for simple checks
  • LLM judges for complex assessments
  • Webhook evaluators for custom logic
  • Out-of-box evaluators for common use cases

​
Evaluation Components

Key elements of an evaluator:
  • Unique identifier
  • Scoring logic
  • Threshold configuration
  • Metadata and documentation

​
Integration Methods

Ways to use evaluators:
  • SDK integration
  • UI-based creation
  • CLI execution
  • CI/CD pipeline

​
Next Steps