Evaluators
Overview
Evaluators
Overview
Learn about Autoblocks Evaluators and how they help you assess your AI applications.
Evaluators Overview
Autoblocks Evaluators provide a comprehensive system for assessing the quality and performance of your AI applications. They enable you to define custom evaluation criteria and integrate them seamlessly into your development workflow.
Key Features
Flexible Evaluation Types
- Rule-based evaluators for simple checks
- LLM-based evaluators for complex assessments
- Webhook evaluators for custom logic
- Out-of-box evaluators for common use cases
Integration Options
- TypeScript and Python SDK support
- UI-based evaluator creation
- CLI integration
- CI/CD pipeline support
Rich Evaluation Capabilities
- Custom scoring logic
- Threshold-based pass/fail
- Detailed evaluation metadata
- Evaluation history tracking
Getting Started
Choose your preferred language to begin:
Core Concepts
Evaluator Types
Different approaches to evaluation:
- Rule-based evaluators for simple checks
- LLM judges for complex assessments
- Webhook evaluators for custom logic
- Out-of-box evaluators for common use cases
Evaluation Components
Key elements of an evaluator:
- Unique identifier
- Scoring logic
- Threshold configuration
- Metadata and documentation
Integration Methods
Ways to use evaluators:
- SDK integration
- UI-based creation
- CLI execution
- CI/CD pipeline