​
Core Concepts

Understanding these core concepts will help you leverage Autoblocks effectively.

​
Apps

Apps are the primary organizational units in Autoblocks, grouping related resources such as prompts, test cases, and evaluations. Apps typically align with specific use cases or business objectives, enabling clear organization, access control, usage tracking, and consistent evaluation standards.

​
Test Cases & Datasets

Test cases and datasets evaluate AI models against realistic scenarios:
  • Dynamic Generation: Automatically generate test cases from real user inputs.
  • Manual and Programmatic Creation: Create test cases manually or via SDK.
  • Versioning and Collaboration: Track changes and collaborate effectively.

​
Evaluations

Evaluations measure AI performance comprehensively:
  • Automated Checks: Programmatic validation of outputs.
  • SME-Aligned Metrics: Incorporate expert feedback directly into evaluation logic.
  • Continuous Improvement: Iterative enhancements based on evaluation outcomes.

​
Human Review

Integrate domain expertise seamlessly:
  • Structured Workflows: Efficiently gather and apply expert feedback.
  • Quality Assurance: Ensure outputs meet high standards before deployment.

​
Agent Simulation

Realistically test AI systems:
  • Scenario and Environment Simulation: Create realistic testing scenarios.
  • Edge Case Identification: Proactively discover and address potential issues.

​
Workflow Builder

Efficiently manage complex testing processes:
  • Visual Interface: Easily create and manage workflows.
  • Flexible Execution: Run workflows on Autoblocks infrastructure or your own.

​
Prompt Management

Optimize AI reliability through effective prompt management:
  • Version Control and A/B Testing: Continuously refine prompts.
  • Performance Monitoring: Track and enhance prompt effectiveness.

​
Tracing

Gain deep insights into AI behavior:
  • Comprehensive Tracking: Monitor interactions end-to-end.
  • Error Detection and Analytics: Quickly identify and resolve issues.

​
Integration

Seamlessly integrate Autoblocks into your existing workflows:
  • SDKs and APIs: Easy integration with Python, TypeScript, and REST APIs.
  • CI/CD and Monitoring: Integrate with your existing development and observability tools.