​
TypeScript Quick Start

This guide will help you get started with creating and using evaluators in TypeScript.

​
Installation

First, install the Autoblocks client:

npm install @autoblocks/client

​
Creating an Evaluator

Let’s create a simple evaluator that checks if a response contains a specific substring:

import { BaseTestEvaluator, Evaluation } from '@autoblocks/client/testing';

interface MyTestCase {
  input: string;
  expectedSubstring: string;
}

class HasSubstring extends BaseTestEvaluator<MyTestCase, string> {
  id = 'has-substring';

  evaluateTestCase(args: { testCase: MyTestCase; output: string }): Evaluation {
    const score = args.output.includes(args.testCase.expectedSubstring) ? 1 : 0;
    return {
      score,
      threshold: { gte: 1 },
    };
  }
}

​
Using an LLM Judge

For more complex evaluations, you can use an LLM as a judge:

import { BaseLLMJudge, Evaluation } from '@autoblocks/client/testing';

class IsProfessionalTone extends BaseLLMJudge<MyTestCase, string> {
  id = 'is-professional-tone';
  maxConcurrency = 2;

  prompt = `Please evaluate the provided text for its professionalism in the context of formal communication.
Consider the following criteria in your assessment:

Tone and Style: Respectful, objective, and appropriately formal tone without bias or excessive emotionality.
Grammar and Punctuation: Correct grammar, punctuation, and capitalization.
Based on these criteria, provide a binary response where:

0 indicates the text does not maintain a professional tone.
1 indicates the text maintains a professional tone.
No further explanation or summary is required; just provide the number that represents your assessment.`;

  private async scoreContent(content: string): Promise<number> {
    // Your LLM call implementation here
    return 1;
  }

  async evaluateTestCase(args: {
    testCase: MyTestCase;
    output: string;
  }): Promise<Evaluation> {
    const score = await this.scoreContent(args.output);
    return { score };
  }
}

​
Using Out of Box Evaluators

Autoblocks provides several out-of-box evaluators that you can use directly:

import { BaseAccuracy } from '@autoblocks/client/testing';

class Accuracy extends BaseAccuracy<MyTestCase, string> {
  id = 'accuracy';

  outputMapper(args: { output: string }): string {
    return args.output;
  }

  expectedOutputMapper(args: { testCase: MyTestCase }): string {
    return args.testCase.expectedOutput;
  }
}

​
Running Evaluations

You can run evaluations using the test suite:

import { runTestSuite } from '@autoblocks/client/testing';

runTestSuite<MyTestCase, string>({
  id: 'my-test-suite',
  testCases: [
    {
      input: 'hello world',
      expectedOutput: 'hello world',
    },
  ],
  testCaseHash: ['input'],
  fn: ({ testCase }) => testCase.input,
  evaluators: [new Accuracy()],
});

​
Next Steps