Autoblocks

What to Track

This guide explains the most popular event types you should send to Autoblocks to maximize the platform's value.

While you can track any type of event related to your AI product in Autoblocks, these are the core events we recommend starting with:

  1. User Input
  2. AI Requests
  3. AI Responses
  4. User Feedback
  5. User Actions

User Input

You will want to track what a user does to trigger an AI request. This should be the first event in your trace. Below is an example of a user asking a support chatbot a simple question.

{
  "message": "user.input",
  "traceId": "my-trace-id",
  "properties": {
    "input": "How do I sign up?",
    "source": "support-chatbot"
  }
}

AI Requests

You'll want to track requests to your AI models as well as the parameters that were used to make the request. Below is an example of an OpenAI request.

{
  "message": "ai.request",
  "traceId": "my-trace-id",
  "properties": {
    "provider": "openai",
    "prompt_id": "chatbot-user-prompt",
    "prompt_version": 1,
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content":
          "You are a helpful assistant. You answer questions about a software product named Acme.",
      },
      {
        "role": "user",
        "content": "How do I sign up?",
      },
    ],
    "temperature": 0.7,
    "top_p": 1,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "stream": false,
    "n": 1
  }
}

AI Responses

You can track the entire response from your AI models or just a few fields you care about. Below is an example of a few important response properties you can track from an OpenAI response.

{
  "message": "ai.response",
  "traceId": "my-trace-id",
  "properties": {
    "latencyMs": "6126",
    "provider": "openai",
    "choices": [
      {
        "finish_reason": "stop",
        "index": "0",
        "message": {
          "content": "To sign up for Acme, you can follow these steps:",
          "role": "assistant"
        }
      }
    ],
    "model": "gpt-3.5-turbo",
    "usage": {
      "completion_tokens": "222",
      "prompt_tokens": "63",
      "total_tokens": "285"
    }
  }
}

You can also track multiple requests in your chain using the same event structure. Simply change the name of the event message to ai.intermediate.request or a similar relevant name.

User Feedback

You can also add user feedback to your trace. Below is an example showcasing user feedback with a score of 3 out of 5. You can learn more about user feedback in the User Feedback guide.

{
  "message": "user.feedback",
  "traceId": "my-trace-id",
  "properties": {
    "score": 3,
    "maxScore": 5
  }
}

User Actions

You may also want to track what the user does with the AI response. This information will be useful down the road when you want to fine-tune a model that is leading to better user outcomes. Below is an example tracking if a user bookmarked the AI response.

{
  "message": "user.bookmark-response",
  "traceId": "my-trace-id"
}