​
Understanding Results Structure

​
Executions vs Test Runs

  • Executions: Individual simulation runs with their specific results
  • Test Runs: Groups of executions bundled together for comparison over time
    • Compare performance across different scenarios
    • Track improvements between iterations
    • Analyze patterns across multiple runs

​
Reviewing Executions

​
Execution Details

Each execution provides detailed information about:

  • Timestamp of the run
  • Duration of the conversation
  • Tokens used
  • Input/Output pairs
  • Pass/Fail status
  • Evaluation results

​
Transcript Review

Review conversations in detail with:

  • Complete conversation transcript
  • Audio playback of the interaction
  • Turn-by-turn message analysis

​
Performance Metrics

Track important metrics including:

  • Response times
  • Token usage
  • Success rates
  • Evaluation results
  • Overall pass rates

​
Search Syntax

Use powerful search operators to find specific executions:

String Search:

  • field:value - Contains search (e.g. source:aws, input:hello)
  • field!:value - Not contains search (e.g. source!:aws)
  • field=value - Exact match (e.g. source=aws)
  • field!=value - Not equals (e.g. source!=aws)
  • field is:empty - Check for empty values

Numeric Search:

  • duration>100 - Greater than
  • duration<500 - Less than
  • duration>=100 - Greater than or equal
  • duration<=500 - Less than or equal
  • duration=100 - Exact match

Free Text Search:

  • Simple text search (e.g. hello) - Searches across all text fields
  • Use AND to combine terms (e.g. hello AND world)
  • Use OR for alternatives (e.g. hello OR world)
  • Use parentheses for grouping (e.g. (hello OR world) AND test)

Combining Searches:

  • Mix and match different operators (e.g. source:aws AND environment=prod)
  • Use parentheses for complex queries (e.g. (duration>100 OR duration<=50) AND environment=prod)
  • Combine free text with specific field searches (e.g. hello AND source:aws)

Quoted Strings:

  • Use quotes for multi-word values (e.g. source:"aws lambda")

Available Fields:

  • source - Source of the execution
  • environment - Environment name
  • input - Input text
  • output - Output text
  • message - Run message
  • duration - Duration in milliseconds

​
Visualization Tools

​
Timeline View

  • Visual representation of execution timing
  • Identify patterns in response times
  • Spot anomalies or performance issues
  • Track conversation flow

​
Performance Graphs

  • Success rate trends
  • Duration distribution
  • Token usage patterns
  • Data capture accuracy over time

​
Comparison Tools

Compare executions across:

  • Different personas
  • Time periods
  • Edge cases
  • Data field variations

​
Best Practices

​
Analysis Workflow

  1. Review Overall Metrics

    • Check success rates
    • Analyze duration patterns
    • Review token usage

  2. Deep Dive into Failures

    • Examine failed executions
    • Review error patterns
    • Identify common issues

  3. Compare Across Runs

    • Track improvements
    • Identify regressions
    • Analyze pattern changes

  4. Document Findings

    • Note successful strategies
    • Document areas for improvement
    • Track action items

​
Tips for Effective Analysis

  • Start with high-level metrics
  • Use search to find specific patterns
  • Compare similar scenarios
  • Track improvements over time
  • Document unusual cases
  • Share insights with team