yao/agent/test/DESIGN_V2.md

Agent Test Framework V2 Design

Overview

This document describes the design for Agent Test Framework V2, which extends the existing testing capabilities with:

• Multi-turn conversations - Test agents across multiple interaction rounds
• Agent-driven testing - Use agents to generate test cases and simulate user responses
• Interactive testing - Human-in-the-loop testing mode

Quick Reference: Format Rules

Context	Format	Example
-i flag (CLI)	Prefix required	agents:workers.test.gen, scripts:tests.gen
JSONL assertion use	Prefix required	"use": "agents:workers.test.validator"
JSONL simulator.use	No prefix (agent only)	"use": "workers.test.user-sim"
--simulator flag	No prefix (agent only)	--simulator workers.test.user-sim
t.assert.Agent()	No prefix (method-bound)	t.assert.Agent(resp, "workers.test.validator", {...})

Problem Statement

Current single-turn testing cannot adequately test:

1. Conversational flows - Agents that guide users through multi-step processes
2. Confirmation dialogs - Agents that ask for user confirmation before actions
3. Clarification requests - Agents that ask follow-up questions when input is ambiguous
4. Stateful interactions - Agents that maintain context across multiple turns

Design Goals

1. Unified format - Single test case format that supports all modes
2. Flexible execution - Static, dynamic (simulator), and interactive modes
3. Graceful degradation - Skip tests when required input is unavailable
4. CI/CD compatible - Non-interactive mode for automated pipelines
5. Agent-driven - Both input generation and user simulation can be agent-powered

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                         yao agent test                                   │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  INPUT SOURCES (-i flag)                                                 │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │ JSONL File  │  │   Message   │  │ Generator   │  │ Interactive │     │
│  │ ./test.jsonl│  │ "Hello..."  │  │ agents:xxx  │  │ (stdin)     │     │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘     │
│         │                │                │                │            │
│         └────────────────┴────────────────┴────────────────┘            │
│                                   │                                      │
│                                   ▼                                      │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                        Test Case Parser                          │    │
│  │  - Single-turn: {input, assertions}                              │    │
│  │  - Multi-turn:  {turns: [{input, assertions}, ...]}              │    │
│  └─────────────────────────────────────────────────────────────────┘    │
│                                   │                                      │
│                                   ▼                                      │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                      Multi-Turn Executor                         │    │
│  │                                                                   │    │
│  │   ┌─────────────────────────────────────────────────────────┐    │    │
│  │   │ MODE SELECTION (based on test case fields)              │    │    │
│  │   │                                                          │    │    │
│  │   │   Has `turns`? ─────────────────▶ STATIC MODE            │    │    │
│  │   │   Has `simulator` + `checkpoints`? ──▶ DYNAMIC MODE      │    │    │
│  │   │   Neither? ─────────────────────▶ SINGLE-TURN (legacy)   │    │    │
│  │   └─────────────────────────────────────────────────────────┘    │    │
│  │                              │                                   │    │
│  │              ┌───────────────┴───────────────┐                   │    │
│  │              ▼                               ▼                   │    │
│  │   ┌─────────────────────┐       ┌─────────────────────────┐      │    │
│  │   │    STATIC MODE      │       │     DYNAMIC MODE        │      │    │
│  │   │                     │       │                         │      │    │
│  │   │ FOR each turn:      │       │ LOOP until terminated:  │      │    │
│  │   │  1. Send input      │       │  1. Simulator → input   │      │    │
│  │   │  2. Get response    │       │  2. Send to Agent       │      │    │
│  │   │  3. Run assertions  │       │  3. Check checkpoints   │      │    │
│  │   │  4. Continue/Fail   │       │  4. Check termination   │      │    │
│  │   │                     │       │                         │      │    │
│  │   │ All passed → PASS   │       │ All checkpoints → PASS  │      │    │
│  │   │ Any failed → FAIL   │       │ Timeout/Missing → FAIL  │      │    │
│  │   └─────────────────────┘       └─────────────────────────┘      │    │
│  │                                                                   │    │
│  └─────────────────────────────────────────────────────────────────┘    │
│                                   │                                      │
│                                   ▼                                      │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                         Assertions                               │    │
│  │  - Static: Per-turn assertions                                   │    │
│  │  - Dynamic: Checkpoint assertions (order-independent)            │    │
│  └─────────────────────────────────────────────────────────────────┘    │
│                                   │                                      │
│                                   ▼                                      │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                          Reporter                                │    │
│  │  - Console output                                                │    │
│  │  - JSONL output                                                  │    │
│  │  - Custom reporter agent                                         │    │
│  └─────────────────────────────────────────────────────────────────┘    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Core Challenge: Detecting "Awaiting Input" State

The key challenge is determining when an agent is waiting for user input vs. when it has completed its task.

Detection Strategies

Strategy 1: Explicit Declaration (Recommended)

Agent explicitly declares its state in the response:

{
  "content": "What is the expense amount?",
  "awaiting_input": true,
  "input_hint": "Enter amount, e.g., $3500"
}

Strategy 2: Finish Reason Analysis

Use the LLM's finish_reason to determine state:

• stop - Agent completed normally (may or may not need input)
• tool_calls - Agent is executing tools (not awaiting input)
• length - Response truncated (not awaiting input)

Strategy 3: Content Heuristics

Analyze response content for question patterns:

func looksLikeQuestion(content string) bool {
    patterns := []string{
        `\?$`,                    // Ends with question mark
        `(?i)^(what|how|when|where|which|who|please|could you)`,
        `(?i)(confirm|verify|proceed|continue)\?`,
    }
    for _, pattern := range patterns {
        if regexp.MustCompile(pattern).MatchString(content) {
            return true
        }
    }
    return false
}

Strategy 4: Tool-Based Detection

Certain tools indicate awaiting input:

func toolRequiresInput(toolCall ToolCall) bool {
    confirmationTools := []string{
        "request_confirmation",
        "ask_user",
        "get_user_input",
    }
    return contains(confirmationTools, toolCall.Name)
}

Recommended Approach: Hybrid Detection

Combine multiple strategies with priority:

func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) {
    // Priority 1: Explicit declaration
    if result.AwaitingInput {
        return true, "agent_declared"
    }

    // Priority 2: Tool-based detection
    for _, tc := range result.ToolCalls {
        if toolRequiresInput(tc) {
            return true, "tool_requires_confirmation"
        }
    }

    // Priority 3: Content heuristics
    if looksLikeQuestion(result.Content) {
        return true, "content_is_question"
    }

    return false, "completed"
}

Standard Agent Interface

All agent-driven features (generator, simulator, validator) use standard Yao Agent interfaces.

Using context.Options

The test framework uses context.Options to pass parameters, aligned with Assistant.Stream():

// context.Options - standard Yao Agent options
type Options struct {
    Skip      *Skip          `json:"skip,omitempty"`      // Skip history, trace, etc.
    Connector string         `json:"connector,omitempty"` // LLM connector to use
    Search    any            `json:"search,omitempty"`    // Search behavior control
    Mode      string         `json:"mode,omitempty"`      // Agent mode (chat, etc.)
    Metadata  map[string]any `json:"metadata,omitempty"`  // Custom metadata
}

Test Framework Usage

// Prepare options for agent invocation
options := &context.Options{
    Skip: &context.Skip{
        History: true,  // Don't save test messages to history
        Trace:   false, // Keep trace for debugging
    },
    Metadata: map[string]any{
        "test_mode": "validator",  // "generator" | "simulator" | "validator"
        "test_id":   "T001",
        "criteria":  "Response should be helpful",
        // ... other custom params from test config
    },
}

// Prepare context
ctx := context.New(parent, authorized, chatID)
ctx.Referer = "agent-test"

// Call agent with standard Stream API
assistant := agent.Get(agentID)
response, err := assistant.Stream(ctx, messages, options)

Test Case Options Field

Test cases can specify options to pass to the target agent or helper agents:

{
  "id": "T001",
  "input": "Hello",
  "options": {
    "connector": "openai-gpt4",
    "skip": {
      "history": true
    },
    "metadata": {
      "scenario": "edge-case"
    }
  }
}

Generator Mode

Used when -i agents:xxx is specified to generate test cases:

// Framework calls generator agent
options := &context.Options{
    Skip: &context.Skip{History: true},
    Metadata: map[string]any{
        "test_mode":          "generator",
        "target_agent":       "assistants.expense",
        "target_description": "Expense reimbursement assistant",
        "target_tools":       []string{"create_expense", "get_policy"},
        // From query params: agents:xxx?count=10&focus=edge-cases
        "count":      10,
        "focus":      "edge-cases",
        "complexity": "medium",
    },
}

Expected structured output:

{
  "cases": [
    {"id": "G001", "input": "...", "assertions": [...]},
    {"id": "G002", "input": "...", "assertions": [...]}
  ]
}

Simulator Mode

Used when test case has simulator config:

// Framework calls simulator agent for next user input
options := &context.Options{
    Skip: &context.Skip{History: true},
    Metadata: map[string]any{
        "test_mode":    "simulator",
        "test_id":      "T001",
        // From simulator.options.metadata in test case
        "persona":      "New employee",
        "goal":         "Submit expense report",
        // Runtime context
        "turn_number":  3,
        "max_turns":    10,
        "tool_results": map[string]any{...},
    },
}

// Messages include full conversation history
messages := conversationHistory

Expected structured output:

{
  "input": "The amount is $500",
  "goal_achieved": false,
  "reasoning": "Providing requested amount info"
}

Validator Mode

Used when assertion has type: "agent":

// Framework calls validator agent
options := &context.Options{
    Skip: &context.Skip{History: true},
    Metadata: map[string]any{
        "test_mode": "validator",
        "test_id":   "T001",
        // From assertion.metadata
        "criteria":        "Response should be helpful",
        "expected_intent": "answer question",
        "tone":            "professional",
    },
}

// Messages include agent response to validate
messages := []context.Message{
    {Role: "user", Content: "Original user question"},
    {Role: "assistant", Content: "Agent's response to validate"},
}

Expected structured output:

{
  "passed": true,
  "score": 0.95,
  "reason": "Response is helpful and addresses the question",
  "suggestions": []
}

Assertion Types

Static Assertions (Existing)

Type	Description	Example
contains	Response contains text	{"type": "contains", "value": "success"}
equals	Exact match	{"type": "equals", "value": "OK"}
regex	Regex pattern match	{"type": "regex", "pattern": "order-\\d+"}
json_path	JSONPath value check	{"type": "json_path", "path": "$.status", "value": "ok"}
tool_called	Tool was invoked	{"type": "tool_called", "name": "create_expense"}
type	Value type check	{"type": "type", "path": "$.count", "value": "number"}

Agent-Driven Assertions (New)

For fuzzy, semantic, or context-aware validation. Uses options aligned with context.Options:

{
  "type": "agent",
  "use": "agents:workers.test.validator",
  "options": {
    "connector": "openai-gpt4",
    "metadata": {
      "criteria": "Response should be helpful and answer the user's question",
      "expected_intent": "provide expense submission guidance",
      "tone": "professional and friendly"
    }
  }
}

Script Assertions (in JSONL)

For custom validation logic in JSONL test cases:

{
  "type": "script",
  "use": "scripts:tests.validate-expense",
  "options": {
    "metadata": {
      "min_amount": 100,
      "max_amount": 10000
    }
  }
}

Combined Assertions

Mix static and agent-driven assertions:

{
  "assertions": [
    {
      "type": "tool_called",
      "name": "create_expense"
    },
    {
      "type": "agent",
      "use": "agents:workers.test.validator",
      "options": {
        "metadata": {
          "criteria": "Confirmation message should include expense amount and be polite"
        }
      }
    }
  ]
}

Script Testing with Agent Assertions

Script tests can also use Agent-driven assertions via the t.assert.Agent() method.

API

// t.assert.Agent(response, agentID, options?) -> ValidatorResult
// agentID: Direct agent ID without prefix, e.g., "workers.test.validator"
interface ValidatorResult {
  passed: boolean;
  score?: number;
  reason: string;
  suggestions?: string[];
}

Usage in Script Tests

// tests/expense_test.ts
export function TestExpenseResponse(t: TestingT, ctx: Context) {
  // Call the agent being tested
  const response = Process("agents.expense.Stream", ctx, [
    { role: "user", content: "How do I submit an expense?" },
  ]);

  // Static assertions
  t.assert.NotNil(response);
  t.assert.Contains(response.content, "expense");

  // Agent-driven assertion - automatically fails test if validation fails
  t.assert.Agent(response.content, "workers.test.validator", {
    metadata: {
      criteria:
        "Response should explain the expense submission process clearly",
      expected_topics: ["receipt", "approval", "deadline"],
      tone: "helpful",
    },
  });
}

With Conversation Context

export function TestMultiTurnExpense(t: TestingT, ctx: Context) {
  const messages = [
    { role: "user", content: "I need to submit a travel expense" },
    { role: "assistant", content: "I'd be happy to help..." },
    { role: "user", content: "It's for a flight to Beijing, $2000" },
  ];

  const response = Process("agents.expense.Stream", ctx, messages);

  // Agent assertion with conversation context
  // Automatically fails test and logs suggestions if validation fails
  t.assert.Agent(response.content, "workers.test.validator", {
    metadata: {
      criteria:
        "Response should confirm the expense details and ask for receipt",
      conversation: messages,
    },
  });
}

Implementation

The t.assert.Agent() method internally:

1. Prepares context.Options with test_mode: "validator"
2. Calls the validator agent via Assistant.Stream()
3. Parses structured output (JSON)
4. Returns ValidatorResult

// In script_assert.go
func assertAgentMethod(iso *v8go.Isolate, t *TestingT, agentCtx *context.Context) *v8go.FunctionTemplate {
    return v8go.NewFunctionTemplate(iso, func(info *v8go.FunctionCallbackInfo) *v8go.Value {
        // Parse arguments: response, agentID, options
        response := args[0].String()
        agentID := args[1].String()  // Direct agent ID, e.g., "workers.test.validator"
        options := parseOptions(args[2])

        // Prepare validator options
        validatorOpts := &context.Options{
            Skip: &context.Skip{History: true},
            Metadata: map[string]any{
                "test_mode": "validator",
                ...options.Metadata,
            },
        }

        // Call validator agent
        assistant, _ := agent.Get(agentID)
        result, _ := assistant.Stream(agentCtx, messages, validatorOpts)

        // Parse and return result
        return toJsValue(parseValidatorResult(result))
    })
}

Test Case Format

Single-Turn (Existing)

{
  "id": "T001",
  "input": "Hello",
  "assertions": [
    {
      "type": "contains",
      "value": "Hi"
    }
  ]
}

Multi-Turn: Static Mode

For deterministic flows where you know the exact conversation sequence:

{
  "id": "T001",
  "name": "Expense Reimbursement - Happy Path",
  "mode": "static",
  "options": {
    "connector": "openai-gpt4",
    "skip": {
      "history": true
    }
  },
  "turns": [
    {
      "input": "I want to submit an expense report",
      "assertions": [
        {
          "type": "contains",
          "value": "type of expense"
        }
      ]
    },
    {
      "input": "Business travel to Beijing, $3500",
      "assertions": [
        {
          "type": "tool_called",
          "name": "create_expense"
        }
      ]
    },
    {
      "input": "Yes, confirm",
      "assertions": [
        {
          "type": "contains",
          "value": "submitted"
        }
      ]
    }
  ]
}

Characteristics:

• Fixed number of turns
• Each turn has specific input and assertions
• Test fails if any turn assertion fails
• Best for regression testing known flows

Multi-Turn: Dynamic Mode (Checkpoints)

For coverage testing where you care about functionality, not exact sequence:

{
  "id": "T002",
  "name": "Expense Submission Coverage",
  "mode": "dynamic",
  "simulator": {
    "use": "workers.test.user-simulator",
    "options": {
      "metadata": {
        "persona": "New employee unfamiliar with expense process",
        "goal": "Submit a $3500 travel expense"
      }
    }
  },
  "checkpoints": [
    {
      "id": "ask_type",
      "description": "Agent asks for expense type",
      "assertion": {
        "type": "contains",
        "value": "type"
      }
    },
    {
      "id": "call_create",
      "description": "Agent calls create_expense tool",
      "assertion": {
        "type": "tool_called",
        "name": "create_expense"
      }
    },
    {
      "id": "confirm_submit",
      "description": "Agent confirms submission",
      "assertion": {
        "type": "contains",
        "value": "submitted"
      }
    }
  ],
  "max_turns": 10,
  "timeout": "2m"
}

Characteristics:

• Simulator drives the conversation
• Checkpoints are verified across all turns (order-independent by default)
• Test passes when ALL checkpoints are reached
• Test fails if max_turns/timeout reached before all checkpoints
• Best for functional coverage testing

Checkpoints with Order Constraints

When checkpoints must occur in a specific order:

{
  "checkpoints": [
    {
      "id": "ask_type",
      "description": "Agent asks for expense type",
      "assertion": {
        "type": "contains",
        "value": "type"
      }
    },
    {
      "id": "call_create",
      "description": "Agent calls create_expense",
      "after": [
        "ask_type"
      ],
      "assertion": {
        "type": "tool_called",
        "name": "create_expense"
      }
    },
    {
      "id": "confirm_submit",
      "description": "Agent confirms submission",
      "after": [
        "call_create"
      ],
      "assertion": {
        "type": "contains",
        "value": "submitted"
      }
    }
  ]
}

Checkpoints with Agent Validation

Use Agent-driven assertions for semantic validation:

{
  "checkpoints": [
    {
      "id": "helpful_guidance",
      "description": "Agent provides helpful expense guidance",
      "assertion": {
        "type": "agent",
        "use": "agents:workers.test.validator",
        "options": {
          "metadata": {
            "criteria": "Response explains expense process clearly and professionally"
          }
        }
      }
    },
    {
      "id": "tool_called",
      "description": "Agent creates expense record",
      "assertion": {
        "type": "tool_called",
        "name": "create_expense"
      }
    }
  ]
}

Dynamic Mode Termination

Condition	Result	Description
All checkpoints reached	✅ PASSED	All functionality verified
Agent completes, checkpoints missing	❌ FAILED	Missing coverage
max_turns exceeded	❌ FAILED	Timeout - flow too long
timeout exceeded	❌ FAILED	Time limit reached
Checkpoint assertion fails	❌ FAILED	Functionality broken
Simulator error	⚠️ SKIPPED	Cannot continue

Field Descriptions

Field	Type	Required	Description
id	string	Yes	Unique test identifier
name	string	No	Human-readable test name
mode	string	No	"static" (default) or "dynamic"
options	object	No	context.Options passed to target agent
Static Mode Fields
turns	array	Yes (static)	Static turn definitions
turns[].input	string	Yes	User input for this turn
turns[].assertions	array	No	Assertions for this turn's response
turns[].options	object	No	Per-turn options override
Dynamic Mode Fields
simulator	object	Yes (dynamic)	User simulator configuration
simulator.use	string	Yes	Simulator agent ID (no prefix)
simulator.options	object	No	context.Options passed to simulator
checkpoints	array	Yes (dynamic)	Functionality checkpoints to verify
checkpoints[].id	string	Yes	Unique checkpoint identifier
checkpoints[].description	string	No	Human-readable description
checkpoints[].assertion	object	Yes	Assertion to verify
checkpoints[].after	array	No	Checkpoint IDs that must occur first
max_turns	int	No	Maximum turns before timeout (default: 20)
timeout	string	No	Maximum time (default: "5m")
Shared Fields
interactive	object	No	Interactive mode configuration
interactive.enabled	bool	No	Enable human input (default: false)
interactive.timeout	string	No	Timeout for human input (default: "5m")

Execution Modes

Static Mode

Uses predefined turns array. Best for regression testing known flows.

┌─────────────────────────────────────────────────────────┐
│                    Static Mode Flow                      │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  FOR each turn in turns[]:                               │
│    1. Send turn.input to Agent                           │
│    2. Get Agent response                                 │
│    3. Run turn.assertions                                │
│       ├─ PASS → Continue to next turn                    │
│       └─ FAIL → Test FAILED, stop                        │
│                                                          │
│  All turns completed → Test PASSED                       │
│                                                          │
└─────────────────────────────────────────────────────────┘

Dynamic Mode (Checkpoints)

Uses simulator + checkpoints. Best for coverage testing functionality.

┌─────────────────────────────────────────────────────────┐
│                   Dynamic Mode Flow                      │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  Initialize: pending_checkpoints = all checkpoints       │
│                                                          │
│  LOOP (until terminated):                                │
│    1. Simulator generates user input                     │
│    2. Send input to Agent                                │
│    3. Get Agent response                                 │
│    4. Check response against pending_checkpoints         │
│       └─ If matched → Move to reached_checkpoints        │
│    5. Check termination conditions:                      │
│       ├─ All checkpoints reached → PASSED                │
│       ├─ Agent completed, missing checkpoints → FAILED   │
│       ├─ max_turns exceeded → FAILED                     │
│       └─ timeout exceeded → FAILED                       │
│                                                          │
└─────────────────────────────────────────────────────────┘

Interactive Mode

For debugging, human can provide input when agent awaits:

# Enable with --interactive flag
yao agent test -i ./tests.jsonl --interactive

In interactive mode:

• Static mode: Human can override any turn input
• Dynamic mode: Human can replace simulator for specific turns

Mode Selection

Has turns?	Has simulator + checkpoints?	Mode
Yes	No	Static
No	Yes	Dynamic
Yes	Yes	❌ Invalid (choose one)
No	No	Single-turn (legacy)

Example: Static vs Dynamic

Same feature, different testing approaches:

// Static Mode - Exact sequence testing
{
  "id": "expense-static",
  "mode": "static",
  "turns": [
    {"input": "Submit expense", "assertions": [{"type": "contains", "value": "type"}]},
    {"input": "Travel, $500", "assertions": [{"type": "tool_called", "name": "create_expense"}]},
    {"input": "Confirm", "assertions": [{"type": "contains", "value": "submitted"}]}
  ]
}

// Dynamic Mode - Coverage testing
{
  "id": "expense-dynamic",
  "mode": "dynamic",
  "simulator": {"use": "workers.test.user-sim", "options": {"metadata": {"goal": "Submit $500 expense"}}},
  "checkpoints": [
    {"id": "ask", "assertion": {"type": "contains", "value": "type"}},
    {"id": "create", "assertion": {"type": "tool_called", "name": "create_expense"}},
    {"id": "done", "assertion": {"type": "contains", "value": "submitted"}}
  ],
  "max_turns": 10
}

Execution Flow

Static Mode Flow

┌─────────────────────────────────────────────────────────────────┐
│                    Static Mode Execution                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  INITIALIZE:                                                     │
│    - Load turns[] from test case                                 │
│    - Set current_turn = 0                                        │
│                              ↓                                   │
│  FOR each turn in turns[]:                                       │
│    ┌─────────────────────────────────────────────────────────┐   │
│    │ 1. Get input from turns[current_turn].input             │   │
│    │                         ↓                               │   │
│    │ 2. Send input to Agent                                  │   │
│    │                         ↓                               │   │
│    │ 3. Get Agent response                                   │   │
│    │                         ↓                               │   │
│    │ 4. Run turns[current_turn].assertions                   │   │
│    │           │                                             │   │
│    │           ├─ PASS → Continue to next turn               │   │
│    │           └─ FAIL → Test FAILED, stop                   │   │
│    └─────────────────────────────────────────────────────────┘   │
│                              ↓                                   │
│  All turns completed → Test PASSED                               │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Dynamic Mode Flow

┌─────────────────────────────────────────────────────────────────┐
│                    Dynamic Mode Execution                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  INITIALIZE:                                                     │
│    - pending_checkpoints = all checkpoints                       │
│    - reached_checkpoints = []                                    │
│    - turn_count = 0                                              │
│    - start_time = now()                                          │
│                              ↓                                   │
│  LOOP:                                                           │
│    ┌─────────────────────────────────────────────────────────┐   │
│    │ 1. Call Simulator Agent → Get user input                │   │
│    │    (pass: persona, goal, conversation history)          │   │
│    │                         ↓                               │   │
│    │ 2. Send input to Target Agent                           │   │
│    │                         ↓                               │   │
│    │ 3. Get Agent response                                   │   │
│    │                         ↓                               │   │
│    │ 4. Check response against pending_checkpoints           │   │
│    │    FOR each pending checkpoint:                         │   │
│    │      - Run checkpoint.assertion                         │   │
│    │      - If PASS and `after` satisfied → move to reached  │   │
│    │                         ↓                               │   │
│    │ 5. Check termination conditions:                        │   │
│    │    ├─ pending_checkpoints empty?                        │   │
│    │    │   → Test PASSED ✅                                 │   │
│    │    │                                                    │   │
│    │    ├─ Agent completed (not awaiting)?                   │   │
│    │    │   → Test FAILED ❌ (missing checkpoints)           │   │
│    │    │                                                    │   │
│    │    ├─ turn_count >= max_turns?                          │   │
│    │    │   → Test FAILED ❌ (turn limit)                    │   │
│    │    │                                                    │   │
│    │    ├─ now() - start_time > timeout?                     │   │
│    │    │   → Test FAILED ❌ (timeout)                       │   │
│    │    │                                                    │   │
│    │    └─ Otherwise → Continue loop                         │   │
│    └─────────────────────────────────────────────────────────┘   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Command Line Interface

Flags Reference

Flag	Long	Description
-i	--input	Input source: file path, message, or type:id reference
-n	--name	Target agent ID (the agent being tested)
-o	--output	Output file path for results
-c	--connector	Override connector for the target agent
-v	--verbose	Verbose output showing all turns
	--interactive	Enable human input when agent awaits
	--simulator	Default simulator agent ID (e.g., workers.test.user-sim)
	--timeout	Timeout per test case (default: 5m)
	--parallel	Number of parallel test cases
	--fail-fast	Stop on first failure
	--dry-run	Generate/parse tests without running

Input Sources (-i flag)

The -i flag supports multiple input sources with unified type:id format:

# 1. File path (default, no prefix needed)
yao agent test -i ./tests/multi-turn.jsonl

# 2. Direct message (no prefix, auto-detected as non-file)
yao agent test -i "Hello, how are you?" -n assistants.chat

# 3. Agent-generated test cases
yao agent test -i agents:workers.test.generator -n assistants.expense

# 4. Script-generated test cases
yao agent test -i scripts:tests.generate -n assistants.expense

# 5. With parameters (query string style)
yao agent test -i "agents:workers.test.generator?count=10&focus=edge-cases" -n assistants.expense

Input Type Prefixes

Prefix	Description	Example
(none)	File path or direct message	./tests.jsonl, "Hello"
agents:	Agent generates test cases	agents:workers.test.gen
scripts:	Script generates test cases	scripts:tests.generate

Input Format

[prefix:]<id>[?param1=value1&param2=value2]

• prefix - Input type: agents: or scripts: (optional, default is file/message)
• id - Agent ID or script ID
• ?params - Query parameters passed to generator

Generator Agent Interface

// Input to generator agent
interface GeneratorInput {
  target_agent: string; // Agent being tested (from -n flag)
  target_description?: string; // Agent's description/purpose
  target_tools?: Tool[]; // Agent's available tools
  count?: number; // Number of test cases to generate
  focus?: string; // Focus area: "happy-path", "edge-cases", "errors"
  complexity?: string; // "simple", "medium", "complex"
}

// Output from generator agent
interface GeneratorOutput {
  cases: TestCase[]; // Generated test cases
}

Example Generator Prompt

You are a test case generator for AI agents.

Target Agent: {{target_agent}}
Description: {{target_description}}
Available Tools: {{target_tools}}

Generate {{count}} test cases with focus on: {{focus}}

For each test case, provide:
- id: Unique identifier
- name: Descriptive name
- input: User message or turns array for multi-turn
- assertions: Expected behaviors to verify

Output as JSON array of test cases.

Complete Examples

# Basic: Run tests from file
yao agent test -i ./tests/expense.jsonl

# With target agent specified (required for message/agent input)
yao agent test -i "Help me file an expense" -n assistants.expense

# Agent generates tests, then runs them
yao agent test \
  -i "agents:workers.test.generator?count=20&focus=edge-cases" \
  -n assistants.expense

# Script generates tests
yao agent test \
  -i "scripts:tests.expense.generate?scenario=approval-flow" \
  -n assistants.expense

# Fully dynamic: Agent generates tests + Agent simulates user responses
yao agent test \
  -i "agents:workers.test.generator?count=10" \
  -n assistants.expense \
  --simulator workers.test.user-simulator

# Generate tests only, save to file (dry-run)
yao agent test \
  -i "agents:workers.test.generator?count=50" \
  -n assistants.expense \
  -o ./tests/generated.jsonl \
  --dry-run

# Interactive mode: human provides input when agent awaits
yao agent test -i ./tests/multi-turn.jsonl --interactive

# CI/CD mode: skip tests requiring human input
yao agent test -i ./tests/multi-turn.jsonl --skip-interactive

# Fail instead of skip when input unavailable
yao agent test -i ./tests/multi-turn.jsonl --on-missing-input=fail

# Verbose output showing all turns
yao agent test -i ./tests/multi-turn.jsonl -v

Output Format

Console Output

═══════════════════════════════════════════════════════════════
  Agent Test (Multi-Turn)
═══════════════════════════════════════════════════════════════
ℹ Agent: assistants.expense
ℹ Input: ./tests/expense-flow.jsonl (5 test cases)

───────────────────────────────────────────────────────────────
  Running Tests
───────────────────────────────────────────────────────────────

► [T001] Expense Reimbursement Flow (3 turns)
  ├─ Turn 1: "I want to submit an expense" → PASSED (2.1s)
  │          Agent: "What type of expense would you like to submit?"
  │          ✓ contains "type of expense"
  │
  ├─ Turn 2: "Business travel, $3500" → PASSED (3.2s)
  │          Agent: [tool: create_expense({amount: 3500, type: "travel"})]
  │          ✓ tool_called "create_expense"
  │
  ├─ Turn 3: "Yes, confirm" → PASSED (1.8s)
  │          Agent: "Expense submitted. Reference: EXP-2025-001"
  │          ✓ contains "submitted"
  │
  └─ Final Assertions: PASSED
     ✓ $.expense.status = "submitted"

► [T002] Large Expense Approval
  ├─ Turn 1: "Submit $100,000 equipment purchase" → PASSED (2.0s)
  │          Agent: "This requires manager approval. Please provide PO number."
  │
  ├─ Turn 2: SKIPPED
  │          Reason: Agent awaiting input, no next turn defined
  │          Agent asked: "Please provide PO number"
  │          Hint: Add more turns, use --interactive, or configure simulator
  │
  └─ Result: SKIPPED

► [T003] Dynamic Expense Flow (simulator: workers.test.user-sim)
  ├─ Turn 1: [Initial] "Help me file an expense" → PASSED (2.1s)
  ├─ Turn 2: [Simulated] "It's for client dinner, $250" → PASSED (2.8s)
  ├─ Turn 3: [Simulated] "Yesterday evening" → PASSED (2.2s)
  ├─ Turn 4: [Simulated] "Confirm" → PASSED (1.9s)
  │          Goal achieved: Expense submitted
  │
  └─ Final Assertions: PASSED

───────────────────────────────────────────────────────────────
  Summary
───────────────────────────────────────────────────────────────
  Total:   3 tests
  Passed:  2
  Failed:  0
  Skipped: 1

  Total turns: 10
  Avg turns/test: 3.3
  Total time: 18.1s

JSONL Output

{
  "id": "T001",
  "name": "Expense Reimbursement Flow",
  "status": "passed",
  "turns": [
    {
      "turn": 1,
      "input": "I want to submit an expense",
      "input_source": "static",
      "output": "What type of expense would you like to submit?",
      "awaiting_input": true,
      "assertions": [
        {
          "type": "contains",
          "value": "type of expense",
          "passed": true
        }
      ],
      "duration_ms": 2100
    },
    {
      "turn": 2,
      "input": "Business travel, $3500",
      "input_source": "static",
      "output": "",
      "tool_calls": [
        {
          "name": "create_expense",
          "args": {
            "amount": 3500
          }
        }
      ],
      "awaiting_input": true,
      "assertions": [
        {
          "type": "tool_called",
          "name": "create_expense",
          "passed": true
        }
      ],
      "duration_ms": 3200
    },
    {
      "turn": 3,
      "input": "Yes, confirm",
      "input_source": "static",
      "output": "Expense submitted. Reference: EXP-2025-001",
      "awaiting_input": false,
      "assertions": [
        {
          "type": "contains",
          "value": "submitted",
          "passed": true
        }
      ],
      "duration_ms": 1800
    }
  ],
  "final_assertions": [
    {
      "type": "json_path",
      "path": "$.expense.status",
      "value": "submitted",
      "passed": true
    }
  ],
  "total_turns": 3,
  "duration_ms": 7100
}

User Simulator Agent

Interface

The simulator agent receives conversation context and generates the next user input:

// Input to simulator
interface SimulatorInput {
  persona: string; // User persona description
  goal: string; // What user wants to achieve
  conversation: Message[]; // Conversation history
  last_response: string; // Agent's last response
  turn_number: number; // Current turn (1-based)
  max_turns: number; // Maximum allowed turns
}

// Output from simulator
interface SimulatorOutput {
  input: string; // Generated user input
  goal_achieved: boolean; // Whether goal is complete
  reasoning?: string; // Why this input was chosen
}

Example Simulator Prompt

You are simulating a user with the following characteristics:

Persona: {{persona}}
Goal: {{goal}}

Current conversation:
{{conversation}}

The agent just responded:
"{{last_response}}"

Generate the next user message to continue toward the goal.
If the goal has been achieved, set goal_achieved to true.

Respond in JSON format:
{
  "input": "your response as the user",
  "goal_achieved": true/false,
  "reasoning": "brief explanation"
}

Backward Compatibility

Existing single-turn tests continue to work unchanged:

// This still works (single-turn, legacy format)
{"id": "T001", "input": "Hello", "assertions": [...]}

// Static mode with one turn (equivalent)
{"id": "T001", "mode": "static", "turns": [{"input": "Hello", "assertions": [...]}]}

Error Handling

Static Mode Errors

Error Type	Behavior	Output
Agent timeout	Mark turn as FAILED	error: "timeout after 30s"
Agent error	Mark turn as FAILED	error: "agent error: ..."
Assertion failed	Mark turn as FAILED	assertion_errors: [...]
All turns passed	Test PASSED	status: "passed"
Any turn failed	Test FAILED	status: "failed"

Dynamic Mode Errors

Error Type	Behavior	Output
All checkpoints reached	Test PASSED	status: "passed"
Checkpoints missing	Test FAILED	error: "missing checkpoints: ..."
Max turns exceeded	Test FAILED	error: "max turns (20) exceeded"
Timeout exceeded	Test FAILED	error: "timeout after 5m"
Simulator error	Test FAILED	error: "simulator error: ..."
Checkpoint assertion failed	Test FAILED	error: "checkpoint X failed"

Context and State

Conversation Context

Each multi-turn test maintains a conversation context:

type ConversationContext struct {
    SessionID    string            // Unique session for this test
    Messages     []Message         // Full conversation history
    ToolResults  map[string]any    // Results from tool calls
    Variables    map[string]any    // Custom variables set during test
    TurnCount    int               // Current turn number
}

Context Passing to Simulator

The simulator receives context via standard Yao Agent Context metadata:

// Framework prepares context for simulator
ctx.Metadata = map[string]any{
    "test_mode":     "simulator",
    "test_id":       "T001",
    // From simulator config
    "persona":       "New employee",
    "goal":          "Submit expense report",
    // Runtime context
    "session_id":    "test-session-001",
    "turn_count":    3,
    "tool_results":  map[string]any{...},
}

// Messages include conversation history
messages := []Message{
    {Role: "user", Content: "I want to submit an expense"},
    {Role: "assistant", Content: "What type of expense?"},
    {Role: "user", Content: "Travel expense"},
    {Role: "assistant", Content: "Please confirm the details..."},
}

Attachments in Multi-Turn

Multi-turn tests support attachments at the turn level:

{
  "id": "T001",
  "name": "Receipt Upload Flow",
  "turns": [
    {
      "input": "I want to submit an expense with receipt",
      "attachments": [
        {
          "type": "image",
          "source": "file://./tests/fixtures/receipt.jpg"
        }
      ]
    },
    {
      "input": "The amount is $150"
    }
  ]
}

Open Questions

1. Session Management: How to handle session state across turns? Use existing session or create new per-test?

2. Timeout Strategy: Per-turn timeout vs. total test timeout?

3. Parallel Execution: Can multi-turn tests run in parallel, or must they be sequential?

4. Retry Logic: If a turn fails, retry just that turn or restart entire conversation?

5. Snapshot Testing: Should we support "golden file" comparison for conversation flows?