diff --git a/agent/test/DESIGN_V2.md b/agent/test/DESIGN_V2.md index b31b1c6d..3b3f498c 100644 --- a/agent/test/DESIGN_V2.md +++ b/agent/test/DESIGN_V2.md @@ -4,9 +4,9 @@ This document describes the design for Agent Test Framework V2, which extends the existing testing capabilities with: -- **Multi-turn conversations** - Test agents across multiple interaction rounds -- **Agent-driven testing** - Use agents to generate test cases and simulate user responses -- **Interactive testing** - Human-in-the-loop testing mode +- **Message history support** - Test agents with conversation context via `messages[]` +- **Agent-driven testing** - Use agents to generate test cases and validate responses +- **Dynamic testing** - Simulator-driven testing with checkpoint validation ## Quick Reference: Format Rules @@ -14,26 +14,17 @@ This document describes the design for Agent Test Framework V2, which extends th | --------------------- | ------------------------ | ------------------------------------------------------- | | `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | | JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` | -| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-sim"` | -| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-sim` | +| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` | +| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` | | `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.validator", {...})` | -## Problem Statement - -Current single-turn testing cannot adequately test: - -1. **Conversational flows** - Agents that guide users through multi-step processes -2. **Confirmation dialogs** - Agents that ask for user confirmation before actions -3. **Clarification requests** - Agents that ask follow-up questions when input is ambiguous -4. **Stateful interactions** - Agents that maintain context across multiple turns - ## Design Goals -1. **Unified format** - Single test case format that supports all modes -2. **Flexible execution** - Static, dynamic (simulator), and interactive modes -3. **Graceful degradation** - Skip tests when required input is unavailable -4. **CI/CD compatible** - Non-interactive mode for automated pipelines -5. **Agent-driven** - Both input generation and user simulation can be agent-powered +1. **Simple** - Single-turn with optional message history, no complex multi-turn state +2. **Stateless** - Each test is independent, no session management needed +3. **Parallel** - Tests can run in parallel since they don't share state +4. **Flexible** - Support both static (messages) and dynamic (simulator) testing +5. **Agent-driven** - Input generation, simulation, and validation can all be agent-powered ## Architecture Overview @@ -43,503 +34,70 @@ Current single-turn testing cannot adequately test: ├─────────────────────────────────────────────────────────────────────────┤ │ │ │ INPUT SOURCES (-i flag) │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ JSONL File │ │ Message │ │ Generator │ │ Interactive │ │ -│ │ ./test.jsonl│ │ "Hello..." │ │ agents:xxx │ │ (stdin) │ │ -│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ -│ │ │ │ │ │ -│ └────────────────┴────────────────┴────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Test Case Parser │ │ -│ │ - Single-turn: {input, assertions} │ │ -│ │ - Multi-turn: {turns: [{input, assertions}, ...]} │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Multi-Turn Executor │ │ -│ │ │ │ -│ │ ┌─────────────────────────────────────────────────────────┐ │ │ -│ │ │ MODE SELECTION (based on test case fields) │ │ │ -│ │ │ │ │ │ -│ │ │ Has `turns`? ─────────────────▶ STATIC MODE │ │ │ -│ │ │ Has `simulator` + `checkpoints`? ──▶ DYNAMIC MODE │ │ │ -│ │ │ Neither? ─────────────────────▶ SINGLE-TURN (legacy) │ │ │ -│ │ └─────────────────────────────────────────────────────────┘ │ │ -│ │ │ │ │ -│ │ ┌───────────────┴───────────────┐ │ │ -│ │ ▼ ▼ │ │ -│ │ ┌─────────────────────┐ ┌─────────────────────────┐ │ │ -│ │ │ STATIC MODE │ │ DYNAMIC MODE │ │ │ -│ │ │ │ │ │ │ │ -│ │ │ FOR each turn: │ │ LOOP until terminated: │ │ │ -│ │ │ 1. Send input │ │ 1. Simulator → input │ │ │ -│ │ │ 2. Get response │ │ 2. Send to Agent │ │ │ -│ │ │ 3. Run assertions │ │ 3. Check checkpoints │ │ │ -│ │ │ 4. Continue/Fail │ │ 4. Check termination │ │ │ -│ │ │ │ │ │ │ │ -│ │ │ All passed → PASS │ │ All checkpoints → PASS │ │ │ -│ │ │ Any failed → FAIL │ │ Timeout/Missing → FAIL │ │ │ -│ │ └─────────────────────┘ └─────────────────────────┘ │ │ -│ │ │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Assertions │ │ -│ │ - Static: Per-turn assertions │ │ -│ │ - Dynamic: Checkpoint assertions (order-independent) │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Reporter │ │ -│ │ - Console output │ │ -│ │ - JSONL output │ │ -│ │ - Custom reporter agent │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ JSONL File │ │ Message │ │ Generator │ │ +│ │ ./test.jsonl│ │ "Hello..." │ │ agents:xxx │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ +│ └────────────────┴────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────────────────────────────┐ │ +│ │ Test Case Parser │ │ +│ │ │ │ +│ │ Standard Mode: {input: "...", messages: [...], assertions} │ │ +│ │ Dynamic Mode: {simulator: {...}, checkpoints: [...]} │ │ +│ └───────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌───────────────┴───────────────┐ │ +│ ▼ ▼ │ +│ ┌───────────────────┐ ┌───────────────────────┐ │ +│ │ STANDARD MODE │ │ DYNAMIC MODE │ │ +│ │ │ │ │ │ +│ │ 1. Build messages │ │ LOOP: │ │ +│ │ 2. Call Agent │ │ 1. Simulator→input │ │ +│ │ 3. Run assertions │ │ 2. Call Agent │ │ +│ │ │ │ 3. Check checkpoints │ │ +│ │ → PASS/FAIL │ │ 4. Until done │ │ +│ └───────────────────┘ └───────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────────────────────────────┐ │ +│ │ Reporter │ │ +│ │ - Console output │ │ +│ │ - JSONL output │ │ +│ └───────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────┘ ``` -## Core Challenge: Detecting "Awaiting Input" State +## Test Modes -The key challenge is determining when an agent is waiting for user input vs. when it has completed its task. +### Standard Mode (Default) -### Detection Strategies +Single call to agent with optional message history. **No multi-turn state management needed.** -#### Strategy 1: Explicit Declaration (Recommended) +| Field | Type | Description | +| ------------ | ------ | ------------------------------------------------ | +| `input` | string | Simple text input (shorthand for single message) | +| `messages` | array | Full message history (overrides `input`) | +| `assertions` | array | Assertions to validate response | +| `options` | object | `context.Options` passed to agent | -Agent explicitly declares its state in the response: +### Dynamic Mode -```json -{ - "content": "What is the expense amount?", - "awaiting_input": true, - "input_hint": "Enter amount, e.g., $3500" -} -``` +Simulator-driven testing with checkpoint validation. -#### Strategy 2: Finish Reason Analysis - -Use the LLM's `finish_reason` to determine state: - -- `stop` - Agent completed normally (may or may not need input) -- `tool_calls` - Agent is executing tools (not awaiting input) -- `length` - Response truncated (not awaiting input) - -#### Strategy 3: Content Heuristics - -Analyze response content for question patterns: - -```go -func looksLikeQuestion(content string) bool { - patterns := []string{ - `\?$`, // Ends with question mark - `(?i)^(what|how|when|where|which|who|please|could you)`, - `(?i)(confirm|verify|proceed|continue)\?`, - } - for _, pattern := range patterns { - if regexp.MustCompile(pattern).MatchString(content) { - return true - } - } - return false -} -``` - -#### Strategy 4: Tool-Based Detection - -Certain tools indicate awaiting input: - -```go -func toolRequiresInput(toolCall ToolCall) bool { - confirmationTools := []string{ - "request_confirmation", - "ask_user", - "get_user_input", - } - return contains(confirmationTools, toolCall.Name) -} -``` - -### Recommended Approach: Hybrid Detection - -Combine multiple strategies with priority: - -```go -func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) { - // Priority 1: Explicit declaration - if result.AwaitingInput { - return true, "agent_declared" - } - - // Priority 2: Tool-based detection - for _, tc := range result.ToolCalls { - if toolRequiresInput(tc) { - return true, "tool_requires_confirmation" - } - } - - // Priority 3: Content heuristics - if looksLikeQuestion(result.Content) { - return true, "content_is_question" - } - - return false, "completed" -} -``` - -## Standard Agent Interface - -All agent-driven features (generator, simulator, validator) use standard Yao Agent interfaces. - -### Using `context.Options` - -The test framework uses `context.Options` to pass parameters, aligned with `Assistant.Stream()`: - -```go -// context.Options - standard Yao Agent options -type Options struct { - Skip *Skip `json:"skip,omitempty"` // Skip history, trace, etc. - Connector string `json:"connector,omitempty"` // LLM connector to use - Search any `json:"search,omitempty"` // Search behavior control - Mode string `json:"mode,omitempty"` // Agent mode (chat, etc.) - Metadata map[string]any `json:"metadata,omitempty"` // Custom metadata -} -``` - -### Test Framework Usage - -```go -// Prepare options for agent invocation -options := &context.Options{ - Skip: &context.Skip{ - History: true, // Don't save test messages to history - Trace: false, // Keep trace for debugging - }, - Metadata: map[string]any{ - "test_mode": "validator", // "generator" | "simulator" | "validator" - "test_id": "T001", - "criteria": "Response should be helpful", - // ... other custom params from test config - }, -} - -// Prepare context -ctx := context.New(parent, authorized, chatID) -ctx.Referer = "agent-test" - -// Call agent with standard Stream API -assistant := agent.Get(agentID) -response, err := assistant.Stream(ctx, messages, options) -``` - -### Test Case Options Field - -Test cases can specify options to pass to the target agent or helper agents: - -```jsonl -{ - "id": "T001", - "input": "Hello", - "options": { - "connector": "openai-gpt4", - "skip": { - "history": true - }, - "metadata": { - "scenario": "edge-case" - } - } -} -``` - -### Generator Mode - -Used when `-i agents:xxx` is specified to generate test cases: - -```go -// Framework calls generator agent -options := &context.Options{ - Skip: &context.Skip{History: true}, - Metadata: map[string]any{ - "test_mode": "generator", - "target_agent": "assistants.expense", - "target_description": "Expense reimbursement assistant", - "target_tools": []string{"create_expense", "get_policy"}, - // From query params: agents:xxx?count=10&focus=edge-cases - "count": 10, - "focus": "edge-cases", - "complexity": "medium", - }, -} -``` - -Expected structured output: - -```json -{ - "cases": [ - {"id": "G001", "input": "...", "assertions": [...]}, - {"id": "G002", "input": "...", "assertions": [...]} - ] -} -``` - -### Simulator Mode - -Used when test case has `simulator` config: - -```go -// Framework calls simulator agent for next user input -options := &context.Options{ - Skip: &context.Skip{History: true}, - Metadata: map[string]any{ - "test_mode": "simulator", - "test_id": "T001", - // From simulator.options.metadata in test case - "persona": "New employee", - "goal": "Submit expense report", - // Runtime context - "turn_number": 3, - "max_turns": 10, - "tool_results": map[string]any{...}, - }, -} - -// Messages include full conversation history -messages := conversationHistory -``` - -Expected structured output: - -```json -{ - "input": "The amount is $500", - "goal_achieved": false, - "reasoning": "Providing requested amount info" -} -``` - -### Validator Mode - -Used when assertion has `type: "agent"`: - -```go -// Framework calls validator agent -options := &context.Options{ - Skip: &context.Skip{History: true}, - Metadata: map[string]any{ - "test_mode": "validator", - "test_id": "T001", - // From assertion.metadata - "criteria": "Response should be helpful", - "expected_intent": "answer question", - "tone": "professional", - }, -} - -// Messages include agent response to validate -messages := []context.Message{ - {Role: "user", Content: "Original user question"}, - {Role: "assistant", Content: "Agent's response to validate"}, -} -``` - -Expected structured output: - -```json -{ - "passed": true, - "score": 0.95, - "reason": "Response is helpful and addresses the question", - "suggestions": [] -} -``` - -## Assertion Types - -### Static Assertions (Existing) - -| Type | Description | Example | -| ------------- | ---------------------- | ---------------------------------------------------------- | -| `contains` | Response contains text | `{"type": "contains", "value": "success"}` | -| `equals` | Exact match | `{"type": "equals", "value": "OK"}` | -| `regex` | Regex pattern match | `{"type": "regex", "pattern": "order-\\d+"}` | -| `json_path` | JSONPath value check | `{"type": "json_path", "path": "$.status", "value": "ok"}` | -| `tool_called` | Tool was invoked | `{"type": "tool_called", "name": "create_expense"}` | -| `type` | Value type check | `{"type": "type", "path": "$.count", "value": "number"}` | - -### Agent-Driven Assertions (New) - -For fuzzy, semantic, or context-aware validation. Uses `options` aligned with `context.Options`: - -```jsonl -{ - "type": "agent", - "use": "agents:workers.test.validator", - "options": { - "connector": "openai-gpt4", - "metadata": { - "criteria": "Response should be helpful and answer the user's question", - "expected_intent": "provide expense submission guidance", - "tone": "professional and friendly" - } - } -} -``` - -### Script Assertions (in JSONL) - -For custom validation logic in JSONL test cases: - -```jsonl -{ - "type": "script", - "use": "scripts:tests.validate-expense", - "options": { - "metadata": { - "min_amount": 100, - "max_amount": 10000 - } - } -} -``` - -### Combined Assertions - -Mix static and agent-driven assertions: - -```jsonl -{ - "assertions": [ - { - "type": "tool_called", - "name": "create_expense" - }, - { - "type": "agent", - "use": "agents:workers.test.validator", - "options": { - "metadata": { - "criteria": "Confirmation message should include expense amount and be polite" - } - } - } - ] -} -``` - -## Script Testing with Agent Assertions - -Script tests can also use Agent-driven assertions via the `t.assert.Agent()` method. - -### API - -```typescript -// t.assert.Agent(response, agentID, options?) -> ValidatorResult -// agentID: Direct agent ID without prefix, e.g., "workers.test.validator" -interface ValidatorResult { - passed: boolean; - score?: number; - reason: string; - suggestions?: string[]; -} -``` - -### Usage in Script Tests - -```typescript -// tests/expense_test.ts -export function TestExpenseResponse(t: TestingT, ctx: Context) { - // Call the agent being tested - const response = Process("agents.expense.Stream", ctx, [ - { role: "user", content: "How do I submit an expense?" }, - ]); - - // Static assertions - t.assert.NotNil(response); - t.assert.Contains(response.content, "expense"); - - // Agent-driven assertion - automatically fails test if validation fails - t.assert.Agent(response.content, "workers.test.validator", { - metadata: { - criteria: - "Response should explain the expense submission process clearly", - expected_topics: ["receipt", "approval", "deadline"], - tone: "helpful", - }, - }); -} -``` - -### With Conversation Context - -```typescript -export function TestMultiTurnExpense(t: TestingT, ctx: Context) { - const messages = [ - { role: "user", content: "I need to submit a travel expense" }, - { role: "assistant", content: "I'd be happy to help..." }, - { role: "user", content: "It's for a flight to Beijing, $2000" }, - ]; - - const response = Process("agents.expense.Stream", ctx, messages); - - // Agent assertion with conversation context - // Automatically fails test and logs suggestions if validation fails - t.assert.Agent(response.content, "workers.test.validator", { - metadata: { - criteria: - "Response should confirm the expense details and ask for receipt", - conversation: messages, - }, - }); -} -``` - -### Implementation - -The `t.assert.Agent()` method internally: - -1. Prepares `context.Options` with `test_mode: "validator"` -2. Calls the validator agent via `Assistant.Stream()` -3. Parses structured output (JSON) -4. Returns `ValidatorResult` - -```go -// In script_assert.go -func assertAgentMethod(iso *v8go.Isolate, t *TestingT, agentCtx *context.Context) *v8go.FunctionTemplate { - return v8go.NewFunctionTemplate(iso, func(info *v8go.FunctionCallbackInfo) *v8go.Value { - // Parse arguments: response, agentID, options - response := args[0].String() - agentID := args[1].String() // Direct agent ID, e.g., "workers.test.validator" - options := parseOptions(args[2]) - - // Prepare validator options - validatorOpts := &context.Options{ - Skip: &context.Skip{History: true}, - Metadata: map[string]any{ - "test_mode": "validator", - ...options.Metadata, - }, - } - - // Call validator agent - assistant, _ := agent.Get(agentID) - result, _ := assistant.Stream(agentCtx, messages, validatorOpts) - - // Parse and return result - return toJsValue(parseValidatorResult(result)) - }) -} -``` +| Field | Type | Description | +| ------------- | ------ | -------------------------------- | +| `simulator` | object | Simulator agent configuration | +| `checkpoints` | array | Functional checkpoints to verify | +| `max_turns` | int | Maximum turns before timeout | +| `timeout` | string | Maximum time (e.g., "5m") | ## Test Case Format -### Single-Turn (Existing) +### Simple Input (Existing) ```jsonl { @@ -554,69 +112,132 @@ func assertAgentMethod(iso *v8go.Isolate, t *TestingT, agentCtx *context.Context } ``` -### Multi-Turn: Static Mode +### With Message History (New) -For **deterministic flows** where you know the exact conversation sequence: +Test agent with conversation context - simulates multi-turn without complex state: ```jsonl { - "id": "T001", - "name": "Expense Reimbursement - Happy Path", - "mode": "static", - "options": { - "connector": "openai-gpt4", - "skip": { - "history": true + "id": "T002", + "name": "Expense submission - final confirmation", + "messages": [ + { + "role": "user", + "content": "I want to submit an expense" + }, + { + "role": "assistant", + "content": "What type of expense would you like to submit?" + }, + { + "role": "user", + "content": "Business travel to Beijing, $3500" + }, + { + "role": "assistant", + "content": "I'll create an expense for business travel, $3500. Please confirm." + }, + { + "role": "user", + "content": "Yes, confirm" } - }, - "turns": [ + ], + "assertions": [ { - "input": "I want to submit an expense report", - "assertions": [ - { - "type": "contains", - "value": "type of expense" - } - ] + "type": "contains", + "value": "submitted" }, { - "input": "Business travel to Beijing, $3500", - "assertions": [ - { - "type": "tool_called", - "name": "create_expense" - } - ] - }, - { - "input": "Yes, confirm", - "assertions": [ - { - "type": "contains", - "value": "submitted" - } - ] + "type": "tool_called", + "name": "create_expense" } ] } ``` -**Characteristics:** +**Key insight**: Instead of executing 3 turns sequentially, we pass the full conversation history. The agent sees the context and responds to the last message. This is: -- Fixed number of turns -- Each turn has specific input and assertions -- Test fails if any turn assertion fails -- Best for regression testing known flows +- **Simpler** - No turn-by-turn execution, no session state +- **Faster** - Single API call instead of multiple +- **Parallelizable** - Each test is independent +- **Debuggable** - Clear input/output for each test -### Multi-Turn: Dynamic Mode (Checkpoints) +### Testing Different Points in a Conversation -For **coverage testing** where you care about functionality, not exact sequence: +To test agent behavior at different conversation stages, create separate test cases: + +```jsonl +// Test 1: First turn - agent should ask for expense type +{ + "id": "expense-turn1", + "messages": [ + {"role": "user", "content": "I want to submit an expense"} + ], + "assertions": [{"type": "contains", "value": "type"}] +} + +// Test 2: Second turn - agent should create expense +{ + "id": "expense-turn2", + "messages": [ + {"role": "user", "content": "I want to submit an expense"}, + {"role": "assistant", "content": "What type of expense would you like to submit?"}, + {"role": "user", "content": "Business travel, $3500"} + ], + "assertions": [{"type": "tool_called", "name": "create_expense"}] +} + +// Test 3: Final turn - agent should confirm submission +{ + "id": "expense-turn3", + "messages": [ + {"role": "user", "content": "I want to submit an expense"}, + {"role": "assistant", "content": "What type of expense?"}, + {"role": "user", "content": "Business travel, $3500"}, + {"role": "assistant", "content": "Confirm $3500 expense?"}, + {"role": "user", "content": "Yes"} + ], + "assertions": [{"type": "contains", "value": "submitted"}] +} +``` + +### With Attachments ```jsonl { - "id": "T002", + "id": "T003", + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "What's in this receipt?" + }, + { + "type": "image", + "source": "file://./fixtures/receipt.jpg" + } + ] + } + ], + "assertions": [ + { + "type": "contains", + "value": "amount" + } + ] +} +``` + +### Dynamic Mode (Simulator + Checkpoints) + +For coverage testing where conversation flow is unpredictable: + +```jsonl +{ + "id": "T004", "name": "Expense Submission Coverage", - "mode": "dynamic", "simulator": { "use": "workers.test.user-simulator", "options": { @@ -626,51 +247,6 @@ For **coverage testing** where you care about functionality, not exact sequence: } } }, - "checkpoints": [ - { - "id": "ask_type", - "description": "Agent asks for expense type", - "assertion": { - "type": "contains", - "value": "type" - } - }, - { - "id": "call_create", - "description": "Agent calls create_expense tool", - "assertion": { - "type": "tool_called", - "name": "create_expense" - } - }, - { - "id": "confirm_submit", - "description": "Agent confirms submission", - "assertion": { - "type": "contains", - "value": "submitted" - } - } - ], - "max_turns": 10, - "timeout": "2m" -} -``` - -**Characteristics:** - -- Simulator drives the conversation -- Checkpoints are verified across all turns (order-independent by default) -- Test passes when ALL checkpoints are reached -- Test fails if max_turns/timeout reached before all checkpoints -- Best for functional coverage testing - -### Checkpoints with Order Constraints - -When checkpoints must occur in a specific order: - -```jsonl -{ "checkpoints": [ { "id": "ask_type", @@ -692,7 +268,7 @@ When checkpoints must occur in a specific order: } }, { - "id": "confirm_submit", + "id": "confirm", "description": "Agent confirms submission", "after": [ "call_create" @@ -702,401 +278,263 @@ When checkpoints must occur in a specific order: "value": "submitted" } } - ] -} -``` - -### Checkpoints with Agent Validation - -Use Agent-driven assertions for semantic validation: - -```jsonl -{ - "checkpoints": [ - { - "id": "helpful_guidance", - "description": "Agent provides helpful expense guidance", - "assertion": { - "type": "agent", - "use": "agents:workers.test.validator", - "options": { - "metadata": { - "criteria": "Response explains expense process clearly and professionally" - } - } - } - }, - { - "id": "tool_called", - "description": "Agent creates expense record", - "assertion": { - "type": "tool_called", - "name": "create_expense" - } - } - ] -} -``` - -### Dynamic Mode Termination - -| Condition | Result | Description | -| ------------------------------------ | ---------- | -------------------------- | -| All checkpoints reached | ✅ PASSED | All functionality verified | -| Agent completes, checkpoints missing | ❌ FAILED | Missing coverage | -| max_turns exceeded | ❌ FAILED | Timeout - flow too long | -| timeout exceeded | ❌ FAILED | Time limit reached | -| Checkpoint assertion fails | ❌ FAILED | Functionality broken | -| Simulator error | ⚠️ SKIPPED | Cannot continue | - -### Field Descriptions - -| Field | Type | Required | Description | -| --------------------------- | ------ | ------------- | ------------------------------------------ | -| `id` | string | Yes | Unique test identifier | -| `name` | string | No | Human-readable test name | -| `mode` | string | No | `"static"` (default) or `"dynamic"` | -| `options` | object | No | `context.Options` passed to target agent | -| **Static Mode Fields** | -| `turns` | array | Yes (static) | Static turn definitions | -| `turns[].input` | string | Yes | User input for this turn | -| `turns[].assertions` | array | No | Assertions for this turn's response | -| `turns[].options` | object | No | Per-turn options override | -| **Dynamic Mode Fields** | -| `simulator` | object | Yes (dynamic) | User simulator configuration | -| `simulator.use` | string | Yes | Simulator agent ID (no prefix) | -| `simulator.options` | object | No | `context.Options` passed to simulator | -| `checkpoints` | array | Yes (dynamic) | Functionality checkpoints to verify | -| `checkpoints[].id` | string | Yes | Unique checkpoint identifier | -| `checkpoints[].description` | string | No | Human-readable description | -| `checkpoints[].assertion` | object | Yes | Assertion to verify | -| `checkpoints[].after` | array | No | Checkpoint IDs that must occur first | -| `max_turns` | int | No | Maximum turns before timeout (default: 20) | -| `timeout` | string | No | Maximum time (default: "5m") | -| **Shared Fields** | -| `interactive` | object | No | Interactive mode configuration | -| `interactive.enabled` | bool | No | Enable human input (default: false) | -| `interactive.timeout` | string | No | Timeout for human input (default: "5m") | - -## Execution Modes - -### Static Mode - -Uses predefined `turns` array. Best for **regression testing** known flows. - -``` -┌─────────────────────────────────────────────────────────┐ -│ Static Mode Flow │ -├─────────────────────────────────────────────────────────┤ -│ │ -│ FOR each turn in turns[]: │ -│ 1. Send turn.input to Agent │ -│ 2. Get Agent response │ -│ 3. Run turn.assertions │ -│ ├─ PASS → Continue to next turn │ -│ └─ FAIL → Test FAILED, stop │ -│ │ -│ All turns completed → Test PASSED │ -│ │ -└─────────────────────────────────────────────────────────┘ -``` - -### Dynamic Mode (Checkpoints) - -Uses simulator + checkpoints. Best for **coverage testing** functionality. - -``` -┌─────────────────────────────────────────────────────────┐ -│ Dynamic Mode Flow │ -├─────────────────────────────────────────────────────────┤ -│ │ -│ Initialize: pending_checkpoints = all checkpoints │ -│ │ -│ LOOP (until terminated): │ -│ 1. Simulator generates user input │ -│ 2. Send input to Agent │ -│ 3. Get Agent response │ -│ 4. Check response against pending_checkpoints │ -│ └─ If matched → Move to reached_checkpoints │ -│ 5. Check termination conditions: │ -│ ├─ All checkpoints reached → PASSED │ -│ ├─ Agent completed, missing checkpoints → FAILED │ -│ ├─ max_turns exceeded → FAILED │ -│ └─ timeout exceeded → FAILED │ -│ │ -└─────────────────────────────────────────────────────────┘ -``` - -### Interactive Mode - -For debugging, human can provide input when agent awaits: - -```bash -# Enable with --interactive flag -yao agent test -i ./tests.jsonl --interactive -``` - -In interactive mode: - -- Static mode: Human can override any turn input -- Dynamic mode: Human can replace simulator for specific turns - -### Mode Selection - -| Has `turns`? | Has `simulator` + `checkpoints`? | Mode | -| ------------ | -------------------------------- | ----------------------- | -| Yes | No | Static | -| No | Yes | Dynamic | -| Yes | Yes | ❌ Invalid (choose one) | -| No | No | Single-turn (legacy) | - -### Example: Static vs Dynamic - -**Same feature, different testing approaches:** - -```jsonl -// Static Mode - Exact sequence testing -{ - "id": "expense-static", - "mode": "static", - "turns": [ - {"input": "Submit expense", "assertions": [{"type": "contains", "value": "type"}]}, - {"input": "Travel, $500", "assertions": [{"type": "tool_called", "name": "create_expense"}]}, - {"input": "Confirm", "assertions": [{"type": "contains", "value": "submitted"}]} - ] -} - -// Dynamic Mode - Coverage testing -{ - "id": "expense-dynamic", - "mode": "dynamic", - "simulator": {"use": "workers.test.user-sim", "options": {"metadata": {"goal": "Submit $500 expense"}}}, - "checkpoints": [ - {"id": "ask", "assertion": {"type": "contains", "value": "type"}}, - {"id": "create", "assertion": {"type": "tool_called", "name": "create_expense"}}, - {"id": "done", "assertion": {"type": "contains", "value": "submitted"}} ], - "max_turns": 10 + "max_turns": 10, + "timeout": "2m" } ``` +## Field Descriptions + +### Standard Mode Fields + +| Field | Type | Required | Description | +| ------------ | ------ | -------- | --------------------------------- | +| `id` | string | Yes | Unique test identifier | +| `name` | string | No | Human-readable test name | +| `input` | string | No\* | Simple text input | +| `messages` | array | No\* | Full message history | +| `assertions` | array | No | Assertions to validate response | +| `options` | object | No | `context.Options` passed to agent | + +\*Either `input` or `messages` required + +### Dynamic Mode Fields + +| Field | Type | Required | Description | +| --------------------------- | ------ | -------- | ------------------------------------------ | +| `id` | string | Yes | Unique test identifier | +| `name` | string | No | Human-readable test name | +| `simulator` | object | Yes | User simulator configuration | +| `simulator.use` | string | Yes | Simulator agent ID (no prefix) | +| `simulator.options` | object | No | `context.Options` passed to simulator | +| `checkpoints` | array | Yes | Functionality checkpoints to verify | +| `checkpoints[].id` | string | Yes | Unique checkpoint identifier | +| `checkpoints[].description` | string | No | Human-readable description | +| `checkpoints[].assertion` | object | Yes | Assertion to verify | +| `checkpoints[].after` | array | No | Checkpoint IDs that must occur first | +| `max_turns` | int | No | Maximum turns before timeout (default: 20) | +| `timeout` | string | No | Maximum time (default: "5m") | +| `options` | object | No | `context.Options` passed to target agent | + ## Execution Flow -### Static Mode Flow +### Standard Mode ``` ┌─────────────────────────────────────────────────────────────────┐ -│ Static Mode Execution │ +│ Standard Mode Execution │ ├─────────────────────────────────────────────────────────────────┤ │ │ -│ INITIALIZE: │ -│ - Load turns[] from test case │ -│ - Set current_turn = 0 │ +│ 1. Parse test case │ +│ ├─ Has `messages`? → Use as-is │ +│ └─ Has `input`? → Convert to [{role: "user", content: input}]│ │ ↓ │ -│ FOR each turn in turns[]: │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ 1. Get input from turns[current_turn].input │ │ -│ │ ↓ │ │ -│ │ 2. Send input to Agent │ │ -│ │ ↓ │ │ -│ │ 3. Get Agent response │ │ -│ │ ↓ │ │ -│ │ 4. Run turns[current_turn].assertions │ │ -│ │ │ │ │ -│ │ ├─ PASS → Continue to next turn │ │ -│ │ └─ FAIL → Test FAILED, stop │ │ -│ └─────────────────────────────────────────────────────────┘ │ +│ 2. Call Agent.Stream(ctx, messages, options) │ │ ↓ │ -│ All turns completed → Test PASSED │ +│ 3. Run assertions against response │ +│ ├─ All PASS → Test PASSED ✅ │ +│ └─ Any FAIL → Test FAILED ❌ │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` -### Dynamic Mode Flow +### Dynamic Mode ``` ┌─────────────────────────────────────────────────────────────────┐ │ Dynamic Mode Execution │ ├─────────────────────────────────────────────────────────────────┤ │ │ -│ INITIALIZE: │ +│ Initialize: │ │ - pending_checkpoints = all checkpoints │ -│ - reached_checkpoints = [] │ +│ - messages = [] │ │ - turn_count = 0 │ -│ - start_time = now() │ │ ↓ │ │ LOOP: │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ 1. Call Simulator Agent → Get user input │ │ -│ │ (pass: persona, goal, conversation history) │ │ -│ │ ↓ │ │ -│ │ 2. Send input to Target Agent │ │ -│ │ ↓ │ │ -│ │ 3. Get Agent response │ │ -│ │ ↓ │ │ -│ │ 4. Check response against pending_checkpoints │ │ -│ │ FOR each pending checkpoint: │ │ -│ │ - Run checkpoint.assertion │ │ -│ │ - If PASS and `after` satisfied → move to reached │ │ -│ │ ↓ │ │ -│ │ 5. Check termination conditions: │ │ -│ │ ├─ pending_checkpoints empty? │ │ -│ │ │ → Test PASSED ✅ │ │ -│ │ │ │ │ -│ │ ├─ Agent completed (not awaiting)? │ │ -│ │ │ → Test FAILED ❌ (missing checkpoints) │ │ -│ │ │ │ │ -│ │ ├─ turn_count >= max_turns? │ │ -│ │ │ → Test FAILED ❌ (turn limit) │ │ -│ │ │ │ │ -│ │ ├─ now() - start_time > timeout? │ │ -│ │ │ → Test FAILED ❌ (timeout) │ │ -│ │ │ │ │ -│ │ └─ Otherwise → Continue loop │ │ -│ └─────────────────────────────────────────────────────────┘ │ +│ 1. Call Simulator → get user input │ +│ 2. Append user message to messages │ +│ 3. Call Agent.Stream(ctx, messages, options) │ +│ 4. Append assistant response to messages │ +│ 5. Check response against pending_checkpoints │ +│ └─ If matched (and `after` satisfied) → move to reached │ +│ 6. Check termination: │ +│ ├─ All checkpoints reached → PASSED ✅ │ +│ ├─ Simulator signals goal_achieved → FAILED ❌ │ +│ ├─ turn_count >= max_turns → FAILED ❌ │ +│ └─ timeout exceeded → FAILED ❌ │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` +## Assertion Types + +### Static Assertions + +| Type | Description | Example | +| ------------- | ---------------------- | ---------------------------------------------------------- | +| `contains` | Response contains text | `{"type": "contains", "value": "success"}` | +| `equals` | Exact match | `{"type": "equals", "value": "OK"}` | +| `regex` | Regex pattern match | `{"type": "regex", "pattern": "order-\\d+"}` | +| `json_path` | JSONPath value check | `{"type": "json_path", "path": "$.status", "value": "ok"}` | +| `tool_called` | Tool was invoked | `{"type": "tool_called", "name": "create_expense"}` | +| `type` | Value type check | `{"type": "type", "path": "$.count", "value": "number"}` | + +### Agent-Driven Assertions + +For semantic or fuzzy validation: + +```jsonl +{ + "type": "agent", + "use": "agents:workers.test.validator", + "options": { + "metadata": { + "criteria": "Response should be helpful and answer the user's question", + "tone": "professional and friendly" + } + } +} +``` + +### Script Assertions + +For custom validation logic: + +```jsonl +{ + "type": "script", + "use": "scripts:tests.validate-expense", + "options": { + "metadata": { + "min_amount": 100, + "max_amount": 10000 + } + } +} +``` + +## Script Testing with Agent Assertions + +Script tests can use Agent-driven assertions via `t.assert.Agent()`: + +```typescript +export function TestExpenseResponse(t: TestingT, ctx: Context) { + const messages = [ + { role: "user", content: "I want to submit an expense" }, + { role: "assistant", content: "What type of expense?" }, + { role: "user", content: "Travel, $500" }, + ]; + + const response = Process("agents.expense.Stream", ctx, messages); + + // Static assertion + t.assert.Contains(response.content, "confirm"); + + // Agent-driven assertion + t.assert.Agent(response.content, "workers.test.validator", { + metadata: { + criteria: "Response should ask for confirmation before creating expense", + conversation: messages, + }, + }); +} +``` + +## Standard Agent Interface + +All agent-driven features use `context.Options`: + +```go +type Options struct { + Skip *Skip `json:"skip,omitempty"` + Connector string `json:"connector,omitempty"` + Search any `json:"search,omitempty"` + Mode string `json:"mode,omitempty"` + Metadata map[string]any `json:"metadata,omitempty"` +} +``` + +### Generator Agent + +Called when `-i agents:xxx` is used: + +```go +options := &context.Options{ + Metadata: map[string]any{ + "test_mode": "generator", + "target_agent": "assistants.expense", + "count": 10, + "focus": "edge-cases", + }, +} +``` + +### Simulator Agent + +Called in dynamic mode to generate user input: + +```go +options := &context.Options{ + Metadata: map[string]any{ + "test_mode": "simulator", + "persona": "New employee", + "goal": "Submit expense", + "turn_number": 3, + }, +} +``` + +### Validator Agent + +Called for agent-driven assertions: + +```go +options := &context.Options{ + Metadata: map[string]any{ + "test_mode": "validator", + "criteria": "Response should be helpful", + }, +} +``` + ## Command Line Interface ### Flags Reference -| Flag | Long | Description | -| ---- | --------------- | ---------------------------------------------------------- | -| `-i` | `--input` | Input source: file path, message, or `type:id` reference | -| `-n` | `--name` | Target agent ID (the agent being tested) | -| `-o` | `--output` | Output file path for results | -| `-c` | `--connector` | Override connector for the target agent | -| `-v` | `--verbose` | Verbose output showing all turns | -| | `--interactive` | Enable human input when agent awaits | -| | `--simulator` | Default simulator agent ID (e.g., `workers.test.user-sim`) | -| | `--timeout` | Timeout per test case (default: 5m) | -| | `--parallel` | Number of parallel test cases | -| | `--fail-fast` | Stop on first failure | -| | `--dry-run` | Generate/parse tests without running | +| Flag | Long | Description | +| ---- | ------------- | -------------------------------------------------------- | +| `-i` | `--input` | Input source: file path, message, or `type:id` reference | +| `-n` | `--name` | Target agent ID (the agent being tested) | +| `-o` | `--output` | Output file path for results | +| `-c` | `--connector` | Override connector for the target agent | +| `-v` | `--verbose` | Verbose output | +| | `--simulator` | Default simulator agent ID | +| | `--timeout` | Timeout per test case (default: 5m) | +| | `--parallel` | Number of parallel test cases | +| | `--fail-fast` | Stop on first failure | +| | `--dry-run` | Generate/parse tests without running | -### Input Sources (`-i` flag) - -The `-i` flag supports multiple input sources with unified `type:id` format: +### Examples ```bash -# 1. File path (default, no prefix needed) -yao agent test -i ./tests/multi-turn.jsonl - -# 2. Direct message (no prefix, auto-detected as non-file) +# Simple test yao agent test -i "Hello, how are you?" -n assistants.chat -# 3. Agent-generated test cases -yao agent test -i agents:workers.test.generator -n assistants.expense - -# 4. Script-generated test cases -yao agent test -i scripts:tests.generate -n assistants.expense - -# 5. With parameters (query string style) -yao agent test -i "agents:workers.test.generator?count=10&focus=edge-cases" -n assistants.expense -``` - -### Input Type Prefixes - -| Prefix | Description | Example | -| ---------- | --------------------------- | -------------------------- | -| (none) | File path or direct message | `./tests.jsonl`, `"Hello"` | -| `agents:` | Agent generates test cases | `agents:workers.test.gen` | -| `scripts:` | Script generates test cases | `scripts:tests.generate` | - -### Input Format - -``` -[prefix:][?param1=value1¶m2=value2] -``` - -- `prefix` - Input type: `agents:` or `scripts:` (optional, default is file/message) -- `id` - Agent ID or script ID -- `?params` - Query parameters passed to generator - -#### Generator Agent Interface - -```typescript -// Input to generator agent -interface GeneratorInput { - target_agent: string; // Agent being tested (from -n flag) - target_description?: string; // Agent's description/purpose - target_tools?: Tool[]; // Agent's available tools - count?: number; // Number of test cases to generate - focus?: string; // Focus area: "happy-path", "edge-cases", "errors" - complexity?: string; // "simple", "medium", "complex" -} - -// Output from generator agent -interface GeneratorOutput { - cases: TestCase[]; // Generated test cases -} -``` - -#### Example Generator Prompt - -``` -You are a test case generator for AI agents. - -Target Agent: {{target_agent}} -Description: {{target_description}} -Available Tools: {{target_tools}} - -Generate {{count}} test cases with focus on: {{focus}} - -For each test case, provide: -- id: Unique identifier -- name: Descriptive name -- input: User message or turns array for multi-turn -- assertions: Expected behaviors to verify - -Output as JSON array of test cases. -``` - -### Complete Examples - -```bash -# Basic: Run tests from file +# From JSONL file yao agent test -i ./tests/expense.jsonl -# With target agent specified (required for message/agent input) -yao agent test -i "Help me file an expense" -n assistants.expense +# Agent-generated tests +yao agent test -i "agents:workers.test.generator?count=10" -n assistants.expense -# Agent generates tests, then runs them -yao agent test \ - -i "agents:workers.test.generator?count=20&focus=edge-cases" \ - -n assistants.expense +# With simulator for dynamic mode +yao agent test -i ./tests/dynamic.jsonl --simulator workers.test.user-simulator -# Script generates tests -yao agent test \ - -i "scripts:tests.expense.generate?scenario=approval-flow" \ - -n assistants.expense +# Parallel execution +yao agent test -i ./tests/expense.jsonl --parallel 5 -# Fully dynamic: Agent generates tests + Agent simulates user responses -yao agent test \ - -i "agents:workers.test.generator?count=10" \ - -n assistants.expense \ - --simulator workers.test.user-simulator - -# Generate tests only, save to file (dry-run) -yao agent test \ - -i "agents:workers.test.generator?count=50" \ - -n assistants.expense \ - -o ./tests/generated.jsonl \ - --dry-run - -# Interactive mode: human provides input when agent awaits -yao agent test -i ./tests/multi-turn.jsonl --interactive - -# CI/CD mode: skip tests requiring human input -yao agent test -i ./tests/multi-turn.jsonl --skip-interactive - -# Fail instead of skip when input unavailable -yao agent test -i ./tests/multi-turn.jsonl --on-missing-input=fail - -# Verbose output showing all turns -yao agent test -i ./tests/multi-turn.jsonl -v +# Verbose output +yao agent test -i ./tests/expense.jsonl -v ``` ## Output Format @@ -1105,136 +543,96 @@ yao agent test -i ./tests/multi-turn.jsonl -v ``` ═══════════════════════════════════════════════════════════════ - Agent Test (Multi-Turn) + Agent Test ═══════════════════════════════════════════════════════════════ ℹ Agent: assistants.expense -ℹ Input: ./tests/expense-flow.jsonl (5 test cases) +ℹ Input: ./tests/expense.jsonl (3 test cases) ─────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────── -► [T001] Expense Reimbursement Flow (3 turns) - ├─ Turn 1: "I want to submit an expense" → PASSED (2.1s) - │ Agent: "What type of expense would you like to submit?" - │ ✓ contains "type of expense" - │ - ├─ Turn 2: "Business travel, $3500" → PASSED (3.2s) - │ Agent: [tool: create_expense({amount: 3500, type: "travel"})] - │ ✓ tool_called "create_expense" - │ - ├─ Turn 3: "Yes, confirm" → PASSED (1.8s) - │ Agent: "Expense submitted. Reference: EXP-2025-001" - │ ✓ contains "submitted" - │ - └─ Final Assertions: PASSED - ✓ $.expense.status = "submitted" +✓ [expense-turn1] First turn - ask type (1.2s) + Messages: 1, Assertions: 1/1 passed -► [T002] Large Expense Approval - ├─ Turn 1: "Submit $100,000 equipment purchase" → PASSED (2.0s) - │ Agent: "This requires manager approval. Please provide PO number." - │ - ├─ Turn 2: SKIPPED - │ Reason: Agent awaiting input, no next turn defined - │ Agent asked: "Please provide PO number" - │ Hint: Add more turns, use --interactive, or configure simulator - │ - └─ Result: SKIPPED +✓ [expense-turn2] Second turn - create expense (2.1s) + Messages: 3, Assertions: 1/1 passed -► [T003] Dynamic Expense Flow (simulator: workers.test.user-sim) - ├─ Turn 1: [Initial] "Help me file an expense" → PASSED (2.1s) - ├─ Turn 2: [Simulated] "It's for client dinner, $250" → PASSED (2.8s) - ├─ Turn 3: [Simulated] "Yesterday evening" → PASSED (2.2s) - ├─ Turn 4: [Simulated] "Confirm" → PASSED (1.9s) - │ Goal achieved: Expense submitted - │ - └─ Final Assertions: PASSED +✓ [expense-turn3] Final turn - confirm (1.8s) + Messages: 5, Assertions: 1/1 passed ─────────────────────────────────────────────────────────────── Summary ─────────────────────────────────────────────────────────────── Total: 3 tests - Passed: 2 + Passed: 3 Failed: 0 - Skipped: 1 - - Total turns: 10 - Avg turns/test: 3.3 - Total time: 18.1s + Time: 5.1s ``` ### JSONL Output ```jsonl { - "id": "T001", - "name": "Expense Reimbursement Flow", + "id": "expense-turn3", + "name": "Final turn - confirm", "status": "passed", - "turns": [ + "messages_count": 5, + "response": "Expense submitted. Reference: EXP-2025-001", + "assertions": [ { - "turn": 1, - "input": "I want to submit an expense", - "input_source": "static", - "output": "What type of expense would you like to submit?", - "awaiting_input": true, - "assertions": [ - { - "type": "contains", - "value": "type of expense", - "passed": true - } - ], - "duration_ms": 2100 - }, - { - "turn": 2, - "input": "Business travel, $3500", - "input_source": "static", - "output": "", - "tool_calls": [ - { - "name": "create_expense", - "args": { - "amount": 3500 - } - } - ], - "awaiting_input": true, - "assertions": [ - { - "type": "tool_called", - "name": "create_expense", - "passed": true - } - ], - "duration_ms": 3200 - }, - { - "turn": 3, - "input": "Yes, confirm", - "input_source": "static", - "output": "Expense submitted. Reference: EXP-2025-001", - "awaiting_input": false, - "assertions": [ - { - "type": "contains", - "value": "submitted", - "passed": true - } - ], - "duration_ms": 1800 - } - ], - "final_assertions": [ - { - "type": "json_path", - "path": "$.expense.status", + "type": "contains", "value": "submitted", "passed": true } ], + "duration_ms": 1800 +} +``` + +## Dynamic Mode Output + +```jsonl +{ + "id": "expense-dynamic", + "name": "Expense Coverage Test", + "status": "passed", + "turns": [ + { + "turn": 1, + "input": "Help me file an expense", + "output": "What type?" + }, + { + "turn": 2, + "input": "Client dinner, $250", + "output": "Confirm?" + }, + { + "turn": 3, + "input": "Yes", + "output": "Submitted!" + } + ], + "checkpoints": [ + { + "id": "ask_type", + "reached_at_turn": 1, + "passed": true + }, + { + "id": "call_create", + "reached_at_turn": 2, + "passed": true + }, + { + "id": "confirm", + "reached_at_turn": 3, + "passed": true + } + ], "total_turns": 3, - "duration_ms": 7100 + "duration_ms": 6800 } ``` @@ -1242,28 +640,23 @@ yao agent test -i ./tests/multi-turn.jsonl -v ### Interface -The simulator agent receives conversation context and generates the next user input: - ```typescript -// Input to simulator interface SimulatorInput { - persona: string; // User persona description - goal: string; // What user wants to achieve - conversation: Message[]; // Conversation history - last_response: string; // Agent's last response - turn_number: number; // Current turn (1-based) - max_turns: number; // Maximum allowed turns + persona: string; + goal: string; + conversation: Message[]; + turn_number: number; + max_turns: number; } -// Output from simulator interface SimulatorOutput { - input: string; // Generated user input - goal_achieved: boolean; // Whether goal is complete - reasoning?: string; // Why this input was chosen + input: string; + goal_achieved: boolean; + reasoning?: string; } ``` -### Example Simulator Prompt +### Example Prompt ``` You are simulating a user with the following characteristics: @@ -1274,9 +667,6 @@ Goal: {{goal}} Current conversation: {{conversation}} -The agent just responded: -"{{last_response}}" - Generate the next user message to continue toward the goal. If the goal has been achieved, set goal_achieved to true. @@ -1290,27 +680,25 @@ Respond in JSON format: ## Backward Compatibility -Existing single-turn tests continue to work unchanged: +Existing single-turn tests work unchanged: ```jsonl -// This still works (single-turn, legacy format) +// This still works {"id": "T001", "input": "Hello", "assertions": [...]} -// Static mode with one turn (equivalent) -{"id": "T001", "mode": "static", "turns": [{"input": "Hello", "assertions": [...]}]} +// Equivalent to +{"id": "T001", "messages": [{"role": "user", "content": "Hello"}], "assertions": [...]} ``` ## Error Handling -### Static Mode Errors +### Standard Mode Errors -| Error Type | Behavior | Output | -| ---------------- | ------------------- | ---------------------------- | -| Agent timeout | Mark turn as FAILED | `error: "timeout after 30s"` | -| Agent error | Mark turn as FAILED | `error: "agent error: ..."` | -| Assertion failed | Mark turn as FAILED | `assertion_errors: [...]` | -| All turns passed | Test PASSED | `status: "passed"` | -| Any turn failed | Test FAILED | `status: "failed"` | +| Error Type | Behavior | Output | +| ---------------- | ----------- | ---------------------------- | +| Agent timeout | Test FAILED | `error: "timeout after 30s"` | +| Agent error | Test FAILED | `error: "agent error: ..."` | +| Assertion failed | Test FAILED | `assertion_errors: [...]` | ### Dynamic Mode Errors @@ -1323,82 +711,21 @@ Existing single-turn tests continue to work unchanged: | Simulator error | Test FAILED | `error: "simulator error: ..."` | | Checkpoint assertion failed | Test FAILED | `error: "checkpoint X failed"` | -## Context and State +## Comparison: Old vs New Design -### Conversation Context - -Each multi-turn test maintains a conversation context: - -```go -type ConversationContext struct { - SessionID string // Unique session for this test - Messages []Message // Full conversation history - ToolResults map[string]any // Results from tool calls - Variables map[string]any // Custom variables set during test - TurnCount int // Current turn number -} -``` - -### Context Passing to Simulator - -The simulator receives context via standard Yao Agent Context metadata: - -```go -// Framework prepares context for simulator -ctx.Metadata = map[string]any{ - "test_mode": "simulator", - "test_id": "T001", - // From simulator config - "persona": "New employee", - "goal": "Submit expense report", - // Runtime context - "session_id": "test-session-001", - "turn_count": 3, - "tool_results": map[string]any{...}, -} - -// Messages include conversation history -messages := []Message{ - {Role: "user", Content: "I want to submit an expense"}, - {Role: "assistant", Content: "What type of expense?"}, - {Role: "user", Content: "Travel expense"}, - {Role: "assistant", Content: "Please confirm the details..."}, -} -``` - -## Attachments in Multi-Turn - -Multi-turn tests support attachments at the turn level: - -```jsonl -{ - "id": "T001", - "name": "Receipt Upload Flow", - "turns": [ - { - "input": "I want to submit an expense with receipt", - "attachments": [ - { - "type": "image", - "source": "file://./tests/fixtures/receipt.jpg" - } - ] - }, - { - "input": "The amount is $150" - } - ] -} -``` +| Aspect | Old (Static Mode) | New (Messages) | +| ------------------ | ------------------------- | --------------------------- | +| Multi-turn testing | Sequential turn execution | Pass message history | +| State management | Session state per test | Stateless | +| Parallelization | Sequential within test | Fully parallel | +| Implementation | Complex turn loop | Single agent call | +| Debugging | Need to trace turns | Clear input/output per test | +| Flexibility | Coupled turns | Independent tests | ## Open Questions -1. **Session Management**: How to handle session state across turns? Use existing session or create new per-test? +1. **Message Generation**: Should we provide a helper to generate message history from a script? -2. **Timeout Strategy**: Per-turn timeout vs. total test timeout? +2. **Snapshot Testing**: Should we support "golden file" comparison for responses? -3. **Parallel Execution**: Can multi-turn tests run in parallel, or must they be sequential? - -4. **Retry Logic**: If a turn fails, retry just that turn or restart entire conversation? - -5. **Snapshot Testing**: Should we support "golden file" comparison for conversation flows? +3. **Retry Logic**: If a test fails, should we support automatic retry? diff --git a/agent/test/TODO_V2.md b/agent/test/TODO_V2.md index c865d9b0..946e3804 100644 --- a/agent/test/TODO_V2.md +++ b/agent/test/TODO_V2.md @@ -2,32 +2,30 @@ ## Format Rules Summary -| Context | Format | Example | -|---------|--------|---------| -| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | -| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` | -| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-sim"` | -| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-sim` | -| `t.assert.Agent()` | No prefix (method is explicit) | `t.assert.Agent(resp, "workers.test.validator", {...})` | +| Context | Format | Example | +| --------------------- | ------------------------ | ------------------------------------------------------- | +| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | +| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` | +| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` | +| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` | +| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.validator", {...})` | -## Phase 1: Static Mode +## Phase 1: Message History Support -- [ ] Add `mode` field to test case parser (`static` | `dynamic`) -- [ ] Extend test case parser for `turns` array +- [ ] Add `messages` field to test case parser +- [ ] Support both `input` (string) and `messages` (array) fields +- [ ] Convert `input` to `messages` format internally +- [ ] Pass messages directly to `Agent.Stream()` - [ ] Add `options` field support (aligned with `context.Options`) -- [ ] Support test-level `options` and per-turn `options` override -- [ ] Implement turn-by-turn execution with options passing -- [ ] Implement conversation context management -- [ ] Add per-turn assertions -- [ ] Support attachments at turn level -- [ ] Update console output for multi-turn display -- [ ] Update JSONL output format for turns +- [ ] Support attachments in message content parts +- [ ] Update console output to show message count +- [ ] Update JSONL output format ## Phase 2: Agent-Driven Input - [ ] Parse `agents:` prefix in `-i` flag - [ ] Parse `scripts:` prefix in `-i` flag -- [ ] Use standard `context.Options` for all agent invocations +- [ ] Use standard `context.Options` for generator invocation - [ ] Pass `test_mode: "generator"` in `options.metadata` - [ ] Pass target agent info (description, tools) in `options.metadata` - [ ] Support query parameters (`?count=10&focus=...`) → merged into `options.metadata` @@ -37,12 +35,13 @@ ## Phase 3: Dynamic Mode (Checkpoints) - [ ] Add `checkpoints` array to test case parser +- [ ] Add `simulator` field to test case parser - [ ] Implement checkpoint matching against agent responses - [ ] Support `after` field for order constraints - [ ] Track pending/reached checkpoints during execution - [ ] Implement termination conditions: - [ ] All checkpoints reached → PASSED - - [ ] Agent completed, missing checkpoints → FAILED + - [ ] Simulator signals goal_achieved but checkpoints missing → FAILED - [ ] max_turns exceeded → FAILED - [ ] timeout exceeded → FAILED - [ ] Implement simulator invocation via `Assistant.Stream()` @@ -52,24 +51,10 @@ - [ ] Pass conversation history as messages - [ ] Create example simulator agent with prompt template -## Phase 4: Interactive Mode - -- [ ] Add `--interactive` flag -- [ ] Implement terminal input prompt with context display -- [ ] Add input timeout handling -- [ ] Support input history/editing -- [ ] Add `--skip-interactive` for CI/CD mode - -## Phase 5: Enhanced Detection - -- [ ] Add `awaiting_input` field to agent response schema -- [ ] Implement tool-based detection (confirmation tools) -- [ ] Add configurable detection rules -- [ ] Support custom detection via script/agent - -## Phase 6: Agent-Driven Assertions +## Phase 4: Agent-Driven Assertions ### In JSONL Test Cases + - [ ] Add `agent` assertion type to assertion parser - [ ] Support `options` field in assertion (aligned with `context.Options`) - [ ] Implement validator agent invocation via `Assistant.Stream()` @@ -79,6 +64,7 @@ - [ ] Add `suggestions` to assertion error output ### In Script Tests + - [ ] Add `t.assert.Agent(response, agentID, options?)` method - [ ] `agentID` is direct ID (e.g., `workers.test.validator`), no prefix needed - [ ] Invoke validator agent with context @@ -86,25 +72,22 @@ - [ ] Support passing conversation history in options ### Shared + - [ ] Create example validator agent with prompt template - [ ] Document `ValidatorResult` interface -## Phase 7: Error Handling & Reporting +## Phase 5: Error Handling & Reporting -- [ ] Implement turn-level error handling -- [ ] Implement test-level error aggregation +- [ ] Implement test-level error handling - [ ] Add detailed error messages with hints -- [ ] Support custom reporter agent +- [ ] Support `--parallel` flag for concurrent test execution +- [ ] Support `--fail-fast` flag to stop on first failure +- [ ] Add verbose mode (`-v`) for detailed output ## Open Questions -1. **Session Management**: How to handle session state across turns? Use existing session or create new per-test? +1. **Message Generation**: Should we provide a helper to generate message history from a script? -2. **Timeout Strategy**: Per-turn timeout vs. total test timeout? - -3. **Parallel Execution**: Can multi-turn tests run in parallel, or must they be sequential? - -4. **Retry Logic**: If a turn fails, retry just that turn or restart entire conversation? - -5. **Snapshot Testing**: Should we support "golden file" comparison for conversation flows? +2. **Snapshot Testing**: Should we support "golden file" comparison for responses? +3. **Retry Logic**: If a test fails, should we support automatic retry?