Enhance Test Framework with Standard Agent Interface and Options Support
- Introduced a comprehensive standard agent interface for agent-driven features, including generator, simulator, and validator modes. - Added support for `context.Options` to pass parameters in test cases, allowing for flexible configuration of agent behavior. - Updated test case format to include options at both the test and per-turn levels, enhancing customization and control over agent interactions. - Expanded documentation to detail the usage of options in various agent modes, improving clarity for developers and users.
This commit is contained in:
Max
parent
2e2ce8bfad
commit
e417944850
2 changed files with 317 additions and 34 deletions
|
|
@ -197,6 +197,247 @@ func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Standard Agent Interface
|
||||||
|
|
||||||
|
All agent-driven features (generator, simulator, validator) use standard Yao Agent interfaces.
|
||||||
|
|
||||||
|
### Using `context.Options`
|
||||||
|
|
||||||
|
The test framework uses `context.Options` to pass parameters, aligned with `Assistant.Stream()`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// context.Options - standard Yao Agent options
|
||||||
|
type Options struct {
|
||||||
|
Skip *Skip `json:"skip,omitempty"` // Skip history, trace, etc.
|
||||||
|
Connector string `json:"connector,omitempty"` // LLM connector to use
|
||||||
|
Search any `json:"search,omitempty"` // Search behavior control
|
||||||
|
Mode string `json:"mode,omitempty"` // Agent mode (chat, etc.)
|
||||||
|
Metadata map[string]any `json:"metadata,omitempty"` // Custom metadata
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Framework Usage
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Prepare options for agent invocation
|
||||||
|
options := &context.Options{
|
||||||
|
Skip: &context.Skip{
|
||||||
|
History: true, // Don't save test messages to history
|
||||||
|
Trace: false, // Keep trace for debugging
|
||||||
|
},
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"test_mode": "validator", // "generator" | "simulator" | "validator"
|
||||||
|
"test_id": "T001",
|
||||||
|
"criteria": "Response should be helpful",
|
||||||
|
// ... other custom params from test config
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prepare context
|
||||||
|
ctx := context.New(parent, authorized, chatID)
|
||||||
|
ctx.Referer = "agent-test"
|
||||||
|
|
||||||
|
// Call agent with standard Stream API
|
||||||
|
assistant := agent.Get(agentID)
|
||||||
|
response, err := assistant.Stream(ctx, messages, options)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Case Options Field
|
||||||
|
|
||||||
|
Test cases can specify options to pass to the target agent or helper agents:
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T001",
|
||||||
|
"input": "Hello",
|
||||||
|
"options": {
|
||||||
|
"connector": "openai-gpt4",
|
||||||
|
"skip": {
|
||||||
|
"history": true
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"scenario": "edge-case"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Generator Mode
|
||||||
|
|
||||||
|
Used when `-i agents:xxx` is specified to generate test cases:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Framework calls generator agent
|
||||||
|
options := &context.Options{
|
||||||
|
Skip: &context.Skip{History: true},
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"test_mode": "generator",
|
||||||
|
"target_agent": "assistants.expense",
|
||||||
|
"target_description": "Expense reimbursement assistant",
|
||||||
|
"target_tools": []string{"create_expense", "get_policy"},
|
||||||
|
// From query params: agents:xxx?count=10&focus=edge-cases
|
||||||
|
"count": 10,
|
||||||
|
"focus": "edge-cases",
|
||||||
|
"complexity": "medium",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected structured output:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"cases": [
|
||||||
|
{"id": "G001", "input": "...", "assertions": [...]},
|
||||||
|
{"id": "G002", "input": "...", "assertions": [...]}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Simulator Mode
|
||||||
|
|
||||||
|
Used when test case has `simulator` config:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Framework calls simulator agent for next user input
|
||||||
|
options := &context.Options{
|
||||||
|
Skip: &context.Skip{History: true},
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"test_mode": "simulator",
|
||||||
|
"test_id": "T001",
|
||||||
|
// From simulator.metadata in test case
|
||||||
|
"persona": "New employee",
|
||||||
|
"goal": "Submit expense report",
|
||||||
|
// Runtime context
|
||||||
|
"turn_number": 3,
|
||||||
|
"max_turns": 10,
|
||||||
|
"tool_results": map[string]any{...},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Messages include full conversation history
|
||||||
|
messages := conversationHistory
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected structured output:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"input": "The amount is $500",
|
||||||
|
"goal_achieved": false,
|
||||||
|
"reasoning": "Providing requested amount info"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validator Mode
|
||||||
|
|
||||||
|
Used when assertion has `type: "agent"`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Framework calls validator agent
|
||||||
|
options := &context.Options{
|
||||||
|
Skip: &context.Skip{History: true},
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"test_mode": "validator",
|
||||||
|
"test_id": "T001",
|
||||||
|
// From assertion.metadata
|
||||||
|
"criteria": "Response should be helpful",
|
||||||
|
"expected_intent": "answer question",
|
||||||
|
"tone": "professional",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Messages include agent response to validate
|
||||||
|
messages := []context.Message{
|
||||||
|
{Role: "user", Content: "Original user question"},
|
||||||
|
{Role: "assistant", Content: "Agent's response to validate"},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected structured output:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"passed": true,
|
||||||
|
"score": 0.95,
|
||||||
|
"reason": "Response is helpful and addresses the question",
|
||||||
|
"suggestions": []
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Assertion Types
|
||||||
|
|
||||||
|
### Static Assertions (Existing)
|
||||||
|
|
||||||
|
| Type | Description | Example |
|
||||||
|
| ------------- | ---------------------- | ---------------------------------------------------------- |
|
||||||
|
| `contains` | Response contains text | `{"type": "contains", "value": "success"}` |
|
||||||
|
| `equals` | Exact match | `{"type": "equals", "value": "OK"}` |
|
||||||
|
| `regex` | Regex pattern match | `{"type": "regex", "pattern": "order-\\d+"}` |
|
||||||
|
| `json_path` | JSONPath value check | `{"type": "json_path", "path": "$.status", "value": "ok"}` |
|
||||||
|
| `tool_called` | Tool was invoked | `{"type": "tool_called", "name": "create_expense"}` |
|
||||||
|
| `type` | Value type check | `{"type": "type", "path": "$.count", "value": "number"}` |
|
||||||
|
|
||||||
|
### Agent-Driven Assertions (New)
|
||||||
|
|
||||||
|
For fuzzy, semantic, or context-aware validation. Uses `options` aligned with `context.Options`:
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"type": "agent",
|
||||||
|
"use": "agents:workers.test.validator",
|
||||||
|
"options": {
|
||||||
|
"connector": "openai-gpt4",
|
||||||
|
"metadata": {
|
||||||
|
"criteria": "Response should be helpful and answer the user's question",
|
||||||
|
"expected_intent": "provide expense submission guidance",
|
||||||
|
"tone": "professional and friendly"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Script Assertions
|
||||||
|
|
||||||
|
For custom validation logic:
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"type": "script",
|
||||||
|
"use": "scripts:tests.validate-expense",
|
||||||
|
"options": {
|
||||||
|
"metadata": {
|
||||||
|
"min_amount": 100,
|
||||||
|
"max_amount": 10000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Combined Assertions
|
||||||
|
|
||||||
|
Mix static and agent-driven assertions:
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"assertions": [
|
||||||
|
{
|
||||||
|
"type": "tool_called",
|
||||||
|
"name": "create_expense"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "agent",
|
||||||
|
"use": "agents:workers.test.validator",
|
||||||
|
"options": {
|
||||||
|
"metadata": {
|
||||||
|
"criteria": "Confirmation message should include expense amount and be polite"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## Test Case Format
|
## Test Case Format
|
||||||
|
|
||||||
### Single-Turn (Existing)
|
### Single-Turn (Existing)
|
||||||
|
|
@ -221,6 +462,15 @@ func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) {
|
||||||
"id": "T001",
|
"id": "T001",
|
||||||
"name": "Expense Reimbursement Flow",
|
"name": "Expense Reimbursement Flow",
|
||||||
"type": "multi_turn",
|
"type": "multi_turn",
|
||||||
|
"options": {
|
||||||
|
"connector": "openai-gpt4",
|
||||||
|
"skip": {
|
||||||
|
"history": true
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"test_scenario": "happy-path"
|
||||||
|
}
|
||||||
|
},
|
||||||
"turns": [
|
"turns": [
|
||||||
{
|
{
|
||||||
"input": "I want to submit an expense report",
|
"input": "I want to submit an expense report",
|
||||||
|
|
@ -252,9 +502,13 @@ func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) {
|
||||||
],
|
],
|
||||||
"simulator": {
|
"simulator": {
|
||||||
"use": "agents:workers.test.user-simulator",
|
"use": "agents:workers.test.user-simulator",
|
||||||
"persona": "New employee unfamiliar with expense process",
|
"options": {
|
||||||
"goal": "Submit a $3500 travel expense",
|
"metadata": {
|
||||||
"max_turns": 10
|
"persona": "New employee unfamiliar with expense process",
|
||||||
|
"goal": "Submit a $3500 travel expense",
|
||||||
|
"max_turns": 10
|
||||||
|
}
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"interactive": {
|
"interactive": {
|
||||||
"enabled": false,
|
"enabled": false,
|
||||||
|
|
@ -278,14 +532,19 @@ func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) {
|
||||||
| `id` | string | Yes | Unique test identifier |
|
| `id` | string | Yes | Unique test identifier |
|
||||||
| `name` | string | No | Human-readable test name |
|
| `name` | string | No | Human-readable test name |
|
||||||
| `type` | string | No | `"single_turn"` (default) or `"multi_turn"` |
|
| `type` | string | No | `"single_turn"` (default) or `"multi_turn"` |
|
||||||
|
| `options` | object | No | `context.Options` passed to target agent |
|
||||||
|
| `options.connector` | string | No | LLM connector to use |
|
||||||
|
| `options.skip` | object | No | Skip config (history, trace, etc.) |
|
||||||
|
| `options.search` | any | No | Search behavior control |
|
||||||
|
| `options.mode` | string | No | Agent mode |
|
||||||
|
| `options.metadata` | object | No | Custom metadata passed to agent |
|
||||||
| `turns` | array | No | Static turn definitions |
|
| `turns` | array | No | Static turn definitions |
|
||||||
| `turns[].input` | string | Yes | User input for this turn |
|
| `turns[].input` | string | Yes | User input for this turn |
|
||||||
| `turns[].assertions` | array | No | Assertions for this turn's response |
|
| `turns[].assertions` | array | No | Assertions for this turn's response |
|
||||||
|
| `turns[].options` | object | No | Per-turn options override |
|
||||||
| `simulator` | object | No | Dynamic input generator configuration |
|
| `simulator` | object | No | Dynamic input generator configuration |
|
||||||
| `simulator.use` | string | Yes | Simulator reference: `agents:id` or `scripts:id` |
|
| `simulator.use` | string | Yes | Simulator reference: `agents:id` or `scripts:id` |
|
||||||
| `simulator.persona` | string | No | User persona description |
|
| `simulator.options` | object | No | `context.Options` passed to simulator agent |
|
||||||
| `simulator.goal` | string | No | What the simulated user wants to achieve |
|
|
||||||
| `simulator.max_turns` | int | No | Maximum turns before timeout (default: 20) |
|
|
||||||
| `interactive` | object | No | Interactive mode configuration |
|
| `interactive` | object | No | Interactive mode configuration |
|
||||||
| `interactive.enabled` | bool | No | Enable human input (default: false) |
|
| `interactive.enabled` | bool | No | Enable human input (default: false) |
|
||||||
| `interactive.timeout` | string | No | Timeout for human input (default: "5m") |
|
| `interactive.timeout` | string | No | Timeout for human input (default: "5m") |
|
||||||
|
|
@ -352,7 +611,9 @@ This allows hybrid testing: define some turns statically, then let simulator han
|
||||||
],
|
],
|
||||||
"simulator": {
|
"simulator": {
|
||||||
"use": "agents:workers.test.user-sim",
|
"use": "agents:workers.test.user-sim",
|
||||||
"goal": "Complete the expense submission"
|
"metadata": {
|
||||||
|
"goal": "Complete the expense submission"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
@ -790,21 +1051,28 @@ type ConversationContext struct {
|
||||||
|
|
||||||
### Context Passing to Simulator
|
### Context Passing to Simulator
|
||||||
|
|
||||||
The simulator receives full context to generate appropriate responses:
|
The simulator receives context via standard Yao Agent Context metadata:
|
||||||
|
|
||||||
```json
|
```go
|
||||||
{
|
// Framework prepares context for simulator
|
||||||
"persona": "New employee",
|
ctx.Metadata = map[string]any{
|
||||||
"goal": "Submit expense report",
|
"test_mode": "simulator",
|
||||||
"context": {
|
"test_id": "T001",
|
||||||
"session_id": "test-session-001",
|
// From simulator config
|
||||||
"messages": [...],
|
"persona": "New employee",
|
||||||
"tool_results": {
|
"goal": "Submit expense report",
|
||||||
"create_expense": {"id": "EXP-001", "status": "pending"}
|
// Runtime context
|
||||||
},
|
"session_id": "test-session-001",
|
||||||
"turn_count": 3
|
"turn_count": 3,
|
||||||
},
|
"tool_results": map[string]any{...},
|
||||||
"last_response": "Please confirm the expense details..."
|
}
|
||||||
|
|
||||||
|
// Messages include conversation history
|
||||||
|
messages := []Message{
|
||||||
|
{Role: "user", Content: "I want to submit an expense"},
|
||||||
|
{Role: "assistant", Content: "What type of expense?"},
|
||||||
|
{Role: "user", Content: "Travel expense"},
|
||||||
|
{Role: "assistant", Content: "Please confirm the details..."},
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,9 @@
|
||||||
## Phase 1: Static Multi-Turn
|
## Phase 1: Static Multi-Turn
|
||||||
|
|
||||||
- [ ] Extend test case parser for `turns` array
|
- [ ] Extend test case parser for `turns` array
|
||||||
- [ ] Implement turn-by-turn execution
|
- [ ] Add `options` field support (aligned with `context.Options`)
|
||||||
|
- [ ] Support test-level `options` and per-turn `options` override
|
||||||
|
- [ ] Implement turn-by-turn execution with options passing
|
||||||
- [ ] Implement conversation context management
|
- [ ] Implement conversation context management
|
||||||
- [ ] Add per-turn assertions
|
- [ ] Add per-turn assertions
|
||||||
- [ ] Support attachments at turn level
|
- [ ] Support attachments at turn level
|
||||||
|
|
@ -17,22 +19,24 @@
|
||||||
|
|
||||||
- [ ] Parse `agents:` prefix in `-i` flag
|
- [ ] Parse `agents:` prefix in `-i` flag
|
||||||
- [ ] Parse `scripts:` prefix in `-i` flag
|
- [ ] Parse `scripts:` prefix in `-i` flag
|
||||||
- [ ] Define generator agent interface (GeneratorInput/GeneratorOutput)
|
- [ ] Use standard `context.Options` for all agent invocations
|
||||||
- [ ] Implement generator invocation
|
- [ ] Pass `test_mode: "generator"` in `options.metadata`
|
||||||
- [ ] Support query parameters (`?count=10&focus=...`)
|
- [ ] Pass target agent info (description, tools) in `options.metadata`
|
||||||
- [ ] Pass target agent metadata to generator (description, tools)
|
- [ ] Support query parameters (`?count=10&focus=...`) → merged into `options.metadata`
|
||||||
- [ ] Add `--dry-run` flag to save generated cases without running
|
- [ ] Add `--dry-run` flag to save generated cases without running
|
||||||
- [ ] Create example generator agent
|
- [ ] Create example generator agent with prompt template
|
||||||
|
|
||||||
## Phase 3: Dynamic Simulator
|
## Phase 3: Dynamic Simulator
|
||||||
|
|
||||||
- [ ] Define simulator agent interface (SimulatorInput/SimulatorOutput)
|
- [ ] Implement simulator invocation via `Assistant.Stream()` with `context.Options`
|
||||||
- [ ] Implement simulator invocation with full context
|
- [ ] Pass `test_mode: "simulator"` in `options.metadata`
|
||||||
- [ ] Pass conversation history and tool results to simulator
|
- [ ] Pass persona, goal, turn_count in `options.metadata`
|
||||||
- [ ] Add goal completion detection
|
- [ ] Pass conversation history as messages
|
||||||
- [ ] Add max_turns limit and timeout
|
- [ ] Pass tool results in `options.metadata`
|
||||||
|
- [ ] Add goal completion detection (`goal_achieved` in response)
|
||||||
|
- [ ] Add max_turns limit and timeout (from `options.metadata`)
|
||||||
- [ ] Support hybrid mode (static turns + simulator fallback)
|
- [ ] Support hybrid mode (static turns + simulator fallback)
|
||||||
- [ ] Create example simulator agent
|
- [ ] Create example simulator agent with prompt template
|
||||||
|
|
||||||
## Phase 4: Interactive Mode
|
## Phase 4: Interactive Mode
|
||||||
|
|
||||||
|
|
@ -49,7 +53,18 @@
|
||||||
- [ ] Add configurable detection rules
|
- [ ] Add configurable detection rules
|
||||||
- [ ] Support custom detection via script/agent
|
- [ ] Support custom detection via script/agent
|
||||||
|
|
||||||
## Phase 6: Error Handling & Reporting
|
## Phase 6: Agent-Driven Assertions
|
||||||
|
|
||||||
|
- [ ] Add `agent` assertion type to assertion parser
|
||||||
|
- [ ] Support `options` field in assertion (aligned with `context.Options`)
|
||||||
|
- [ ] Implement validator agent invocation via `Assistant.Stream()`
|
||||||
|
- [ ] Pass `test_mode: "validator"` in `options.metadata`
|
||||||
|
- [ ] Pass conversation context and criteria in `options.metadata`
|
||||||
|
- [ ] Support score-based pass/fail threshold (configurable in `options.metadata`)
|
||||||
|
- [ ] Add `suggestions` to assertion error output
|
||||||
|
- [ ] Create example validator agent with prompt template
|
||||||
|
|
||||||
|
## Phase 7: Error Handling & Reporting
|
||||||
|
|
||||||
- [ ] Implement turn-level error handling
|
- [ ] Implement turn-level error handling
|
||||||
- [ ] Implement test-level error aggregation
|
- [ ] Implement test-level error aggregation
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue