opencode/yao
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
yao/agent/test/README.md
Max 7f44e442da Enhance File Handling and Input Parsing in Test Framework 
- Introduced support for file attachments in test inputs using the `file://` protocol, allowing images, audio, and documents to be loaded and converted to appropriate formats.
- Updated `ParseInput` and related functions to handle file references, ensuring seamless integration of file content into messages.
- Enhanced error handling and path resolution for file loading, considering both relative paths and the `YAO_ROOT` environment variable.
- Expanded documentation to include examples of file attachments and their usage in test cases, improving clarity for users.
2025-12-25 11:50:58 +08:00

708 lines
21 KiB
Markdown
Raw Blame History

# Agent Test Framework
A testing framework for Yao AI agents with support for assertions, stability analysis, and CI integration.
## Quick Start
### Agent Tests
```bash
# Test with direct message (auto-detect agent from current directory)
cd assistants/keyword
yao agent test -i "Extract keywords from: AI and machine learning"
# Test with direct message (specify agent explicitly)
yao agent test -i "Hello world" -n workers.system.keyword
# Test with JSONL file (auto-detect agent from path)
yao agent test -i assistants/keyword/tests/inputs.jsonl
# Generate HTML report
yao agent test -i tests/inputs.jsonl -o report.html
# Stability analysis (run each test 5 times)
yao agent test -i tests/inputs.jsonl --runs 5
```
### Script Tests
```bash
# Test agent handler scripts (hooks, tools, setup functions)
yao agent test -i scripts.expense.setup -v
# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady" -v
# Run with custom context (authorization, metadata)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```
## Input Modes
The `-i` flag supports three input modes:
### 1. JSONL File Mode
Load test cases from a file:
```bash
yao agent test -i tests/inputs.jsonl
```
Agent is auto-detected by traversing up from the input file to find `package.yao`.
### 2. Direct Message Mode
Test with a single message:
```bash
# Auto-detect agent from current working directory
cd assistants/keyword
yao agent test -i "Extract keywords from this text"
# Or specify agent explicitly
yao agent test -i "Hello" -n workers.system.keyword
```
Output is printed to stdout (or saved to `-o` if specified).
### 3. Script Test Mode
Test agent handler scripts (hooks, tools, setup functions):
```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v
# Run specific tests with filtering
yao agent test -i scripts.expense.setup --run "TestSystemReady"
# Run with custom context
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```
Script test input format: `scripts.<assistant>.<module>` (e.g., `scripts.expense.setup` → `assistants/expense/src/setup_test.ts`).
**Writing Test Scripts:**
Test scripts should be placed alongside the source files with `_test.ts` or `_test.js` suffix:
```
assistants/expense/src/
├── setup.ts # Source file
├── setup_test.ts # Test file
├── tools.ts
└── tools_test.ts
```
Test functions must follow the naming convention `Test*` and accept `(t: testing.T, ctx: agent.Context)`:
```typescript
// assistants/expense/src/setup_test.ts
import { SystemReady } from "./setup";
// Test function signature: function Test*(t: testing.T, ctx: agent.Context)
export function TestSystemReady(t: testing.T, ctx: agent.Context) {
const result = SystemReady(ctx);
// Use t.assert for assertions
t.assert.True(result.success, "SystemReady should succeed");
t.assert.Equal(result.status, "ready", "Status should be ready");
t.assert.NotNil(result.data, "Data should not be nil");
}
export function TestSystemReadyError(t: testing.T, ctx: agent.Context) {
// Access context properties
console.log("Testing with user:", ctx.authorized.user_id);
// Test error handling
const result = SystemReady(ctx);
t.assert.False(result.error, "Should not have error");
}
```
**Available Assertions:**
| Method | Description |
| -------------------------------- | ------------------------------ |
| `t.assert.True(value, msg)` | Assert value is true |
| `t.assert.False(value, msg)` | Assert value is false |
| `t.assert.Equal(a, b, msg)` | Assert a equals b |
| `t.assert.NotEqual(a, b, msg)` | Assert a not equals b |
| `t.assert.Nil(value, msg)` | Assert value is null/undefined |
| `t.assert.NotNil(value, msg)` | Assert value is not nil |
| `t.assert.Contains(s, sub, msg)` | Assert string contains substr |
| `t.assert.Len(arr, n, msg)` | Assert array/string length |
**Test Control:**
| Method | Description |
| -------------- | ---------------------------- |
| `t.Log(msg)` | Log a message |
| `t.Error(msg)` | Mark test as failed with msg |
| `t.Fatal(msg)` | Mark failed and stop test |
| `t.Skip(msg)` | Skip this test |
## Command Line Options
| Flag | Description | Default |
| ------------- | -------------------------------------------------- | -------------------------- |
| `-i` | Input: JSONL file path, message, or script ID | (required) |
| `-o` | Output file path | `output-{timestamp}.jsonl` |
| `-n` | Agent ID (optional, auto-detected) | auto-detect |
| `-c` | Override connector | agent default |
| `-u` | Test user ID | `test-user` |
| `-t` | Test team ID | `test-team` |
| `--ctx` | Path to context JSON file for custom authorization | - |
| `-r` | Reporter agent ID | built-in |
| `--runs` | Runs per test (stability analysis) | 1 |
| `--run` | Regex pattern to filter which tests to run | - |
| `--timeout` | Timeout per test | 5m |
| `--parallel` | Parallel test cases | 1 |
| `-v` | Verbose output | false |
| `--fail-fast` | Stop on first failure | false |
## Agent Resolution
The agent is resolved in the following priority order:
1. **Explicit `-n` flag**: `yao agent test -i "msg" -n my.agent`
2. **Path-based detection**: Traverse up from input file to find `package.yao`
3. **Current directory**: For direct message mode, look for `package.yao` in cwd
Example directory structure:
```
assistants/workers/system/keyword/
├── package.yao <- Agent definition (auto-detected)
├── prompts.yml
├── src/
│ └── index.ts
└── tests/
└── inputs.jsonl <- Input file
```
## Input Format (JSONL)
Each line is a JSON object:
```jsonl
{"id": "T001", "input": "Simple text"}
{"id": "T002", "input": {"role": "user", "content": "Message with role"}}
{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]}
{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}}
{"id": "T005", "input": "Skip this", "skip": true}
```
### Fields
| Field | Type | Required | Description |
| ---------- | ------------------------------ | -------- | ------------------------------ |
| `id` | string | Yes | Test case ID |
| `input` | string \| Message \| []Message | Yes | Test input |
| `assert` | Assertion \| []Assertion | No | Assertion rules |
| `expected` | any | No | Expected output (exact match) |
| `user` | string | No | Override user ID |
| `team` | string | No | Override team ID |
| `options` | Options | No | Context options (see below) |
| `timeout` | string | No | Override timeout (e.g., "30s") |
| `skip` | bool | No | Skip this test |
| `metadata` | map | No | Additional metadata |
### Options
The `options` field allows per-test-case configuration that maps to `context.Options`:
| Field | Type | Description |
| ------------------------ | ------ | -------------------------------------------- |
| `connector` | string | Override connector (e.g., `"deepseek.v3"`) |
| `mode` | string | Agent mode (default: `"chat"`) |
| `search` | bool | Enable/disable search mode (default: `true`) |
| `disable_global_prompts` | bool | Temporarily disable global prompts |
| `metadata` | map | Custom data passed to hooks (e.g., scenario) |
| `skip` | object | Skip configuration (see below) |
#### Options.skip
| Field | Type | Description |
| --------- | ---- | ----------------------- |
| `history` | bool | Skip history loading |
| `trace` | bool | Skip trace logging |
| `output` | bool | Skip output to client |
| `keyword` | bool | Skip keyword extraction |
| `search` | bool | Skip auto search |
**Example with options:**
```jsonl
{
"id": "T001",
"input": "Query users with status active",
"options": {
"connector": "deepseek.v3",
"metadata": {
"scenario": "filter"
},
"skip": {
"trace": true
}
},
"assert": {
"type": "json_path",
"path": "from",
"value": "users"
}
}
```
**Using metadata for hook scenarios:**
The `options.metadata` field is passed to agent hooks. For example, a Create Hook can read `options.metadata.scenario` to select different prompt presets:
```jsonl
{"id": "T001", "input": "...", "options": {"metadata": {"scenario": "aggregation"}}}
{"id": "T002", "input": "...", "options": {"metadata": {"scenario": "join"}}}
```
### Input Types
| Type | Description | Example |
| ----------- | -------------------- | ----------------------------------------------------- |
| `string` | Simple text | `"Hello world"` |
| `Message` | Single message | `{"role": "user", "content": "..."}` |
| `[]Message` | Conversation history | `[{"role": "user", ...}, {"role": "assistant", ...}]` |
### File Attachments
Test inputs support file attachments (images, audio, documents) using the `file://` protocol. Files are loaded and converted to appropriate formats for the LLM.
**Supported file types:**
| Type | Extensions | Format |
| ------ | ---------------------------------------------------------------------- | ------------------------------ |
| Image | `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.bmp` | Base64 data URL in `image_url` |
| Audio | `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a` | Base64 in `input_audio` |
| Doc | `.pdf`, `.doc`, `.docx`, `.xls`, `.xlsx`, `.txt`, `.csv`, `.json` | Base64 data URL in `file` |
| Source | `.yao`, `.ts`, `.js`, `.go`, `.py`, `.rs`, `.java`, `.sql`, `.yaml`... | Base64 data URL in `file` |
**File path resolution:**
- **Relative paths**: Resolved relative to the JSONL input file's directory (for file mode) or current working directory (for message mode)
- **Absolute paths**: Used as-is
**Example with image attachment:**
```jsonl
{
"id": "T001",
"input": {
"role": "user",
"content": [
{
"type": "text",
"text": "Please analyze this invoice"
},
{
"type": "image",
"source": "file://fixtures/invoice.jpg"
}
]
},
"assert": {
"type": "contains",
"value": "amount"
}
}
```
**Example with multiple attachments:**
```jsonl
{
"id": "T002",
"input": {
"role": "user",
"content": [
{
"type": "text",
"text": "Process these receipts"
},
{
"type": "image",
"source": "file://fixtures/receipt1.png"
},
{
"type": "image",
"source": "file://fixtures/receipt2.png"
},
{
"type": "file",
"source": "file://fixtures/policy.pdf",
"name": "expense_policy.pdf"
}
]
}
}
```
**Example with audio:**
```jsonl
{
"id": "T003",
"input": {
"role": "user",
"content": [
{
"type": "text",
"text": "Transcribe this audio"
},
{
"type": "audio",
"source": "file://fixtures/recording.wav"
}
]
}
}
```
**Content part types:**
| Type | Fields | Description |
| ----------- | --------------------------------------- | -------------------------------- |
| `text` | `text` | Text content |
| `image` | `source` (file://) or `url` | Image attachment |
| `image_url` | `image_url: {url, detail?}` | Direct image URL (OpenAI format) |
| `audio` | `source` (file://) or `data`, `format` | Audio attachment |
| `file` | `source` (file://) or `url`, `filename` | Document attachment |
| `data` | `data: {sources: [...]}` | Data source references |
## Assertions
Use `assert` for flexible validation. If `assert` is defined, it takes precedence over `expected`.
### Assertion Types
| Type | Description | Example |
| -------------- | ----------------------------- | --------------------------------------------------------- |
| `equals` | Exact match | `{"type": "equals", "value": {"key": "val"}}` |
| `contains` | Output contains value | `{"type": "contains", "value": "keyword"}` |
| `not_contains` | Output does not contain value | `{"type": "not_contains", "value": "error"}` |
| `json_path` | Extract JSON path and compare | `{"type": "json_path", "path": "$.field", "value": true}` |
| `regex` | Match regex pattern | `{"type": "regex", "value": "\\d+"}` |
| `type` | Check output type | `{"type": "type", "value": "object"}` |
| `script` | Run custom assertion script | `{"type": "script", "script": "scripts.test.Check"}` |
### Assertion Options
| Field | Type | Description |
| --------- | ------ | --------------------------- |
| `type` | string | Assertion type (required) |
| `value` | any | Expected value or pattern |
| `path` | string | JSON path (for `json_path`) |
| `script` | string | Script name (for `script`) |
| `message` | string | Custom failure message |
| `negate` | bool | Invert the result |
### Examples
**JSON path validation:**
```jsonl
{
"id": "T001",
"input": "What's the weather?",
"assert": {
"type": "json_path",
"path": "need_search",
"value": true
}
}
```
**Multiple assertions (all must pass):**
```jsonl
{
"id": "T002",
"input": "Hello",
"assert": [
{
"type": "json_path",
"path": "need_search",
"value": false
},
{
"type": "not_contains",
"value": "error"
}
]
}
```
**Custom script assertion:**
```jsonl
{
"id": "T003",
"input": "Test",
"assert": {
"type": "script",
"script": "scripts.test.Validate"
}
}
```
Script receives `(output, input, expected)` and returns:
```javascript
// Boolean
return true;
// Or detailed result
return { pass: true, message: "Validation passed" };
```
**Negated assertion:**
```jsonl
{
"id": "T004",
"input": "Hello",
"assert": {
"type": "contains",
"value": "error",
"negate": true
}
}
```
### JSON Path Notes
- Supports dot-notation: `$.field.subfield` or `field.subfield`
- Supports array indexing: `field[0]`, `field[0].subfield`, `field[0].nested[1]`
- Supports multiple expected values (OR logic): `"value": ["a", "b"]` - passes if actual matches any
- Auto-extracts JSON from markdown code blocks (` ```json ... ``` `)
- Works with both string output and structured objects
**Array index examples:**
```jsonl
{"id": "T001", "assert": {"type": "json_path", "path": "wheres[0].like", "value": "%test%"}}
{"id": "T002", "assert": {"type": "json_path", "path": "wheres[0].in[0]", "value": "pending"}}
{"id": "T003", "assert": {"type": "json_path", "path": "joins[0].from", "value": "users"}}
{"id": "T004", "assert": {"type": "json_path", "path": "groups[0]", "value": "category"}}
```
**Multiple expected values (OR logic):**
```jsonl
{
"id": "T005",
"assert": {
"type": "json_path",
"path": "error",
"value": [
"missing_schema",
"missing_query"
]
}
}
```
This passes if `error` equals either `"missing_schema"` or `"missing_query"`.
## Output Formats
Determined by `-o` file extension:
| Extension | Format | Description |
| --------- | -------- | ---------------------- |
| `.jsonl` | JSONL | Streaming (default) |
| `.json` | JSON | Complete structured |
| `.md` | Markdown | Human-readable |
| `.html` | HTML | Interactive web report |
### Default Output Path
When `-o` is not specified in file mode:
```
{input_directory}/output-{timestamp}.jsonl
```
Example: `tests/output-20241217100000.jsonl`
In direct message mode without `-o`, output is printed to stdout.
## Stability Analysis
Run each test multiple times to measure consistency:
```bash
yao agent test -i tests/inputs.jsonl --runs 5 -o stability.json
```
Output includes:
- Pass rate per test
- Stability classification (stable, mostly_stable, unstable, highly_unstable)
- Average/min/max duration
- Standard deviation
### Stability Classification
| Pass Rate | Classification |
| --------- | --------------- |
| 100% | Stable |
| 80-99% | Mostly Stable |
| 50-79% | Unstable |
| < 50% | Highly Unstable |
## Test Environment
The test framework creates a context with configurable environment:
| Setting | Flag | Default |
| ---------- | ---- | ----------- |
| User ID | `-u` | `test-user` |
| Team ID | `-t` | `test-team` |
| Locale | - | `en-us` |
| ClientType | - | `test` |
| ClientIP | - | `127.0.0.1` |
Priority: Command line flags > Test case fields > Defaults
## Custom Reporter Agent
Use `-r` to specify a custom agent for report generation:
```bash
yao agent test -i tests/inputs.jsonl -r report.beautiful -o report.html
```
The reporter agent receives:
```json
{
"report": { "summary": {...}, "results": [...] },
"format": "html",
"options": { "verbose": true }
}
```
## CI Integration
```bash
# Exit code: 0 = all passed, 1 = failures
yao agent test -i tests/inputs.jsonl -o results.jsonl --fail-fast
# Parse JSONL results
cat results.jsonl | jq 'select(.type == "summary")'
# Run script tests
yao agent test -i scripts.expense.setup --fail-fast
```
### GitHub Actions Example
```yaml
- name: Run Agent Tests
run: |
yao agent test -i assistants/keyword/tests/inputs.jsonl \
-u ci-user -t ci-team \
--runs 3 \
-o report.json
- name: Run Script Tests
run: |
yao agent test -i scripts.expense.setup -v
yao agent test -i scripts.expense.tools -v
- name: Run Script Tests with Custom Context
run: |
yao agent test -i scripts.expense.setup \
--ctx tests/context.json \
--run "TestSystem.*" \
-v
- name: Check Stability
run: |
jq -e '.results | all(.pass_rate >= 80)' report.json
```
## Examples
### Agent Tests
```bash
# Quick development test (auto-detect agent)
cd assistants/keyword
yao agent test -i "Extract keywords: AI and ML"
# Quick development test (specify agent)
yao agent test -i "Hello" -n workers.system.keyword
# Full test suite with HTML report
yao agent test -i tests/inputs.jsonl -o report.html -v
# Override connector
yao agent test -i tests/inputs.jsonl -c openai.gpt4
# Stability analysis
yao agent test -i tests/inputs.jsonl --runs 10 -o stability.json
# Parallel execution with timeout
yao agent test -i tests/inputs.jsonl --parallel 4 --timeout 2m
# Custom test environment
yao agent test -i tests/inputs.jsonl -u admin -t prod-team
# Custom reporter agent
yao agent test -i tests/inputs.jsonl -r report.beautiful -o custom-report.md
# Full example with all options
yao agent test -i tests/inputs.jsonl \
-n keyword.agent \
-c deepseek.v3 \
-u test-user \
-t test-team \
--runs 3 \
--timeout 10m \
--parallel 4 \
-r report.html \
-o report.html
```
### Script Tests
```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v
# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady"
# Run tests matching a pattern
yao agent test -i scripts.expense.setup --run "TestSystem.*" -v
# Run with custom context (authorization, metadata, etc.)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
# Run with specific user/team
yao agent test -i scripts.expense.setup -u admin -t ops-team -v
# Combine options
yao agent test -i scripts.expense.setup \
--ctx tests/context.json \
--run "TestSystem.*" \
--timeout 30s \
-v
```
## Exit Codes
| Code | Description |
| ---- | --------------------------------------------------- |
| 0 | All tests passed |
| 1 | Tests failed, configuration error, or runtime error |
Reference in a new issue View git blame Copy permalink