diff --git a/.gitignore b/.gitignore index d11f893d..d1b23f38 100644 --- a/.gitignore +++ b/.gitignore @@ -53,3 +53,4 @@ agent/assistant/hook/*.test.md agent/search/TODO.md agent/search/job-logs.txt agent/test/MULTI_TURN_DESIGN.md +agent/test/UPGRADE_PLAN.md diff --git a/agent/test/TODO_V2.md b/agent/test/TODO_V2.md index 6309ed87..8b66e7ff 100644 --- a/agent/test/TODO_V2.md +++ b/agent/test/TODO_V2.md @@ -1,114 +1,87 @@ -# Agent Test Framework V2 - Implementation TODO +# Agent Test Framework V2 - TODO -## Format Rules Summary +> 详细实施计划见 [UPGRADE_PLAN.md](./UPGRADE_PLAN.md) -| Context | Format | Example | -| -------------------- | ------------------------ | ------------------------------------------------------- | -| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | -| JSONL assertion `use`| Prefix required | `"use": "agents:workers.test.validator"` | -| JSONL `simulator.use`| No prefix (agent only) | `"use": "workers.test.user-simulator"` | -| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` | -| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.validator", {...})` | -| JSONL `before/after` | `scripts:` prefix | `"before": "scripts:tests.env.Before"` | -| `--before/--after` | `scripts:` prefix | `--before scripts:tests.env.BeforeAll` | +## Format Rules -## Phase 1: Message History Support ✅ (Already Implemented) +| Context | Format | Example | +| --------------------- | ------------------------ | ---------------------------------------------- | +| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | +| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` | +| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` | +| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` | +| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.val")` | +| JSONL `before/after` | `scripts:` prefix | `"before": "scripts:tests.env.Before"` | +| `--before/--after` | `scripts:` prefix | `--before scripts:tests.env.BeforeAll` | -The `input` field already supports: -- `string`: Simple text input -- `object`: Single message `{role, content}` -- `array`: Message history `[{role, content}, ...]` +## Phase 1: Before/After Scripts -See `input.go` → `ParseInputWithOptions()` for implementation. +**新增文件**: `script_hooks.go` -**Remaining tasks:** -- [ ] Add `options` field support (aligned with `context.Options`) - partially done via `CaseOptions` -- [x] Support attachments in message content parts (file:// protocol) -- [ ] Update console output to show message count -- [ ] Update JSON output format with `messages_count` +- [ ] `types.go`: 添加 `Before`, `After` 字段到 `Case` +- [ ] `types.go`: 添加 `BeforeAll`, `AfterAll` 字段到 `Options` +- [ ] `script_hooks.go`: 实现 `HookExecutor` +- [ ] `script_hooks.go`: 解析 `scripts:` 前缀 +- [ ] `runner.go`: 集成 before/after 到 `runSingleTest` +- [ ] `runner.go`: 集成 beforeAll/afterAll 到 `RunTests` +- [ ] `cmd/agent/agent.go`: 添加 `--before`, `--after` flags +- [ ] 创建示例脚本 `tests/env_test.ts` -## Phase 2: Agent-Driven Input +## Phase 2: Agent-Driven Assertions -- [ ] Parse `agents:` prefix in `-i` flag -- [ ] Parse `scripts:` prefix in `-i` flag -- [ ] Use standard `context.Options` for generator invocation -- [ ] Pass `test_mode: "generator"` in `options.metadata` -- [ ] Pass target agent info (description, tools) in `options.metadata` -- [ ] Support query parameters (`?count=10&focus=...`) → merged into `options.metadata` -- [ ] Add `--dry-run` flag to save generated cases without running -- [ ] Create example generator agent with prompt template +**修改文件**: `assert.go`, `script_assert.go` -## Phase 3: Dynamic Mode (Checkpoints) +- [ ] `types.go`: 添加 `Use`, `Options` 字段到 `Assertion` +- [ ] `assert.go`: 实现 `assertAgent` 方法 +- [ ] `assert.go`: 在 `evaluateAssertion` 添加 `agent` 类型 +- [ ] `script_assert.go`: 添加 `AssertAgent` 方法到 `TestingT` +- [ ] 创建示例 validator agent -- [ ] Add `checkpoints` array to test case parser -- [ ] Add `simulator` field to test case parser -- [ ] Implement checkpoint matching against agent responses -- [ ] Support `after` field for order constraints -- [ ] Track pending/reached checkpoints during execution -- [ ] Implement termination conditions: - - [ ] All checkpoints reached → PASSED - - [ ] Simulator signals goal_achieved but checkpoints missing → FAILED - - [ ] max_turns exceeded → FAILED - - [ ] timeout exceeded → FAILED -- [ ] Implement simulator invocation via `Assistant.Stream()` -- [ ] `simulator.use` is direct agent ID (no prefix needed) -- [ ] Pass `test_mode: "simulator"` in `options.metadata` -- [ ] Pass persona, goal, turn_count from `simulator.options.metadata` -- [ ] Pass conversation history as messages -- [ ] Create example simulator agent with prompt template +## Phase 3: Dynamic Mode (Simulator + Checkpoints) -## Phase 4: Agent-Driven Assertions +**新增文件**: `dynamic_runner.go`, `dynamic_types.go` -### In JSONL Test Cases +- [ ] `types.go`: 添加 `Simulator`, `Checkpoints` 字段到 `Case` +- [ ] `dynamic_types.go`: 定义 `Checkpoint`, `DynamicResult` 等类型 +- [ ] `dynamic_runner.go`: 实现 `DynamicRunner` +- [ ] `dynamic_runner.go`: 实现 checkpoint 匹配逻辑 +- [ ] `dynamic_runner.go`: 实现终止条件判断 +- [ ] `runner.go`: 在 `runSingleTest` 判断并调用动态模式 +- [ ] 创建示例 simulator agent -- [ ] Add `agent` assertion type to assertion parser -- [ ] Support `options` field in assertion (aligned with `context.Options`) -- [ ] Implement validator agent invocation via `Assistant.Stream()` -- [ ] Pass `test_mode: "validator"` in `options.metadata` -- [ ] Pass conversation context and criteria in `options.metadata` -- [ ] Support score-based pass/fail threshold (configurable in `options.metadata`) -- [ ] Add `suggestions` to assertion error output +## Phase 4: Agent-Driven Input -### In Script Tests +**新增文件**: `input_source.go` -- [ ] Add `t.assert.Agent(response, agentID, options?)` method -- [ ] `agentID` is direct ID (e.g., `workers.test.validator`), no prefix needed -- [ ] Invoke validator agent with context -- [ ] Return `ValidatorResult` object to JavaScript -- [ ] Support passing conversation history in options +- [ ] `input_source.go`: 实现 `ParseInputSource` +- [ ] `input_source.go`: 实现 `GenerateTestCases` +- [ ] `loader.go`: 添加 `LoadFromAgent` 方法 +- [ ] `loader.go`: 添加 `LoadFromScript` 方法 +- [ ] `runner.go`: 在 `RunTests` 支持不同输入源 +- [ ] `cmd/agent/agent.go`: 添加 `--dry-run` flag +- [ ] 创建示例 generator agent -### Shared +## Phase 5: Console Output Optimization -- [ ] Create example validator agent with prompt template -- [ ] Document `ValidatorResult` interface +**修改文件**: `output.go` -## Phase 5: Before and After Scripts +- [ ] `output.go`: 添加 `DynamicTestStart` 方法 +- [ ] `output.go`: 添加 `DynamicTurn` 方法 +- [ ] `output.go`: 添加 `DynamicTestResult` 方法 +- [ ] `output.go`: 添加 `ParallelResults` 方法 +- [ ] 测试并行模式输出效果 -- [ ] Add `before` field to test case parser -- [ ] Add `after` field to test case parser -- [ ] Add `--before` CLI flag for global before -- [ ] Add `--after` CLI flag for global after -- [ ] Parse `scripts:` prefix in before/after fields -- [ ] Implement script function invocation via `Process()` -- [ ] Pass `TestCase` object to before function -- [ ] Pass `TestResult` and before data to after function -- [ ] Ensure after runs even if test fails -- [ ] Support `BeforeAll()` for one-time initialization -- [ ] Support `AfterAll()` for final cleanup -- [ ] Create example before/after script +## Already Implemented ✅ -## Phase 6: Error Handling & Reporting - -- [ ] Implement test-level error handling -- [ ] Add detailed error messages with hints -- [ ] Support `--parallel` flag for concurrent test execution -- [ ] Support `--fail-fast` flag to stop on first failure -- [ ] Add verbose mode (`-v`) for detailed output +- [x] Message history support (`input` as array) +- [x] File attachments (`file://` protocol) +- [x] `--parallel` flag +- [x] `--fail-fast` flag +- [x] `-v` verbose mode +- [x] Script testing (`*_test.ts`) ## Open Questions -1. **Message Generation**: Should we provide a helper to generate message history from a script? - -2. **Snapshot Testing**: Should we support "golden file" comparison for responses? - -3. **Retry Logic**: If a test fails, should we support automatic retry? +1. **Message Generation**: 是否提供 helper 从脚本生成 message history? +2. **Snapshot Testing**: 是否支持 "golden file" 对比? +3. **Retry Logic**: 测试失败是否支持自动重试?