Update TODO_V2.md and .gitignore for New Test Plan Documentation
- Added `UPGRADE_PLAN.md` to .gitignore to exclude the new upgrade plan documentation from version control. - Revised TODO_V2.md to include a detailed implementation plan for the Agent Test Framework, outlining phases for before/after scripts, agent-driven assertions, and dynamic mode features. - Improved overall documentation clarity to facilitate understanding of upcoming enhancements and tasks within the framework.
This commit is contained in:
Max
parent
2ec28a1710
commit
45c01f9c04
2 changed files with 65 additions and 91 deletions
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -53,3 +53,4 @@ agent/assistant/hook/*.test.md
|
||||||
agent/search/TODO.md
|
agent/search/TODO.md
|
||||||
agent/search/job-logs.txt
|
agent/search/job-logs.txt
|
||||||
agent/test/MULTI_TURN_DESIGN.md
|
agent/test/MULTI_TURN_DESIGN.md
|
||||||
|
agent/test/UPGRADE_PLAN.md
|
||||||
|
|
|
||||||
|
|
@ -1,114 +1,87 @@
|
||||||
# Agent Test Framework V2 - Implementation TODO
|
# Agent Test Framework V2 - TODO
|
||||||
|
|
||||||
## Format Rules Summary
|
> 详细实施计划见 [UPGRADE_PLAN.md](./UPGRADE_PLAN.md)
|
||||||
|
|
||||||
| Context | Format | Example |
|
## Format Rules
|
||||||
| -------------------- | ------------------------ | ------------------------------------------------------- |
|
|
||||||
| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` |
|
|
||||||
| JSONL assertion `use`| Prefix required | `"use": "agents:workers.test.validator"` |
|
|
||||||
| JSONL `simulator.use`| No prefix (agent only) | `"use": "workers.test.user-simulator"` |
|
|
||||||
| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` |
|
|
||||||
| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.validator", {...})` |
|
|
||||||
| JSONL `before/after` | `scripts:` prefix | `"before": "scripts:tests.env.Before"` |
|
|
||||||
| `--before/--after` | `scripts:` prefix | `--before scripts:tests.env.BeforeAll` |
|
|
||||||
|
|
||||||
## Phase 1: Message History Support ✅ (Already Implemented)
|
| Context | Format | Example |
|
||||||
|
| --------------------- | ------------------------ | ---------------------------------------------- |
|
||||||
|
| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` |
|
||||||
|
| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` |
|
||||||
|
| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` |
|
||||||
|
| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` |
|
||||||
|
| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.val")` |
|
||||||
|
| JSONL `before/after` | `scripts:` prefix | `"before": "scripts:tests.env.Before"` |
|
||||||
|
| `--before/--after` | `scripts:` prefix | `--before scripts:tests.env.BeforeAll` |
|
||||||
|
|
||||||
The `input` field already supports:
|
## Phase 1: Before/After Scripts
|
||||||
- `string`: Simple text input
|
|
||||||
- `object`: Single message `{role, content}`
|
|
||||||
- `array`: Message history `[{role, content}, ...]`
|
|
||||||
|
|
||||||
See `input.go` → `ParseInputWithOptions()` for implementation.
|
**新增文件**: `script_hooks.go`
|
||||||
|
|
||||||
**Remaining tasks:**
|
- [ ] `types.go`: 添加 `Before`, `After` 字段到 `Case`
|
||||||
- [ ] Add `options` field support (aligned with `context.Options`) - partially done via `CaseOptions`
|
- [ ] `types.go`: 添加 `BeforeAll`, `AfterAll` 字段到 `Options`
|
||||||
- [x] Support attachments in message content parts (file:// protocol)
|
- [ ] `script_hooks.go`: 实现 `HookExecutor`
|
||||||
- [ ] Update console output to show message count
|
- [ ] `script_hooks.go`: 解析 `scripts:` 前缀
|
||||||
- [ ] Update JSON output format with `messages_count`
|
- [ ] `runner.go`: 集成 before/after 到 `runSingleTest`
|
||||||
|
- [ ] `runner.go`: 集成 beforeAll/afterAll 到 `RunTests`
|
||||||
|
- [ ] `cmd/agent/agent.go`: 添加 `--before`, `--after` flags
|
||||||
|
- [ ] 创建示例脚本 `tests/env_test.ts`
|
||||||
|
|
||||||
## Phase 2: Agent-Driven Input
|
## Phase 2: Agent-Driven Assertions
|
||||||
|
|
||||||
- [ ] Parse `agents:` prefix in `-i` flag
|
**修改文件**: `assert.go`, `script_assert.go`
|
||||||
- [ ] Parse `scripts:` prefix in `-i` flag
|
|
||||||
- [ ] Use standard `context.Options` for generator invocation
|
|
||||||
- [ ] Pass `test_mode: "generator"` in `options.metadata`
|
|
||||||
- [ ] Pass target agent info (description, tools) in `options.metadata`
|
|
||||||
- [ ] Support query parameters (`?count=10&focus=...`) → merged into `options.metadata`
|
|
||||||
- [ ] Add `--dry-run` flag to save generated cases without running
|
|
||||||
- [ ] Create example generator agent with prompt template
|
|
||||||
|
|
||||||
## Phase 3: Dynamic Mode (Checkpoints)
|
- [ ] `types.go`: 添加 `Use`, `Options` 字段到 `Assertion`
|
||||||
|
- [ ] `assert.go`: 实现 `assertAgent` 方法
|
||||||
|
- [ ] `assert.go`: 在 `evaluateAssertion` 添加 `agent` 类型
|
||||||
|
- [ ] `script_assert.go`: 添加 `AssertAgent` 方法到 `TestingT`
|
||||||
|
- [ ] 创建示例 validator agent
|
||||||
|
|
||||||
- [ ] Add `checkpoints` array to test case parser
|
## Phase 3: Dynamic Mode (Simulator + Checkpoints)
|
||||||
- [ ] Add `simulator` field to test case parser
|
|
||||||
- [ ] Implement checkpoint matching against agent responses
|
|
||||||
- [ ] Support `after` field for order constraints
|
|
||||||
- [ ] Track pending/reached checkpoints during execution
|
|
||||||
- [ ] Implement termination conditions:
|
|
||||||
- [ ] All checkpoints reached → PASSED
|
|
||||||
- [ ] Simulator signals goal_achieved but checkpoints missing → FAILED
|
|
||||||
- [ ] max_turns exceeded → FAILED
|
|
||||||
- [ ] timeout exceeded → FAILED
|
|
||||||
- [ ] Implement simulator invocation via `Assistant.Stream()`
|
|
||||||
- [ ] `simulator.use` is direct agent ID (no prefix needed)
|
|
||||||
- [ ] Pass `test_mode: "simulator"` in `options.metadata`
|
|
||||||
- [ ] Pass persona, goal, turn_count from `simulator.options.metadata`
|
|
||||||
- [ ] Pass conversation history as messages
|
|
||||||
- [ ] Create example simulator agent with prompt template
|
|
||||||
|
|
||||||
## Phase 4: Agent-Driven Assertions
|
**新增文件**: `dynamic_runner.go`, `dynamic_types.go`
|
||||||
|
|
||||||
### In JSONL Test Cases
|
- [ ] `types.go`: 添加 `Simulator`, `Checkpoints` 字段到 `Case`
|
||||||
|
- [ ] `dynamic_types.go`: 定义 `Checkpoint`, `DynamicResult` 等类型
|
||||||
|
- [ ] `dynamic_runner.go`: 实现 `DynamicRunner`
|
||||||
|
- [ ] `dynamic_runner.go`: 实现 checkpoint 匹配逻辑
|
||||||
|
- [ ] `dynamic_runner.go`: 实现终止条件判断
|
||||||
|
- [ ] `runner.go`: 在 `runSingleTest` 判断并调用动态模式
|
||||||
|
- [ ] 创建示例 simulator agent
|
||||||
|
|
||||||
- [ ] Add `agent` assertion type to assertion parser
|
## Phase 4: Agent-Driven Input
|
||||||
- [ ] Support `options` field in assertion (aligned with `context.Options`)
|
|
||||||
- [ ] Implement validator agent invocation via `Assistant.Stream()`
|
|
||||||
- [ ] Pass `test_mode: "validator"` in `options.metadata`
|
|
||||||
- [ ] Pass conversation context and criteria in `options.metadata`
|
|
||||||
- [ ] Support score-based pass/fail threshold (configurable in `options.metadata`)
|
|
||||||
- [ ] Add `suggestions` to assertion error output
|
|
||||||
|
|
||||||
### In Script Tests
|
**新增文件**: `input_source.go`
|
||||||
|
|
||||||
- [ ] Add `t.assert.Agent(response, agentID, options?)` method
|
- [ ] `input_source.go`: 实现 `ParseInputSource`
|
||||||
- [ ] `agentID` is direct ID (e.g., `workers.test.validator`), no prefix needed
|
- [ ] `input_source.go`: 实现 `GenerateTestCases`
|
||||||
- [ ] Invoke validator agent with context
|
- [ ] `loader.go`: 添加 `LoadFromAgent` 方法
|
||||||
- [ ] Return `ValidatorResult` object to JavaScript
|
- [ ] `loader.go`: 添加 `LoadFromScript` 方法
|
||||||
- [ ] Support passing conversation history in options
|
- [ ] `runner.go`: 在 `RunTests` 支持不同输入源
|
||||||
|
- [ ] `cmd/agent/agent.go`: 添加 `--dry-run` flag
|
||||||
|
- [ ] 创建示例 generator agent
|
||||||
|
|
||||||
### Shared
|
## Phase 5: Console Output Optimization
|
||||||
|
|
||||||
- [ ] Create example validator agent with prompt template
|
**修改文件**: `output.go`
|
||||||
- [ ] Document `ValidatorResult` interface
|
|
||||||
|
|
||||||
## Phase 5: Before and After Scripts
|
- [ ] `output.go`: 添加 `DynamicTestStart` 方法
|
||||||
|
- [ ] `output.go`: 添加 `DynamicTurn` 方法
|
||||||
|
- [ ] `output.go`: 添加 `DynamicTestResult` 方法
|
||||||
|
- [ ] `output.go`: 添加 `ParallelResults` 方法
|
||||||
|
- [ ] 测试并行模式输出效果
|
||||||
|
|
||||||
- [ ] Add `before` field to test case parser
|
## Already Implemented ✅
|
||||||
- [ ] Add `after` field to test case parser
|
|
||||||
- [ ] Add `--before` CLI flag for global before
|
|
||||||
- [ ] Add `--after` CLI flag for global after
|
|
||||||
- [ ] Parse `scripts:` prefix in before/after fields
|
|
||||||
- [ ] Implement script function invocation via `Process()`
|
|
||||||
- [ ] Pass `TestCase` object to before function
|
|
||||||
- [ ] Pass `TestResult` and before data to after function
|
|
||||||
- [ ] Ensure after runs even if test fails
|
|
||||||
- [ ] Support `BeforeAll()` for one-time initialization
|
|
||||||
- [ ] Support `AfterAll()` for final cleanup
|
|
||||||
- [ ] Create example before/after script
|
|
||||||
|
|
||||||
## Phase 6: Error Handling & Reporting
|
- [x] Message history support (`input` as array)
|
||||||
|
- [x] File attachments (`file://` protocol)
|
||||||
- [ ] Implement test-level error handling
|
- [x] `--parallel` flag
|
||||||
- [ ] Add detailed error messages with hints
|
- [x] `--fail-fast` flag
|
||||||
- [ ] Support `--parallel` flag for concurrent test execution
|
- [x] `-v` verbose mode
|
||||||
- [ ] Support `--fail-fast` flag to stop on first failure
|
- [x] Script testing (`*_test.ts`)
|
||||||
- [ ] Add verbose mode (`-v`) for detailed output
|
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
1. **Message Generation**: Should we provide a helper to generate message history from a script?
|
1. **Message Generation**: 是否提供 helper 从脚本生成 message history?
|
||||||
|
2. **Snapshot Testing**: 是否支持 "golden file" 对比?
|
||||||
2. **Snapshot Testing**: Should we support "golden file" comparison for responses?
|
3. **Retry Logic**: 测试失败是否支持自动重试?
|
||||||
|
|
||||||
3. **Retry Logic**: If a test fails, should we support automatic retry?
|
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue