opencode/yao
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
yao/agent/test/TODO_V2.md
Max 5351128f89 Refactor Before/After Script Integration in Agent Test Framework 
- Updated DESIGN_V2.md to reflect changes in the handling of before/after scripts, removing the `scripts:` prefix and clarifying their usage in JSONL test cases.
- Enhanced runner.go to integrate global before/after hooks, ensuring they execute correctly before and after test cases.
- Revised types.go to include new fields for before/after scripts in test case and options structures.
- Improved TODO_V2.md to track the implementation progress of before/after script functionality and related tasks.
- Added utility function LoadAgentTestScripts to facilitate loading of test scripts from the agent's src directory.
2025-12-26 09:44:56 +08:00

3.7 KiB
Raw Blame History

Agent Test Framework V2 - TODO

详细实施计划见 UPGRADE_PLAN.md

Format Rules

Context Format Example
-i flag (CLI) Prefix required agents:workers.test.gen, scripts:tests.gen
JSONL assertion use Prefix required "use": "agents:workers.test.validator"
JSONL simulator.use No prefix (agent only) "use": "workers.test.user-simulator"
--simulator flag No prefix (agent only) --simulator workers.test.user-simulator
t.assert.Agent() No prefix (method-bound) t.assert.Agent(resp, "workers.test.val")
JSONL before/after No prefix (in src/) "before": "env_test.Before"
--before/--after No prefix (in src/) --before env_test.BeforeAll

Phase 1: Before/After Scripts ✅

新增文件: script_hooks.go

  • types.go: 添加 Before, After 字段到 Case
  • types.go: 添加 BeforeAll, AfterAll 字段到 Options
  • script_hooks.go: 实现 HookExecutor
  • script_hooks.go: 通过 V8 直接执行 *_test.ts 脚本
  • runner.go: 集成 before/after 到 runSingleTest
  • runner.go: 集成 beforeAll/afterAll 到 RunTests
  • cmd/agent/test.go: 添加 --before, --after flags
  • test/utils.go: 添加 LoadAgentTestScripts() 通用函数
  • 创建示例脚本 assistants/tests/hooks-test/src/env_test.ts
  • 创建单元测试 script_hooks_test.go (黑盒测试)

Phase 2: Agent-Driven Assertions

修改文件: assert.go, script_assert.go

  • types.go: 添加 Use, Options 字段到 Assertion
  • assert.go: 实现 assertAgent 方法
  • assert.go: 在 evaluateAssertion 添加 agent 类型
  • script_assert.go: 添加 AssertAgent 方法到 TestingT
  • 创建示例 validator agent

Phase 3: Dynamic Mode (Simulator + Checkpoints)

新增文件: dynamic_runner.go, dynamic_types.go

  • types.go: 添加 Simulator, Checkpoints 字段到 Case
  • dynamic_types.go: 定义 Checkpoint, DynamicResult 等类型
  • dynamic_runner.go: 实现 DynamicRunner
  • dynamic_runner.go: 实现 checkpoint 匹配逻辑
  • dynamic_runner.go: 实现终止条件判断
  • runner.go: 在 runSingleTest 判断并调用动态模式
  • 创建示例 simulator agent

Phase 4: Agent-Driven Input

新增文件: input_source.go

  • input_source.go: 实现 ParseInputSource
  • input_source.go: 实现 GenerateTestCases
  • loader.go: 添加 LoadFromAgent 方法
  • loader.go: 添加 LoadFromScript 方法
  • runner.go: 在 RunTests 支持不同输入源
  • cmd/agent/agent.go: 添加 --dry-run flag
  • 创建示例 generator agent

Phase 5: Console Output Optimization

修改文件: output.go

  • output.go: 添加 DynamicTestStart 方法
  • output.go: 添加 DynamicTurn 方法
  • output.go: 添加 DynamicTestResult 方法
  • output.go: 添加 ParallelResults 方法
  • 测试并行模式输出效果

Already Implemented ✅

  • Message history support (input as array)
  • File attachments (file:// protocol)
  • --parallel flag
  • --fail-fast flag
  • -v verbose mode
  • Script testing (*_test.ts)
  • Before/After hooks (Phase 1)

Open Questions

  1. Message Generation: 是否提供 helper 从脚本生成 message history?
  2. Snapshot Testing: 是否支持 "golden file" 对比?
  3. Retry Logic: 测试失败是否支持自动重试?