diff --git a/ROADMAP.md b/ROADMAP.md index 117c8ce3a..6de747ab1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -228,23 +228,31 @@ flowchart LR ## Ideas and improvements +- [ ] Add performance metrics to the eval harness + - [ ] Raw metrics of tool calls, LLM calls, token counts, duration, etc + - [ ] Per-test scores + - [ ] Side-by-side comparison matrix + - [ ] Compare to other agent runtimes + - [ ] Compare to upstream origin picoclaw + - [ ] What benchmarks should we be running? + - [ ] What are the key performance metrics we should be tracking? - [ ] Add a new tool for the agent to use: `focus_search` - [ ] Clean up the main.go and extract to modules - [ ] Create a pure Application API that I/O calls into - - [ ] cli - - [ ] daemon - - [ ] web - - [ ] grpc - - [ ] http - - [ ] websocket - - [ ] tcp - - [ ] udp - - [ ] serial - - [ ] i2c - - [ ] spi - - [ ] pwm - - [ ] etc - [ ] all of these should be able to be configured and plugged in/out at runtime + - [ ] cli + - [ ] daemon + - [ ] web + - [ ] grpc + - [ ] http + - [ ] websocket + - [ ] tcp + - [ ] udp + - [ ] serial + - [ ] i2c + - [ ] spi + - [ ] pwm + - [ ] etc - [ ] Migrate to Cobra CLI framework - [ ] Use command-palette pattern for subcommands - [ ] keep cli commands as pure cli that calls into the application diff --git a/eval/README.md b/eval/README.md index 541905041..f06e73270 100644 --- a/eval/README.md +++ b/eval/README.md @@ -21,6 +21,14 @@ make eval-test make eval-compare ``` +## Runtime Invariants + +The current agent architecture has these always-on behaviors: + +- Memory is always enabled. +- Meta tools are always registered (`tool_search`, `tool_call`). +- Eval runs against a single runtime profile via `eval/bin/eval-runner`. + ## Architecture ``` @@ -30,7 +38,13 @@ eval/ │ ├── tool_calling.yaml │ ├── token_efficiency.yaml │ ├── multi_step.yaml -│ └── edge_cases.yaml +│ ├── edge_cases.yaml +│ ├── memory_ops.yaml +│ ├── meta_tools.yaml +│ ├── skills.yaml +│ ├── subagent.yaml +│ ├── reasoning.yaml +│ └── error_recovery.yaml ├── go_evals/ # Go-native component tests (memory, tools) ├── scripts/ # CI/comparison scripts │ └── compare.sh @@ -41,11 +55,9 @@ eval/ ## How It Works -1. **eval-runner** wraps picoclaw with an instrumented language model that captures - every LLM call, tool invocation, token count, and timing. +1. **eval-runner** wraps picoclaw with an instrumented language model that captures every LLM call, tool invocation, token count, and timing. -2. **promptfoo** invokes `eval-runner` via `exec:` provider, sending prompts as JSON - on stdin and parsing the structured trace JSON from stdout. +2. **promptfoo** invokes `eval-runner` via `exec:` provider, sending prompts as JSON on stdin and parsing the structured trace JSON from stdout. 3. **Assertions** are JavaScript functions that inspect the trace to score: - Tool selection correctness @@ -98,9 +110,7 @@ Create a new YAML file in `eval/cases/` following this pattern: ## A/B Comparison -`make eval-compare` builds both your current branch and main, then runs the -identical test suite against both. Results show a side-by-side comparison -matrix with per-test scores. +`make eval-compare` builds both your current branch and main, then runs the identical test suite against both. Results show a side-by-side comparison matrix with per-test scores. ## Environment Variables