diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 2da0f7a81..cdb890a7c 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -2,27 +2,51 @@ This directory contains C4 model diagrams (rendered with Mermaid) documenting the multi-agent collaboration framework for PicoClaw. +## Reference Implementation + +**OpenClaw (moltbot)** — the state-of-the-art personal AI gateway whose founder was hired by OpenAI. picoclaw ports and improves upon OpenClaw's validated patterns in a lightweight Go single-binary. + +- Reference code: `/home/leeaandrob/Projects/Personal/llm/auto-agents/moltbot` +- See [PRP](../prp/multi-agent-hardening.md) for detailed implementation plan + ## Documents | Document | Scope | Description | |----------|-------|-------------| | [C1 - System Context](./c1-system-context.md) | Highest level | PicoClaw in its ecosystem: users, channels, LLM providers | | [C2 - Container](./c2-container.md) | Runtime containers | Gateway, Agent Loop, Provider Layer, Channels | -| [C3 - Component](./c3-component-multi-agent.md) | Multi-agent internals | Blackboard, Handoff, Routing, Registry, Fallback | +| [C3 - Component](./c3-component-multi-agent.md) | Multi-agent internals | Current + planned components across 4 hardening phases | | [C4 - Code](./c4-code-detail.md) | Key structs/interfaces | Go interfaces, data flow, tool execution | | [Sequence Diagrams](./sequences.md) | Runtime flows | Handoff, Blackboard sync, Fallback chain | -| [Roadmap](./roadmap.md) | Phased plan | What's done, what's next, dependencies | +| [Roadmap](./roadmap.md) | Phased plan | 4-phase hardening based on OpenClaw gap analysis | -## Related Issues +## Related -- [#294 - Base Multi-agent Collaboration Framework & Shared Context](https://github.com/sipeed/picoclaw/issues/294) -- [#283 - Refactor Provider Architecture: By Protocol Instead of By Vendor](https://github.com/sipeed/picoclaw/issues/283) -- [Discussion #122 - Provider Architecture Proposal](https://github.com/sipeed/picoclaw/discussions/122) +- **PRP**: [Multi-Agent Hardening](../prp/multi-agent-hardening.md) — Full implementation plan with acceptance criteria +- **Issue**: [#294 - Base Multi-agent Collaboration Framework](https://github.com/sipeed/picoclaw/issues/294) +- **Issue**: [#283 - Refactor Provider Architecture](https://github.com/sipeed/picoclaw/issues/283) +- **Discussion**: [#122 - Provider Architecture Proposal](https://github.com/sipeed/picoclaw/discussions/122) ## Status | Phase | Status | PR | |-------|--------|----| | Provider Protocol Refactor | Merged | [#213](https://github.com/sipeed/picoclaw/pull/213) | -| Model Fallback + Multi-agent Routing | Merged | [#131](https://github.com/sipeed/picoclaw/pull/131) | -| Multi-agent Collaboration Framework | WIP | [#423](https://github.com/sipeed/picoclaw/pull/423) | +| Model Fallback + Routing | Merged | [#131](https://github.com/sipeed/picoclaw/pull/131) | +| Blackboard + Handoff + Discovery | WIP | [#423](https://github.com/sipeed/picoclaw/pull/423) | +| Phase 1: Foundation Fix | Planned | PR #423 | +| Phase 2: Tool Policy | Planned | PR #423 | +| Phase 3: Resilience | Planned | TBD | +| Phase 4: Async Multi-Agent | Planned | TBD | +| SOUL.md Bootstrap | In Progress (other dev) | TBD | + +## picoclaw Advantages Over OpenClaw + +| Area | picoclaw | OpenClaw | +|------|----------|----------| +| Shared agent state | Blackboard (real-time) | None (announce-only) | +| Runtime | Go single binary | Node.js | +| Memory footprint | ~10x smaller | Node.js overhead | +| Deployment | Copy binary and run | npm install + config | +| Concurrency | goroutines (native) | async/await | +| Type safety | Compile-time | Runtime | diff --git a/docs/architecture/c3-component-multi-agent.md b/docs/architecture/c3-component-multi-agent.md index ca0155ce7..a7d0ba898 100644 --- a/docs/architecture/c3-component-multi-agent.md +++ b/docs/architecture/c3-component-multi-agent.md @@ -1,8 +1,9 @@ # C3 - Component Diagram: Multi-Agent Framework Detailed view of the multi-agent collaboration components. +Includes both current (PR #423) and planned (Phases 1-4) components. -## Core Multi-Agent Components +## Core Multi-Agent Components (Current) ```mermaid C4Component @@ -62,6 +63,84 @@ C4Component Rel(fallback, error_cls, "ClassifyError()") ``` +## Planned Components (Phases 1-4) + +```mermaid +C4Component + title Planned Components - Hardening Phases + + Container_Boundary(tools_pkg, "pkg/tools (Phase 1-3)") { + Component(hooks, "ToolHook", "hooks.go", "BeforeExecute/AfterExecute interface for tool call interception") + Component(groups, "ToolGroups", "groups.go", "Named tool groups: fs, web, exec, sessions, memory") + Component(policy, "PolicyPipeline", "policy.go", "Layered allow/deny: global -> per-agent -> per-depth") + Component(loop_det, "LoopDetector", "loop_detector.go", "Generic repeat + ping-pong detection with configurable thresholds") + } + + Container_Boundary(multiagent_new, "pkg/multiagent (Phase 3-4)") { + Component(cascade, "CascadeStop", "cascade.go", "RunRegistry + recursive context cancellation") + Component(spawn, "AsyncSpawn", "spawn.go", "Non-blocking agent invocation via goroutines") + Component(announce, "AnnounceProtocol", "announce.go", "Result delivery: steer/queue/direct modes") + } + + Container_Boundary(providers_new, "pkg/providers (Phase 3)") { + Component(auth_rot, "AuthRotator", "auth_rotation.go", "Round-robin profiles + 2-track cooldown (transient + billing)") + } + + Container_Boundary(gateway_new, "pkg/gateway (Phase 4)") { + Component(dedup, "DedupCache", "dedup.go", "Idempotency layer with TTL-based deduplication") + } + + Rel(hooks, loop_det, "AfterExecute feeds detection") + Rel(hooks, policy, "BeforeExecute applies policy") + Rel(policy, groups, "Resolves group references") + Rel(cascade, spawn, "Tracks child runs") + Rel(spawn, announce, "Delivers results") + Rel(auth_rot, fallback, "Enhances with profile rotation") +``` + +## Known Issues (Pre-Phase 1) + +```mermaid +graph TD + BUG1[Blackboard Split-Brain]:::critical + BUG2[No Recursion Guard]:::critical + BUG3[Handoff Ignores Allowlist]:::high + BUG4[SubagentsConfig.Model Unused]:::low + + BUG1 --> FIX1[Phase 1a: Unify board per session] + BUG2 --> FIX2[Phase 1b: Depth + cycle detection] + BUG3 --> FIX3[Phase 1c: Check CanSpawnSubagent] + BUG4 --> FIX4[Defer to Phase 4] + + classDef critical fill:#ef4444,color:#fff + classDef high fill:#f59e0b,color:#000 + classDef low fill:#6b7280,color:#fff +``` + +### Blackboard Split-Brain Detail + +```mermaid +sequenceDiagram + participant RS as registerSharedTools + participant BT as BlackboardTool + participant RL as runAgentLoop + participant SP as System Prompt + + Note over RS: At startup + RS->>RS: sharedBoard := NewBlackboard() + RS->>BT: NewBlackboardTool(sharedBoard, agentID) + + Note over RL: At runtime (per message) + RL->>RL: sessionBoard := getOrCreateBlackboard(sessionKey) + RL->>SP: sessionBoard.Snapshot() → inject into system prompt + + Note over BT,SP: BUG: sharedBoard ≠ sessionBoard + BT->>RS: Write to sharedBoard ← WRONG BOARD + SP->>RL: Read from sessionBoard ← DIFFERENT OBJECT + + Note over BT,SP: FIX: SetBoard(sessionBoard) before execution +``` + ## Blackboard Data Model ```mermaid @@ -93,13 +172,16 @@ classDiagram -agentID string +Name() string +Execute(args) string + +SetBoard(board) void } class HandoffRequest { - +TargetAgentID string + +FromAgentID string + +ToAgentID string +Task string +Context map[string]string - +SessionKey string + +Depth int + +Visited []string } class HandoffResult { @@ -107,6 +189,7 @@ classDiagram +Response string +Success bool +Error string + +Iterations int } class AgentResolver { @@ -115,6 +198,11 @@ classDiagram +ListAgents() []AgentInfo } + class AllowlistChecker { + <> + +CanHandoff(from, to) bool + } + class AgentInfo { +ID string +Name string @@ -125,62 +213,88 @@ classDiagram Blackboard "1" --> "*" BlackboardEntry : stores BlackboardTool --> Blackboard : operates on HandoffRequest ..> AgentResolver : resolved via + HandoffRequest ..> AllowlistChecker : verified by AgentResolver --> AgentInfo : returns ``` -## Agent Registry & Instance Model +## Tool Policy Pipeline (Phase 2) ```mermaid -classDiagram - class AgentRegistry { - -agents map[string]*AgentInstance - -defaultID string - +Register(instance) - +GetInstance(id) *AgentInstance - +GetDefault() *AgentInstance - +ListAgentIDs() []string - } +graph LR + subgraph "Input" + ALL[All Registered Tools] + end - class AgentInstance { - +ID string - +Name string - +Role string - +SystemPrompt string - +Workspace string - +Model string - +Skills []string - +Tools []Tool - +AllowedSubagents []string - } + subgraph "Pipeline Steps" + S1[Global Allow/Deny] + S2[Per-Agent Allow/Deny] + S3[Per-Depth Deny] + S4[Sandbox Override] + end - class AgentConfig { - +ID string - +Default bool - +Name string - +Role string - +SystemPrompt string - +Workspace string - +Model *AgentModelConfig - +Skills []string - +Subagents *SubagentsConfig - } + subgraph "Output" + FINAL[Filtered Tools for Agent] + end - class RouteResolver { - -bindings []AgentBinding - +ResolveAgent(channel, chatID, peerKind, peerID) string - } + ALL --> S1 --> S2 --> S3 --> S4 --> FINAL - class FallbackChain { - -primary ModelRef - -fallbacks []ModelRef - -cooldown *CooldownTracker - +Chat(ctx, messages, tools, model, opts) *LLMResponse - } + style S1 fill:#3b82f6,color:#fff + style S2 fill:#f59e0b,color:#000 + style S3 fill:#ef4444,color:#fff + style S4 fill:#8b5cf6,color:#fff +``` - AgentRegistry "1" --> "*" AgentInstance : manages - AgentConfig ..> AgentInstance : creates - AgentInstance --> FallbackChain : uses for LLM calls - RouteResolver --> AgentRegistry : resolves agent from +```mermaid +graph TD + subgraph "Tool Groups" + GFS["group:fs
read_file, write_file, edit_file, append_file, list_dir"] + GWEB["group:web
web_search, web_fetch"] + GEXEC["group:exec
exec"] + GSESS["group:sessions
blackboard, handoff, list_agents, spawn"] + GMEM["group:memory
memory_search, memory_get"] + end + + subgraph "Depth Deny Rules" + D0["Depth 0 (main)
Full access"] + D1["Depth 1+ (subagent)
Deny: gateway"] + DL["Depth = max (leaf)
Deny: spawn, handoff, list_agents"] + end +``` + +## Async Multi-Agent Flow (Phase 4) + +```mermaid +sequenceDiagram + participant U as User + participant MA as Main Agent + participant SP as AsyncSpawn + participant SA1 as Subagent: Researcher + participant SA2 as Subagent: Analyst + participant AN as AnnounceProtocol + participant BB as Blackboard + + U->>MA: "Research and analyze market trends" + MA->>SP: AsyncSpawn(researcher, "find market data") + SP-->>MA: RunID: abc-123 + MA->>SP: AsyncSpawn(analyst, "prepare analysis framework") + SP-->>MA: RunID: def-456 + MA-->>U: "Working on it — spawned 2 agents..." + + par Parallel execution + SA1->>BB: write("market_data", findings) + SA1-->>AN: Complete: "Found 5 key trends" + and + SA2->>BB: write("framework", analysis_template) + SA2-->>AN: Complete: "Framework ready" + end + + AN->>MA: Announce(researcher result) [steer] + AN->>MA: Announce(analyst result) [queue] + + MA->>BB: read("market_data") + MA->>BB: read("framework") + MA->>MA: Synthesize results + MA-->>U: "Here's the market analysis..." ``` ## Provider Protocol Architecture (PR #213 + #283) @@ -210,55 +324,23 @@ graph TB CC[ClaudeCliProvider] CX[CodexCliProvider] end - subgraph "OAuth/Token" - CA[CodexProvider - OAuth] - CL[ClaudeProvider - OAuth] - end subgraph "Resilience" FB[FallbackChain] CD[CooldownTracker] EC[ErrorClassifier] + AR[AuthRotator - Phase 3] end end - subgraph "External LLMs" - OPENAI[OpenAI] - GROQ[Groq] - DEEP[DeepSeek] - OR[OpenRouter] - ANTH[Anthropic] - GEM[Gemini] - OLL[Ollama] - CLICLI[claude CLI] - CODCLI[codex CLI] - end - CFG --> RS ML -.-> RS RS --> CP - CP --> HTTP CP --> ANT CP --> CC CP --> CX - CP --> CA - CP --> CL - HTTP --> OC - OC --> OPENAI - OC --> GROQ - OC --> DEEP - OC --> OR - OC --> OLL - ANT --> ANTH - CP2 --> ANTH - OC --> GEM - CC --> CLICLI - CX --> CODCLI - CA --> OPENAI - FB --> CD FB --> EC - FB -.-> HTTP - FB -.-> ANT + AR -.-> FB ``` diff --git a/docs/architecture/roadmap.md b/docs/architecture/roadmap.md index 3beb74685..921253f06 100644 --- a/docs/architecture/roadmap.md +++ b/docs/architecture/roadmap.md @@ -1,6 +1,7 @@ # Multi-Agent Feature Roadmap Phased implementation plan based on issues #294, #283, and discussion #122. +Updated with OpenClaw (moltbot) gap analysis — patterns validated by OpenAI (founder hired). ## Phase Overview @@ -12,79 +13,95 @@ gantt section Provider Refactor (#283) Phase 1: Protocol packages (PR #213) :done, p1, 2026-02-01, 2026-02-18 - Phase 2: model_list + explicit api_type :active, p2, 2026-02-19, 2026-03-05 + Phase 2: model_list + explicit api_type :p2, 2026-03-15, 2026-04-01 Phase 3: Independent Gemini protocol :p3, after p2, 7d Phase 4: Local LLM + cleanup :p4, after p3, 5d section Multi-Agent (#294) Fallback chain + routing (PR #131) :done, ma1, 2026-02-01, 2026-02-18 - Blackboard + Handoff + Discovery (PR #423) :active, ma2, 2026-02-18, 2026-03-01 - Swarm mode (async agent negotiation) :ma3, after ma2, 14d - Visual AIEOS dashboard :ma4, after ma3, 14d + Blackboard + Handoff + Discovery (PR #423) :done, ma2, 2026-02-18, 2026-03-01 + Phase 1: Foundation Fix + Guardrails :active, h1, 2026-02-19, 2026-02-28 + Phase 2: Tool Policy Pipeline :h2, after h1, 10d + Phase 3: Resilience :h3, after h2, 10d + Phase 4: Async Multi-Agent :h4, after h3, 14d + + section Workspace + SOUL.md Bootstrap (separate PR) :active, soul, 2026-02-19, 2026-03-01 section Integration model_list + multi-agent config merge :int1, after p2, 7d - Community agent marketplace :int2, after ma3, 21d + Community agent marketplace :int2, after h4, 21d ``` -## Detailed Status +## Hardening Phases (Based on OpenClaw Gap Analysis) -### Done +### Phase 1: Foundation Fix + Guardrails + +| Task | Description | OpenClaw Reference | +|------|-------------|-------------------| +| Fix blackboard split-brain | Unify static board and session board | N/A (picoclaw-specific bug) | +| Recursion guard | Depth counter + cycle detection in handoff | `subagent-depth.ts` | +| Handoff allowlist | Enforce CanSpawnSubagent in ExecuteHandoff | `subagent-spawn.ts` (allowlist check) | +| Before-tool-call hooks | ToolHook interface for extensibility | `pi-tools.before-tool-call.ts` | + +### Phase 2: Tool Policy Pipeline + +| Task | Description | OpenClaw Reference | +|------|-------------|-------------------| +| Tool groups | Named groups: fs, web, exec, sessions, memory | `tool-policy.ts` (TOOL_GROUPS) | +| Per-agent allow/deny | Config-driven tool filtering | `tool-policy-pipeline.ts` | +| Subagent deny-by-depth | Leaf agents can't spawn/handoff | `pi-tools.policy.ts` | +| Pipeline composition | Layered: global → agent → depth | `tool-policy-pipeline.ts` | + +### Phase 3: Resilience + +| Task | Description | OpenClaw Reference | +|------|-------------|-------------------| +| Loop detection | Generic repeat + ping-pong detectors | `tool-loop-detection.ts` | +| Context overflow recovery | Compaction → truncation → user error | `pi-embedded-runner/run.ts` | +| Auth profile rotation | Round-robin + 2-track cooldown | `auth-profiles/order.ts` + `usage.ts` | +| Cascade stop | Context cancellation propagation | `subagents-tool.ts` | + +### Phase 4: Async Multi-Agent + +| Task | Description | OpenClaw Reference | +|------|-------------|-------------------| +| Async spawn | Non-blocking via goroutines | `subagent-spawn.ts` | +| Announce protocol | Result injection: steer/queue/direct | `subagent-announce.ts` | +| Process isolation | Scope-keyed exec tool | Process supervisor (scope-keyed) | +| Idempotency | Dedup cache for gateway RPC | `server-methods/agent.ts` | + +## Completed | Phase | PR | Description | |-------|----|-------------| -| Provider Protocol Refactor | [#213](https://github.com/sipeed/picoclaw/pull/213) | `protocoltypes/`, `openai_compat/`, `anthropic/` packages, thin delegates, factory refactor | -| Fallback Chain + Routing | [#131](https://github.com/sipeed/picoclaw/pull/131) | `FallbackChain`, `CooldownTracker`, `ErrorClassifier`, `AgentRegistry`, `RouteResolver`, `AgentInstance`, channel peer metadata | - -### In Progress (PR #423) - -| Component | Package | Files | Status | -|-----------|---------|-------|--------| -| Agent Config extensions | `pkg/config` | `config.go` | `Role`, `SystemPrompt` fields added | -| Blackboard shared context | `pkg/multiagent` | `blackboard.go`, `blackboard_tool.go` | Complete, 18 tests | -| Agent Handoff | `pkg/multiagent` | `handoff.go`, `handoff_tool.go` | Complete, 10 tests | -| Agent Discovery | `pkg/multiagent` | `list_agents_tool.go` | Complete | -| AgentLoop integration | `pkg/agent` | `loop.go` | Snapshot injection, tool registration, per-session blackboards | -| AgentResolver interface | `pkg/multiagent` | `handoff.go` | Decouples multiagent from agent pkg | - -### Next: Provider Phase 2 - `model_list` (#283) - -Based on @yinwm's design in [issue #283](https://github.com/sipeed/picoclaw/issues/283#issuecomment-3915867555): - -``` -model_list config -> model-centric resolution -> protocol prefix routing -``` - -| Task | Description | Impact | -|------|-------------|--------| -| `ModelConfig` struct | `model_name`, `model` (protocol/id), `api_base`, `api_key` | Eliminates per-vendor code changes | -| Protocol prefix routing | `openai/`, `anthropic/`, `antigravity/` prefixes | Clean protocol selection | -| Backward compatibility | Support both `providers` (deprecated) and `model_list` | Smooth migration | -| Agent model reference | Agents reference `model_name` instead of `provider` + `model` | Simplifies multi-agent config | - -### Future: Out of Scope for #294 - -| Feature | Issue | Depends on | -|---------|-------|------------| -| Intelligent Model Routing (small/large model token saving) | TBD | model_list + multi-agent | -| Swarm Mode (autonomous agent-to-agent negotiation) | TBD | Handoff foundation | -| Visual AIEOS Dashboard | TBD | All above | -| Community Agent Marketplace | TBD | Stable agent interface | +| Provider Protocol Refactor | [#213](https://github.com/sipeed/picoclaw/pull/213) | `protocoltypes/`, `openai_compat/`, `anthropic/` packages | +| Fallback Chain + Routing | [#131](https://github.com/sipeed/picoclaw/pull/131) | `FallbackChain`, `CooldownTracker`, `RouteResolver` | +| Blackboard + Handoff + Discovery | [#423](https://github.com/sipeed/picoclaw/pull/423) | Blackboard, HandoffTool, ListAgentsTool, AgentResolver | +| golangci-lint compliance | [#304](https://github.com/sipeed/picoclaw/pull/304) | 62 lint issues fixed in our code | +| C4 Architecture docs | [#423](https://github.com/sipeed/picoclaw/pull/423) | This directory | ## Dependency Graph ```mermaid graph TD PR213[PR #213: Protocol Refactor]:::done --> PR131[PR #131: Fallback + Routing]:::done - PR131 --> PR423[PR #423: Blackboard + Handoff]:::active - PR213 --> P2[Phase 2: model_list]:::planned - P2 --> P3[Phase 3: Gemini Protocol]:::planned - P3 --> P4[Phase 4: Local LLM]:::planned - PR423 --> SWARM[Swarm Mode]:::future - PR423 --> P2 - P2 --> MCONFIG[model_list + multi-agent config merge]:::planned - SWARM --> DASH[AIEOS Dashboard]:::future - MCONFIG --> MARKET[Agent Marketplace]:::future + PR131 --> PR423[PR #423: Blackboard + Handoff]:::done + + PR423 --> H1[Phase 1: Foundation Fix]:::active + H1 --> H2[Phase 2: Tool Policy]:::planned + H2 --> H3[Phase 3: Resilience]:::planned + H3 --> H4[Phase 4: Async Multi-Agent]:::planned + + PR213 --> P2[Provider Phase 2: model_list]:::future + P2 --> P3[Provider Phase 3: Gemini]:::future + P3 --> P4[Provider Phase 4: Local LLM]:::future + + SOUL[SOUL.md Bootstrap]:::active + + H4 --> SWARM[Swarm Mode]:::future + H4 --> DASH[AIEOS Dashboard]:::future + P2 --> MCONFIG[model_list + multi-agent merge]:::future classDef done fill:#22c55e,color:#fff classDef active fill:#eab308,color:#000 @@ -96,9 +113,24 @@ graph TD | Decision | Choice | Rationale | |----------|--------|-----------| -| Shared context pattern | Blackboard (key-value) | Simple, auditable, no coupling between agents | -| Handoff mechanism | Synchronous via RunToolLoop | Predictable, debuggable; async deferred to Swarm Mode | -| Circular import avoidance | `AgentResolver` interface in pkg/multiagent | Clean dependency direction: agent -> multiagent, not reverse | -| Multi-agent tool activation | Conditional on `len(agents) > 1` | Zero overhead for single-agent setups | -| Provider abstraction | Protocol-first (openai_compat, anthropic) | Adding new OpenAI-compat providers = config only | -| Config evolution | model_list (LiteLLM-inspired) | Model-centric aligns with multi-agent where agents pick models | +| Shared context pattern | Blackboard (key-value) | OpenClaw has no shared state (announce-only). Blackboard is more flexible. | +| Handoff mechanism | Synchronous → async in Phase 4 | Start simple, add async when foundation is solid | +| Tool policy model | Layered pipeline (like OpenClaw) | Composable, debuggable, per-layer narrowing | +| Loop detection | 2 detectors (repeat + ping-pong) | OpenClaw has 4 — start with 2 most impactful | +| Auth rotation | 2-track cooldown (transient + billing) | Directly ported from OpenClaw, proven in production | +| Recursion guard | Depth counter + visited set | Simple, O(1) check, prevents both depth and cycles | +| Reference implementation | OpenClaw (moltbot) | Founder hired by OpenAI — patterns are industry-validated | + +## picoclaw vs. OpenClaw Comparison + +| Feature | OpenClaw (Node.js) | picoclaw (Go) | Advantage | +|---------|-------------------|---------------|-----------| +| Shared agent state | None (announce-only) | Blackboard | picoclaw | +| Performance | Node.js runtime | Single Go binary | picoclaw (10x less memory) | +| Deployment | npm install + node | Copy binary | picoclaw | +| Tool policy | 8-layer pipeline | Planned (Phase 2) | OpenClaw (for now) | +| Loop detection | 4 detectors | Planned (Phase 3) | OpenClaw (for now) | +| Auth rotation | 2-track + file lock | FallbackChain only | OpenClaw (for now) | +| Async spawn | Lane-based + announce | Planned (Phase 4) | OpenClaw (for now) | +| Concurrency model | async/await | goroutines | picoclaw | +| Type safety | TypeScript | Go compiler | picoclaw | diff --git a/docs/architecture/sequences.md b/docs/architecture/sequences.md index f182fd2d1..380b9e51c 100644 --- a/docs/architecture/sequences.md +++ b/docs/architecture/sequences.md @@ -2,7 +2,7 @@ Runtime interaction flows for multi-agent collaboration. -## 1. Agent Handoff Flow +## 1. Agent Handoff Flow (Current) A main agent delegates a sub-task to a specialized agent. @@ -47,7 +47,7 @@ sequenceDiagram CH->>U: "The coder agent translated your code: ..." ``` -## 2. Blackboard Shared Context Flow +## 2. Blackboard Shared Context Flow (Current) Multiple agents share data through the blackboard within a session. @@ -85,7 +85,7 @@ sequenceDiagram Note over BB: Blackboard state:
findings (by researcher)
sources (by researcher)
draft (by writer) ``` -## 3. Model Fallback Chain Flow +## 3. Model Fallback Chain Flow (Current) Provider resilience with automatic failover. @@ -134,7 +134,7 @@ sequenceDiagram FB-->>AL: LLMResponse ``` -## 4. Route Resolution Flow +## 4. Route Resolution Flow (Current) How incoming messages are routed to the correct agent. @@ -163,7 +163,7 @@ sequenceDiagram Note over MSG: Session key used for:
- Session history
- Blackboard lookup
- State persistence ``` -## 5. Multi-Agent Configuration Lifecycle +## 5. Multi-Agent Configuration Lifecycle (Current) From config.json to running agents. @@ -202,3 +202,188 @@ sequenceDiagram Note over TOOLS: No multi-agent tools (zero overhead) end ``` + +## 6. Handoff with Guardrails (Phase 1 — Planned) + +Handoff with depth limit, cycle detection, and allowlist enforcement. + +```mermaid +sequenceDiagram + participant MA as Main Agent (depth=0) + participant HT as HandoffTool + participant GD as Guards + participant AL as AllowlistChecker + participant SA as Agent B (depth=1) + participant SC as Agent C (depth=2) + + MA->>HT: handoff(target="B", task="research") + HT->>GD: Check depth (0 < maxDepth=3) + GD-->>HT: OK + HT->>GD: Check cycle (visited=["main"]) + GD-->>HT: OK (B not in visited) + HT->>AL: CanHandoff("main", "B") + AL-->>HT: Allowed + + HT->>SA: ExecuteHandoff(depth=1, visited=["main","B"]) + SA->>HT: handoff(target="C", task="analyze") + HT->>GD: Check depth (1 < maxDepth=3) + GD-->>HT: OK + HT->>GD: Check cycle (visited=["main","B"]) + GD-->>HT: OK (C not in visited) + + HT->>SC: ExecuteHandoff(depth=2, visited=["main","B","C"]) + SC-->>SA: Result + SA-->>MA: Combined result + + Note over MA: Cycle detection example + MA->>HT: handoff(target="B", task="...") + SA->>HT: handoff(target="main", task="...") + HT->>GD: Check cycle (visited=["main","B"]) + GD-->>HT: BLOCKED: "main" already in visited + HT-->>SA: Error: handoff cycle detected +``` + +## 7. Tool Policy Pipeline (Phase 2 — Planned) + +How tools are filtered before agent execution. + +```mermaid +sequenceDiagram + participant CFG as Config + participant PP as PolicyPipeline + participant GR as ToolGroups + participant AG as Agent Tools + participant DEPTH as DepthPolicy + + Note over PP: Agent "researcher" at depth=1 + + CFG->>PP: Global deny: ["gateway"] + PP->>GR: Resolve "gateway" → ["gateway"] + PP->>PP: Remove "gateway" from tool set + + CFG->>PP: Agent allow: ["group:web", "group:fs", "blackboard"] + PP->>GR: Resolve groups → [web_search, web_fetch, read_file, ...] + PP->>PP: Keep only allowed tools + + DEPTH->>PP: Depth=1 deny: ["spawn"] + PP->>PP: Remove "spawn" from tool set + + PP->>AG: Final tools: [web_search, web_fetch, read_file, write_file, blackboard, handoff] + Note over AG: Each layer narrows, never widens +``` + +## 8. Loop Detection (Phase 3 — Planned) + +How tool call loops are detected and blocked. + +```mermaid +sequenceDiagram + participant LLM as LLM Provider + participant HK as ToolHook + participant LD as LoopDetector + participant T as Tool + + loop Normal execution (calls 1-9) + LLM->>HK: BeforeExecute("web_search", {query: "same query"}) + HK->>LD: Check(hash("web_search:same query")) + LD-->>HK: OK (repeat count < 10) + HK->>T: Execute + T-->>HK: Result + HK->>LD: Record(hash, outcome) + end + + LLM->>HK: BeforeExecute("web_search", {query: "same query"}) + HK->>LD: Check (repeat count = 10) + LD-->>HK: WARNING: possible loop + Note over HK: Warning injected into tool result + + loop Calls 11-19 (with warning) + LLM->>HK: BeforeExecute("web_search", {query: "same query"}) + HK->>LD: Check + LD-->>HK: WARNING + end + + LLM->>HK: BeforeExecute("web_search", {query: "same query"}) + HK->>LD: Check (repeat count = 20) + LD-->>HK: BLOCKED: tool call repeated too many times + HK-->>LLM: Error: loop detected, try a different approach +``` + +## 9. Async Spawn + Announce (Phase 4 — Planned) + +Parallel agent execution with result delivery. + +```mermaid +sequenceDiagram + participant U as User + participant MA as Main Agent + participant SP as AsyncSpawn + participant RR as RunRegistry + participant SA1 as Researcher + participant SA2 as Analyst + participant AN as AnnounceProtocol + participant BB as Blackboard + + U->>MA: "Research market trends and analyze competitors" + + MA->>SP: AsyncSpawn(researcher, "find market data") + SP->>RR: Register(runID=abc, parent=main) + SP-->>MA: RunID: abc (non-blocking) + + MA->>SP: AsyncSpawn(analyst, "competitor analysis") + SP->>RR: Register(runID=def, parent=main) + SP-->>MA: RunID: def (non-blocking) + + MA-->>U: "Working on it — 2 agents spawned..." + + par Parallel Execution + SA1->>BB: write("market_data", findings) + SA1->>RR: Complete(abc) + SA1->>AN: Announce(to=main, content=findings) + and + SA2->>BB: write("competitors", analysis) + SA2->>RR: Complete(def) + SA2->>AN: Announce(to=main, content=analysis) + end + + AN->>MA: Steer: "Researcher completed: found 5 trends" + AN->>MA: Queue: "Analyst completed: 3 competitors analyzed" + + MA->>BB: read("market_data") + read("competitors") + MA->>MA: Synthesize + MA-->>U: "Here's the complete market analysis..." +``` + +## 10. Cascade Stop (Phase 3 — Planned) + +Stopping a parent agent cascades to all children. + +```mermaid +sequenceDiagram + participant U as User + participant RR as RunRegistry + participant MA as Main Agent + participant SA1 as Subagent 1 + participant SA2 as Subagent 2 + participant SA3 as Sub-subagent (child of SA1) + + Note over MA: Active run tree:
main → SA1 → SA3
main → SA2 + + U->>RR: CascadeStop("main") + + RR->>MA: Cancel context + MA->>MA: Stop processing + + RR->>RR: Find children of "main" + RR->>SA1: Cancel context + SA1->>SA1: Stop processing + + RR->>RR: Find children of SA1 + RR->>SA3: Cancel context + SA3->>SA3: Stop processing + + RR->>SA2: Cancel context + SA2->>SA2: Stop processing + + RR-->>U: Killed 4 runs (main + SA1 + SA2 + SA3) +``` diff --git a/docs/prp/multi-agent-hardening.md b/docs/prp/multi-agent-hardening.md new file mode 100644 index 000000000..8435438e4 --- /dev/null +++ b/docs/prp/multi-agent-hardening.md @@ -0,0 +1,770 @@ +# PRP: Multi-Agent Framework Hardening + +## Goal + +Harden the picoclaw multi-agent collaboration framework (PR #423) to production-grade quality by porting validated patterns from OpenClaw (moltbot) — the state-of-the-art personal AI gateway whose founder was hired by OpenAI. + +**PR**: [#423](https://github.com/sipeed/picoclaw/pull/423) +**Issue**: [#294](https://github.com/sipeed/picoclaw/issues/294) +**Branch**: `feat/multi-agent-framework` +**Reference**: `/home/leeaandrob/Projects/Personal/llm/auto-agents/moltbot` + +--- + +## What + +Transform picoclaw's multi-agent framework from a functional prototype (Blackboard + Handoff + Routing) into a production-ready orchestration system with guardrails, tool policy, resilience, and async capabilities — matching and exceeding OpenClaw's patterns in a lightweight Go single-binary. + +--- + +## Success Criteria + +### Phase 1: Foundation Fix + Guardrails +- [ ] Blackboard split-brain bug is fixed — tools and system prompt use the same board per session +- [ ] Handoff recursion is bounded — max depth enforced, cycle detection prevents A→B→A loops +- [ ] Handoff respects allowlist — same CanSpawnSubagent check as spawn tool +- [ ] Before-tool-call hook infrastructure exists — extensible for loop detection and policy +- [ ] All existing tests pass + new tests for each fix +- [ ] Zero regression in single-agent mode + +### Phase 2: Tool Policy Pipeline +- [ ] Tool groups defined: `group:fs`, `group:web`, `group:exec`, `group:sessions`, `group:memory` +- [ ] Per-agent `tools.allow` / `tools.deny` in config, supports group references +- [ ] Subagent tool restriction by depth (leaf agents can't spawn/handoff) +- [ ] Pipeline composes: global → per-agent → per-depth (each layer narrows, never widens) +- [ ] Config backward-compatible (no tools config = full access, like today) + +### Phase 3: Resilience +- [ ] Loop detection: generic repeat (hash-based) + ping-pong detector +- [ ] Context overflow recovery: auto-compaction + tool result truncation + user error +- [ ] Auth profile rotation: round-robin with 2-track cooldown (transient + billing) +- [ ] Cascade stop: context cancellation propagates through handoff/spawn chains + +### Phase 4: Async Multi-Agent +- [ ] Async spawn: non-blocking agent invocation via goroutines +- [ ] Announce protocol: result injection into parent session (steer/queue/direct) +- [ ] Scope-keyed process isolation: exec tool scoped by session key +- [ ] Idempotency: dedup map for duplicate message prevention + +--- + +## Current State Analysis + +### What exists (PR #423) + +| Component | Package | Status | Quality | +|-----------|---------|--------|---------| +| Blackboard | `pkg/multiagent` | Implemented | Good base, has split-brain bug | +| BlackboardTool | `pkg/multiagent` | Implemented | Works but uses wrong board | +| ExecuteHandoff | `pkg/multiagent` | Implemented | No recursion guard, ignores allowlist | +| HandoffTool | `pkg/multiagent` | Implemented | No self-handoff guard | +| ListAgentsTool | `pkg/multiagent` | Implemented | Exposes too little info | +| AgentResolver | `pkg/multiagent` | Implemented | Clean interface | +| RouteResolver | `pkg/routing` | Implemented | 7-tier cascade, complete | +| SessionKeyBuilder | `pkg/routing` | Implemented | DM scope support | +| AgentRegistry | `pkg/agent` | Implemented | CanSpawnSubagent exists | +| AgentLoop integration | `pkg/agent` | Implemented | Snapshot injection, conditional tools | + +### Critical Bug: Blackboard Split-Brain + +``` +registerSharedTools() → creates static `sharedBoard` per agent +runAgentLoop() → creates per-session board via getOrCreateBlackboard(sessionKey) + +BlackboardTool.Execute() → writes to static sharedBoard ← WRONG +messages[0].Content → reads from session board ← DIFFERENT OBJECT +``` + +**Impact**: Agents think they're sharing context but they're writing to void. Any multi-agent demo fails silently. + +### What's missing vs. OpenClaw + +| Feature | OpenClaw | picoclaw | Gap | +|---------|----------|----------|-----| +| Tool policy | 8-layer pipeline | None | Critical | +| Recursion guard | maxSpawnDepth + maxChildren | None | Critical | +| Loop detection | 4 detectors + circuit breaker | None | High | +| Context overflow | 3-tier cascade recovery | Basic forceCompression | High | +| Auth rotation | Round-robin + 2-track cooldown | FallbackChain only | Medium | +| Cascade stop | cascadeKillChildren + abort | None | Medium | +| Before-tool hook | wrapToolWithBeforeToolCallHook | None | Medium | +| Async spawn | Lane-based + announce | None | Low (Phase 4) | +| Process isolation | Scope-keyed | None | Low (Phase 4) | +| Idempotency | Gateway dedup + announce dedup | None | Low (Phase 4) | + +--- + +## Phase 1: Foundation Fix + Guardrails + +### 1a. Fix Blackboard Split-Brain + +**Problem**: `registerSharedTools` in `loop.go:178` creates one `sharedBoard` per agent at startup. `runAgentLoop` in `loop.go` creates separate per-session blackboards in `AgentLoop.blackboards` sync.Map. Tools write to one, system prompt reads from other. + +**Solution**: Make BlackboardTool and HandoffTool session-aware. + +**Approach A (Recommended)**: ContextualTool pattern +```go +// BlackboardTool already has agentID — add board setter +type BlackboardTool struct { + board *Blackboard + agentID string + mu sync.RWMutex +} + +func (t *BlackboardTool) SetBoard(board *Blackboard) { + t.mu.Lock() + defer t.mu.Unlock() + t.board = board +} +``` + +In `runAgentLoop`, before tool execution: +```go +bb := al.getOrCreateBlackboard(opts.SessionKey) +// Update all session-aware tools with the current session's board +for _, tool := range agent.Tools.List() { + if setter, ok := tool.(BoardSetter); ok { + setter.SetBoard(bb) + } +} +``` + +**Approach B**: Lazy board resolution via callback +```go +type BlackboardTool struct { + resolveBoard func() *Blackboard + agentID string +} +``` + +**Files to modify**: +- `pkg/multiagent/blackboard_tool.go` — Add `SetBoard` or callback +- `pkg/multiagent/handoff_tool.go` — Same pattern +- `pkg/agent/loop.go` — Wire session board to tools before execution + +**Tests**: +- Verify tool writes are visible in system prompt snapshot +- Verify cross-agent writes via handoff are visible +- Verify session isolation (board A != board B) + +### 1b. Recursion Guard + +**Problem**: Agent A can hand off to B, which hands off to A, causing infinite recursion. + +**Solution**: Add depth tracking and cycle detection to ExecuteHandoff. + +```go +// Add to HandoffRequest +type HandoffRequest struct { + FromAgentID string + ToAgentID string + Task string + Context map[string]string + Depth int // NEW: current nesting depth + Visited []string // NEW: agent IDs in the chain +} + +// In ExecuteHandoff +const DefaultMaxHandoffDepth = 3 + +func ExecuteHandoff(ctx context.Context, resolver AgentResolver, board *Blackboard, + req *HandoffRequest, channel, chatID string, maxDepth int) *HandoffResult { + + // Depth check + if req.Depth >= maxDepth { + return &HandoffResult{ + Error: fmt.Sprintf("handoff depth limit reached (%d/%d)", req.Depth, maxDepth), + Success: false, + } + } + + // Cycle detection + for _, visited := range req.Visited { + if visited == req.ToAgentID { + return &HandoffResult{ + Error: fmt.Sprintf("handoff cycle detected: %s already in chain %v", + req.ToAgentID, req.Visited), + Success: false, + } + } + } + + // Propagate depth + visited to nested handoffs + // ... (inject into target agent's tool context) +} +``` + +**Config**: +```json +{ + "agents": { + "defaults": { + "max_handoff_depth": 3 + } + } +} +``` + +**Files to modify**: +- `pkg/multiagent/handoff.go` — Depth + cycle detection +- `pkg/multiagent/handoff_tool.go` — Propagate depth/visited +- `pkg/config/config.go` — Add `MaxHandoffDepth` to `AgentDefaults` + +**Tests**: +- Direct handoff succeeds (depth 0→1) +- Chain handoff A→B→C succeeds (depth 0→1→2) +- Depth limit exceeded returns error +- Cycle A→B→A returns error +- Self-handoff returns error + +### 1c. Handoff Allowlist Enforcement + +**Problem**: `CanSpawnSubagent` only checked by spawn tool, not by handoff. + +**Solution**: Add allowlist check to `ExecuteHandoff`. + +```go +func ExecuteHandoff(...) *HandoffResult { + // ... depth/cycle checks above ... + + // Allowlist check (new) + if checker, ok := resolver.(AllowlistChecker); ok { + if !checker.CanHandoff(req.FromAgentID, req.ToAgentID) { + return &HandoffResult{ + Error: fmt.Sprintf("agent %q is not allowed to handoff to %q", + req.FromAgentID, req.ToAgentID), + Success: false, + } + } + } + + // ... rest of execution +} +``` + +**Files to modify**: +- `pkg/multiagent/handoff.go` — Add `AllowlistChecker` interface + check +- `pkg/agent/loop.go` — Make `registryResolver` implement `AllowlistChecker` + +**Tests**: +- Allowed handoff succeeds +- Disallowed handoff returns error +- Wildcard `"*"` allows all + +### 1d. Before-Tool-Call Hook Infrastructure + +**Problem**: No extensibility point for tool execution. Needed for loop detection (Phase 3) and tool policy (Phase 2). + +**Solution**: ToolHook interface wrapping tool execution. + +```go +// pkg/tools/hooks.go +type ToolHook interface { + BeforeExecute(ctx context.Context, toolName string, args map[string]any) (map[string]any, error) + AfterExecute(ctx context.Context, toolName string, args map[string]any, result *ToolResult, err error) +} + +// pkg/tools/registry.go — Add hook chain +func (r *ToolRegistry) SetHooks(hooks []ToolHook) { ... } + +func (r *ToolRegistry) ExecuteWithHooks(ctx context.Context, toolName string, args map[string]any) (*ToolResult, error) { + // Run BeforeExecute hooks in order + currentArgs := args + for _, hook := range r.hooks { + var err error + currentArgs, err = hook.BeforeExecute(ctx, toolName, currentArgs) + if err != nil { + return &ToolResult{Content: err.Error(), IsError: true}, nil + } + } + + // Execute tool + result := tool.Execute(ctx, currentArgs) + + // Run AfterExecute hooks (fire-and-forget) + for _, hook := range r.hooks { + hook.AfterExecute(ctx, toolName, currentArgs, result, nil) + } + + return result, nil +} +``` + +**Files to create**: +- `pkg/tools/hooks.go` — ToolHook interface + chain execution + +**Files to modify**: +- `pkg/tools/registry.go` — Add hooks field and ExecuteWithHooks +- `pkg/agent/loop.go` — Use ExecuteWithHooks in runLLMIteration + +**Tests**: +- Hook blocks tool → error returned +- Hook modifies args → tool receives modified args +- AfterExecute called with result +- Multiple hooks execute in order + +--- + +## Phase 2: Tool Policy Pipeline + +### 2a. Tool Groups + +```go +// pkg/tools/groups.go +var DefaultToolGroups = map[string][]string{ + "group:fs": {"read_file", "write_file", "edit_file", "append_file", "list_dir"}, + "group:web": {"web_search", "web_fetch"}, + "group:exec": {"exec"}, + "group:sessions": {"blackboard", "handoff", "list_agents", "spawn"}, + "group:memory": {"memory_search", "memory_get"}, + "group:message": {"message"}, + "group:image": {"image_generation"}, +} + +func ResolveToolNames(refs []string, groups map[string][]string) []string { + var resolved []string + for _, ref := range refs { + if tools, ok := groups[ref]; ok { + resolved = append(resolved, tools...) + } else { + resolved = append(resolved, ref) + } + } + return resolved +} +``` + +### 2b. Per-Agent Allow/Deny Config + +```go +// pkg/config/config.go — Add to AgentConfig +type ToolPolicyConfig struct { + Allow []string `json:"allow,omitempty"` // tool names or group refs + Deny []string `json:"deny,omitempty"` // tool names or group refs +} + +type AgentConfig struct { + // ... existing fields ... + Tools *ToolPolicyConfig `json:"tools,omitempty"` // NEW +} +``` + +### 2c. Subagent Deny-by-Depth + +```go +// pkg/tools/policy.go +type DepthPolicy struct { + MaxDepth int +} + +func (p *DepthPolicy) DenyListForDepth(depth int) []string { + if depth == 0 { + return nil // main agent: full access + } + + // All subagents lose dangerous tools + deny := []string{"gateway"} + + if depth >= p.MaxDepth { + // Leaf workers: no spawning + deny = append(deny, "spawn", "handoff", "list_agents") + } + return deny +} +``` + +### 2d. Pipeline Composition + +```go +// pkg/tools/policy.go +type PolicyStep struct { + Allow []string + Deny []string + Label string +} + +func ApplyPolicyPipeline(tools []Tool, steps []PolicyStep) []Tool { + remaining := tools + for _, step := range steps { + if len(step.Allow) > 0 { + allowed := toSet(ResolveToolNames(step.Allow, DefaultToolGroups)) + remaining = filter(remaining, func(t Tool) bool { + return allowed[t.Name()] + }) + } + if len(step.Deny) > 0 { + denied := toSet(ResolveToolNames(step.Deny, DefaultToolGroups)) + remaining = filter(remaining, func(t Tool) bool { + return !denied[t.Name()] + }) + } + } + return remaining +} +``` + +**Files to create**: +- `pkg/tools/groups.go` — Tool group definitions +- `pkg/tools/policy.go` — PolicyStep, ApplyPolicyPipeline, DepthPolicy + +**Files to modify**: +- `pkg/config/config.go` — ToolPolicyConfig in AgentConfig +- `pkg/agent/loop.go` — Apply policy pipeline in registerSharedTools +- `pkg/multiagent/handoff.go` — Apply depth policy for handoff target + +--- + +## Phase 3: Resilience + +### 3a. Loop Detection + +```go +// pkg/tools/loop_detector.go +type LoopDetector struct { + history []toolCallRecord + maxHistory int // default 30 + warnAt int // default 10 + blockAt int // default 20 +} + +type toolCallRecord struct { + Hash string + ToolName string + Timestamp time.Time + Outcome string // hash of result for no-progress detection +} + +func (d *LoopDetector) Check(toolName string, args map[string]any) LoopVerdict { + hash := hashToolCall(toolName, args) + + // Generic repeat detection + repeatCount := d.countRepeats(hash) + if repeatCount >= d.blockAt { + return LoopVerdict{Blocked: true, Reason: "tool call repeated too many times"} + } + if repeatCount >= d.warnAt { + return LoopVerdict{Warning: true, Reason: "possible loop detected"} + } + + // Ping-pong detection (A,B,A,B pattern) + if d.detectPingPong(hash) { + return LoopVerdict{Blocked: true, Reason: "ping-pong loop detected"} + } + + return LoopVerdict{} +} + +func (d *LoopDetector) Record(toolName string, args map[string]any, result string) { + // Append to history ring buffer, trim to maxHistory +} +``` + +**Files to create**: `pkg/tools/loop_detector.go`, `pkg/tools/loop_detector_test.go` + +### 3b. Context Overflow Recovery + +Enhance existing `forceCompression` in `loop.go`: + +```go +func (al *AgentLoop) recoverContextOverflow(ctx context.Context, agent *AgentInstance, + sessionKey string, err error) RecoveryResult { + + // Tier 1: LLM-based compaction (summarize history) + for attempt := 0; attempt < 3; attempt++ { + if al.compactSession(ctx, agent, sessionKey) { + return RecoveryResult{Recovered: true, Method: "compaction"} + } + } + + // Tier 2: Truncate oversized tool results in history + if al.truncateToolResults(ctx, agent, sessionKey) { + return RecoveryResult{Recovered: true, Method: "tool_truncation"} + } + + // Tier 3: Give up with user-facing error + return RecoveryResult{Recovered: false, Method: "none"} +} +``` + +**Files to modify**: `pkg/agent/loop.go` + +### 3c. Auth Profile Rotation + +Enhance existing `FallbackChain`: + +```go +// pkg/providers/auth_rotation.go +type AuthProfile struct { + ID string + Provider string + APIKey string + ErrorCount int + CooldownUntil time.Time + DisabledUntil time.Time // billing track + LastUsed time.Time +} + +type AuthRotator struct { + profiles []AuthProfile + mu sync.RWMutex +} + +func (r *AuthRotator) NextAvailable() *AuthProfile { + // Round-robin: sort by lastUsed (oldest first), skip cooldown +} + +func (r *AuthRotator) MarkFailure(profileID string, reason FailoverReason) { + // Transient: exponential 1min → 5min → 25min → 1hr + // Billing: separate disabledUntil track +} + +func (r *AuthRotator) MarkSuccess(profileID string) { + // Reset errorCount, update lastUsed +} +``` + +**Files to create**: `pkg/providers/auth_rotation.go` + +### 3d. Cascade Stop + +```go +// pkg/multiagent/cascade.go +type RunRegistry struct { + active sync.Map // sessionKey -> *ActiveRun +} + +type ActiveRun struct { + SessionKey string + Cancel context.CancelFunc + Children []string // child session keys +} + +func (r *RunRegistry) CascadeStop(sessionKey string) int { + killed := 0 + if run, ok := r.active.Load(sessionKey); ok { + ar := run.(*ActiveRun) + ar.Cancel() + killed++ + for _, child := range ar.Children { + killed += r.CascadeStop(child) + } + r.active.Delete(sessionKey) + } + return killed +} +``` + +**Files to create**: `pkg/multiagent/cascade.go` + +--- + +## Phase 4: Async Multi-Agent + +### 4a. Async Spawn + +```go +// pkg/multiagent/spawn.go +type SpawnResult struct { + RunID string + SessionKey string +} + +func AsyncSpawn(ctx context.Context, resolver AgentResolver, board *Blackboard, + req *HandoffRequest, channel, chatID string) *SpawnResult { + + runID := uuid.New().String() + childKey := fmt.Sprintf("agent:%s:subagent:%s", req.ToAgentID, runID) + + go func() { + result := ExecuteHandoff(ctx, resolver, board, req, channel, chatID, maxDepth) + // Announce result back to parent + announceResult(req.FromAgentID, result, childKey) + }() + + return &SpawnResult{RunID: runID, SessionKey: childKey} +} +``` + +### 4b. Announce Protocol + +```go +// pkg/multiagent/announce.go +type AnnounceMode string + +const ( + AnnounceSteer AnnounceMode = "steer" // inject into active LLM call + AnnounceQueue AnnounceMode = "queue" // hold until idle + AnnounceDirect AnnounceMode = "direct" // send immediately +) + +type Announcement struct { + FromSessionKey string + ToSessionKey string + Content string + Mode AnnounceMode + RunID string +} +``` + +### 4c. Scope-Keyed Process Isolation + +Scope the exec tool's process visibility by session key: + +```go +// In exec tool construction +type ScopedExecTool struct { + scopeKey string + // processes only visible within this scope +} +``` + +### 4d. Idempotency + +```go +// pkg/gateway/dedup.go +type DedupCache struct { + entries sync.Map // key -> *DedupEntry + ttl time.Duration // default 5min + maxSize int // default 1000 +} +``` + +--- + +## Dependencies + +``` +Phase 1a (blackboard fix) ← no deps, start immediately +Phase 1b (recursion guard) ← no deps +Phase 1c (allowlist) ← 1b (uses same HandoffRequest changes) +Phase 1d (tool hooks) ← no deps + +Phase 2a (groups) ← no deps +Phase 2b (per-agent policy) ← 2a +Phase 2c (depth policy) ← 2b + 1b (needs depth tracking) +Phase 2d (pipeline) ← 2a + 2b + 2c + 1d (hooks infrastructure) + +Phase 3a (loop detection) ← 1d (hooks) +Phase 3b (context recovery) ← no deps +Phase 3c (auth rotation) ← no deps (enhances existing FallbackChain) +Phase 3d (cascade stop) ← 1b (depth tracking) + +Phase 4a (async spawn) ← 1b + 3d (depth + cascade) +Phase 4b (announce) ← 4a +Phase 4c (process isolation) ← no deps +Phase 4d (idempotency) ← no deps +``` + +```mermaid +graph TD + P1A[1a: Fix Blackboard] --> P2D + P1B[1b: Recursion Guard] --> P1C[1c: Allowlist] + P1B --> P2C + P1B --> P3D + P1D[1d: Tool Hooks] --> P2D[2d: Pipeline] + P1D --> P3A + + P2A[2a: Tool Groups] --> P2B[2b: Per-Agent Policy] + P2B --> P2C[2c: Depth Policy] + P2C --> P2D + + P3A[3a: Loop Detection] -.-> P2D + P3B[3b: Context Recovery] + P3C[3c: Auth Rotation] + P3D[3d: Cascade Stop] --> P4A + + P4A[4a: Async Spawn] --> P4B[4b: Announce Protocol] + P4C[4c: Process Isolation] + P4D[4d: Idempotency] + + style P1A fill:#ef4444,color:#fff + style P1B fill:#ef4444,color:#fff + style P1C fill:#ef4444,color:#fff + style P1D fill:#ef4444,color:#fff + style P2A fill:#f59e0b,color:#000 + style P2B fill:#f59e0b,color:#000 + style P2C fill:#f59e0b,color:#000 + style P2D fill:#f59e0b,color:#000 + style P3A fill:#3b82f6,color:#fff + style P3B fill:#3b82f6,color:#fff + style P3C fill:#3b82f6,color:#fff + style P3D fill:#3b82f6,color:#fff + style P4A fill:#8b5cf6,color:#fff + style P4B fill:#8b5cf6,color:#fff + style P4C fill:#8b5cf6,color:#fff + style P4D fill:#8b5cf6,color:#fff +``` + +--- + +## File Inventory + +### New Files + +| File | Phase | Description | +|------|-------|-------------| +| `pkg/tools/hooks.go` | 1d | ToolHook interface, chain execution | +| `pkg/tools/groups.go` | 2a | Tool group definitions | +| `pkg/tools/policy.go` | 2b-d | PolicyStep, pipeline, DepthPolicy | +| `pkg/tools/policy_test.go` | 2 | Policy pipeline tests | +| `pkg/tools/loop_detector.go` | 3a | Loop detection (repeat + ping-pong) | +| `pkg/tools/loop_detector_test.go` | 3a | Loop detection tests | +| `pkg/providers/auth_rotation.go` | 3c | Auth profile rotation + cooldown | +| `pkg/providers/auth_rotation_test.go` | 3c | Auth rotation tests | +| `pkg/multiagent/cascade.go` | 3d | RunRegistry, CascadeStop | +| `pkg/multiagent/cascade_test.go` | 3d | Cascade stop tests | +| `pkg/multiagent/spawn.go` | 4a | AsyncSpawn | +| `pkg/multiagent/announce.go` | 4b | Announce protocol | +| `pkg/gateway/dedup.go` | 4d | Idempotency cache | + +### Modified Files + +| File | Phase | Changes | +|------|-------|---------| +| `pkg/multiagent/blackboard_tool.go` | 1a | Add SetBoard / BoardSetter interface | +| `pkg/multiagent/handoff_tool.go` | 1a, 1b | SetBoard + depth propagation | +| `pkg/multiagent/handoff.go` | 1b, 1c | Depth/cycle guard + allowlist check | +| `pkg/agent/loop.go` | 1a, 1d, 2d | Wire session board, hook chain, policy pipeline | +| `pkg/config/config.go` | 1b, 2b | MaxHandoffDepth, ToolPolicyConfig | +| `pkg/tools/registry.go` | 1d | Add hooks field, ExecuteWithHooks | + +--- + +## OpenClaw Reference Map + +For each phase, the OpenClaw file to study: + +| Phase | picoclaw Target | OpenClaw Reference | +|-------|-----------------|-------------------| +| 1d | `pkg/tools/hooks.go` | `src/agents/pi-tools.before-tool-call.ts` | +| 2a | `pkg/tools/groups.go` | `src/agents/tool-policy.ts` (TOOL_GROUPS) | +| 2b-d | `pkg/tools/policy.go` | `src/agents/tool-policy-pipeline.ts` | +| 2c | depth deny | `src/agents/pi-tools.policy.ts` (resolveSubagentDenyList) | +| 3a | `pkg/tools/loop_detector.go` | `src/agents/tool-loop-detection.ts` | +| 3b | context recovery | `src/agents/pi-embedded-runner/run.ts` (overflow cascade) | +| 3c | `pkg/providers/auth_rotation.go` | `src/agents/auth-profiles/order.ts` + `usage.ts` | +| 3d | `pkg/multiagent/cascade.go` | `src/agents/tools/subagents-tool.ts` (cascadeKillChildren) | +| 4a | `pkg/multiagent/spawn.go` | `src/agents/subagent-spawn.ts` | +| 4b | `pkg/multiagent/announce.go` | `src/agents/subagent-announce.ts` | + +--- + +## Risk Assessment + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Blackboard fix breaks existing tests | Medium | Fix is additive — SetBoard is optional, existing behavior preserved if not called | +| Tool policy breaks single-agent mode | High | Default: no policy config = full access (backward compatible) | +| Loop detection false positives | Medium | Start with high thresholds (warn=10, block=20), tune based on real usage | +| Auth rotation race conditions | High | Use sync.Mutex for profile state, file lock for cross-session (like OpenClaw) | +| Async spawn goroutine leaks | High | Always use context.WithTimeout, track in RunRegistry | + +--- + +## Non-Goals + +- Visual AIEOS dashboard (future) +- Community agent marketplace (future) +- A2A protocol compatibility (future — after Phase 4) +- SOUL.md bootstrap enhancement (separate PR, handled by other developer) +- model_list / provider Phase 2-4 (separate track per issue #283)