diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md new file mode 100644 index 000000000..2605c7413 --- /dev/null +++ b/.planning/codebase/ARCHITECTURE.md @@ -0,0 +1,148 @@ +# Architecture + +**Analysis Date:** 2026-04-10 + +## Pattern Overview + +**Overall:** Event-driven agent gateway with message bus architecture + +**Key Characteristics:** +- Multi-agent support with per-instance workspace, session, and tool registry +- Message bus (`pkg/bus`) decouples channels from agent processing loops +- Channel manager (`pkg/channels`) abstracts 17+ messaging platforms behind unified interfaces +- Hot-reloadable gateway with graceful shutdown and provider fallback chains +- Embedded web launcher that serves both the dashboard UI and manages the gateway process + +## Layers + +**Gateway Layer (`pkg/gateway/`):** +- Purpose: Top-level runtime orchestrator — starts agent loops, channels, services +- Location: `pkg/gateway/gateway.go` +- Contains: Service lifecycle, config loading, signal handling +- Depends on: `pkg/agent`, `pkg/bus`, `pkg/channels`, `pkg/cron`, `pkg/health` +- Used by: CLI entry point (`cmd/picoclaw`), Web launcher (`web/backend/`) + +**Agent Layer (`pkg/agent/`):** +- Purpose: Core AI agent loop — LLM interaction, tool execution, context management +- Location: `pkg/agent/loop.go`, `pkg/agent/instance.go`, `pkg/agent/turn.go` +- Contains: AgentLoop, AgentInstance, AgentRegistry, EventBus, HookManager +- Depends on: `pkg/bus`, `pkg/providers`, `pkg/tools`, `pkg/session`, `pkg/memory` +- Used by: Gateway service + +**Bus Layer (`pkg/bus/`):** +- Purpose: Asynchronous message routing between channels and agents +- Location: `pkg/bus/bus.go`, `pkg/bus/types.go` +- Contains: MessageBus with inbound, outbound, media, audio, voice channels +- Depends on: `pkg/logger` +- Used by: All agent and channel code + +**Channel Layer (`pkg/channels/`):** +- Purpose: Platform-agnostic messaging interface with per-platform adapters +- Location: `pkg/channels/` with subpackages for each platform +- Contains: Channel interface, Manager, dynamic mux, per-platform implementations +- Depends on: `pkg/bus`, `pkg/config`, `pkg/health` +- Used by: Gateway service + +**Provider Layer (`pkg/providers/`):** +- Purpose: LLM provider abstraction with fallback, routing, and rate limiting +- Location: `pkg/providers/` with subpackages for Anthropic, OpenAI, Bedrock, etc. +- Contains: LLMProvider interface, factory, fallback chain, model router +- Depends on: `pkg/logger`, `pkg/config` +- Used by: Agent instances + +**Tool Layer (`pkg/tools/`):** +- Purpose: Tool registry and built-in tool implementations +- Location: `pkg/tools/` +- Contains: ToolRegistry, shell, filesystem, web, MCP, spawn, SPI/I2C hardware tools +- Depends on: `pkg/logger`, `pkg/providers` +- Used by: Agent instances during turn execution + +**Session Layer (`pkg/session/`):** +- Purpose: Session persistence and management with JSONL backend +- Location: `pkg/session/manager.go`, `pkg/session/session_store.go` +- Contains: SessionManager, SessionStore interface, JSONL backend +- Depends on: `pkg/providers` +- Used by: Agent instances for conversation history + +## Data Flow + +**Message Processing Flow:** + +1. External message arrives on a channel (e.g., Telegram webhook, Discord event, Pico WebSocket) +2. Channel adapter converts platform message to `bus.InboundMessage` and publishes to the bus +3. `AgentLoop` receives the inbound message from the bus +4. `EventBus` fires pre-processing hooks; `HookManager` executes registered hooks +5. `ContextBuilder` assembles the prompt (system, history, skills, tools) +6. Agent routes to the correct `AgentInstance` via `AgentRegistry` (with optional model routing) +7. LLM call via `Provider` (with fallback chain on failure) +8. Tool calls are dispatched via `ToolRegistry.Execute()` in a tool loop +9. Final content is published as `bus.OutboundMessage` +10. `ChannelManager` routes the response back through the originating channel +11. Channel sends response to the user on the platform + +**State Management:** +- Session state persisted as JSONL files in `~/.picoclaw/sessions/` +- Agent context managed in-memory with `ContextBuilder` + `ContextManager` +- Long-term memory via `pkg/memory` package (JSONL-based store) +- Config loaded from `~/.picoclaw/config.json` with hot-reload support + +## Key Abstractions + +**Channel Interface:** +- Purpose: Abstracts 17+ messaging platforms behind a common interface +- Examples: `pkg/channels/telegram/`, `pkg/channels/discord/`, `pkg/channels/pico/` +- Pattern: Capability-based interfaces (`TypingCapable`, `StreamingCapable`, `MessageEditor`, `ReactionCapable`, `PlaceholderCapable`, `CommandRegistrarCapable`) + +**LLMProvider Interface:** +- Purpose: Unified LLM interaction contract +- Examples: `pkg/providers/anthropic/`, `pkg/providers/openai_compat/`, `pkg/providers/bedrock/` +- Pattern: Factory-based provider creation with per-candidate credentials + +**Tool Interface:** +- Purpose: Pluggable tool execution with TTL-based registration +- Examples: `pkg/tools/shell.go`, `pkg/tools/filesystem.go`, `pkg/tools/mcp_tool.go` +- Pattern: `Tool` interface with `Name()`, `Description()`, `Parameters()`, `Execute()` + +**ContextManager:** +- Purpose: Manages conversation context window with budget-based truncation +- Examples: `pkg/agent/context_budget.go`, `pkg/agent/context_seahorse.go`, `pkg/agent/context_legacy.go` +- Pattern: Strategy pattern with multiple implementations for different context strategies + +## Entry Points + +**CLI Agent (`cmd/picoclaw/main.go`):** +- Location: `cmd/picoclaw/main.go` +- Triggers: Command-line invocation +- Responsibilities: Subcommands for agent, auth, cron, gateway, skills, model, migrate, status, version + +**Web Launcher (`web/backend/main.go`):** +- Location: `web/backend/main.go` +- Triggers: Direct execution or system tray +- Responsibilities: Embedded HTTP server (default port 18800), dashboard auth, gateway auto-start, system tray + +**Gateway Service (`pkg/gateway/gateway.go`):** +- Location: `pkg/gateway/gateway.go` +- Triggers: `gateway` subcommand or launcher auto-start +- Responsibilities: Full runtime — agent loops, channel workers, cron, heartbeat, health server + +## Error Handling + +**Strategy:** Structured logging with provider fallback chains + +**Patterns:** +- `providers.FallbackChain` — automatic failover to backup models on error +- `routing.Router` — intelligent model selection based on message complexity +- `channels.Manager` — per-channel rate limiting with exponential backoff +- `providers.ErrorClassifier` — categorizes errors (rate limit, auth, context, etc.) for appropriate retry behavior + +## Cross-Cutting Concerns + +**Logging:** `pkg/logger` — structured JSON logging with console/file modes, component-tagged output (`InfoCF`, `ErrorCF`, etc.) +**Validation:** `pkg/tools/validate.go` — tool parameter validation with JSON schema +**Authentication:** Dashboard auth (`web/backend/dashboardauth/`), OAuth flows (`pkg/auth/oauth.go`), PKCE for external platforms +**Configuration:** JSON/YAML config with environment variable override support (`pkg/config/`) +**Security:** Sensitive data filtering before LLM calls, credential storage isolation + +--- + +*Architecture analysis: 2026-04-10* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md new file mode 100644 index 000000000..c47345379 --- /dev/null +++ b/.planning/codebase/CONCERNS.md @@ -0,0 +1,251 @@ +# Codebase Concerns + +**Analysis Date:** 2026-04-10 + +--- + +## Security + +### 1. Hardcoded OAuth Client Credentials (Obfuscated but Reversible) + +**Files:** `pkg/auth/oauth.go:47-50` + +The Google Antigravity OAuth client ID and secret are base64-encoded inline strings, not environment variables. These are public OAuth client credentials (used by OpenCode/pi-ai) rather than secrets, so the risk is low. However, the pattern of embedding credentials in source code is a concern if extended to actual secrets. + +### 2. Self-Update Endpoint with No Binary Signature Verification + +**Files:** `pkg/updater/updater.go:33-80`, `pkg/updater/updater.go:611-646` + +The self-update mechanism downloads release archives from GitHub and extracts them. SHA256 checksum verification exists for release downloads, but there is no code signing verification of the binary itself. The `minio/selfupdate` library handles binary replacement. + +- **Mitigation present:** SHA256 checksum verification on release downloads +- **Remaining risk:** No cryptographic signature verification before applying the update +- **Fix approach:** Add signature verification (minisign is already an indirect dependency) + +### 3. WebSocket Proxy Token Validation via Custom Header + +**Files:** `web/backend/api/pico.go:57-100` + +The Pico WebSocket proxy validates tokens via a custom header. The token is compared against a cached config value. If the token changes in config while the gateway is running, there is a brief window where the cached token may be stale, allowing old tokens to work or rejecting valid ones. + +### 4. Login Rate Limiting is In-Memory Only + +**Files:** `web/backend/api/auth_login_limiter.go:17-40` + +The dashboard login rate limiter uses in-memory maps keyed by IP address. Rate limits reset on process restart, and in distributed deployments each instance has independent limits. + +### 5. No CSRF Protection on API Endpoints + +**Files:** `web/backend/api/router.go:52-95` + +The API routes use `http.ServeMux` directly with no CSRF middleware. If the server is run with `-public` flag, any website could make cross-origin requests unless CORS is properly configured. + +--- + +## Performance + +### 1. AgentLoop is a God File (3685 lines) + +**File:** `pkg/agent/loop.go` (3685 lines) + +Largest non-test file. Contains message routing (lines ~444-580), turn execution with tool loop (lines ~1800-2700), provider hot-reloading with goroutine isolation (lines ~982-1077), tool registration for all agents (lines ~165-442), and media resolution. + +**Fix approach:** Extract sub-components into separate files within `pkg/agent/`. + +### 2. Seahorse Store is Large (1542 lines) + +**File:** `pkg/seahorse/store.go` (1542 lines) + +All database operations in a single file with 44+ `ExecContext`/`QueryContext` calls. Uses parameterized queries (safe from injection), but size makes auditing difficult. + +### 3. JSONL Store maxLineSize = 10 MB + +**File:** `pkg/memory/jsonl.go:32` + +A single tool result can be up to 10 MB. Messages are never physically deleted from JSONL files -- only logically skipped via metadata offset. + +### 4. Media Cleanup Disabled + +**File:** `pkg/agent/loop.go:487-498` + +```go +// TODO: Re-enable media cleanup after inbound media is properly consumed by the agent. +// Currently disabled because files are deleted before the LLM can access their content. +``` + +Media files are never cleaned up, consuming disk space over time. + +--- + +## Complexity + +### 1. Complex Goroutine Lifecycle Management + +Key unbounded goroutine spawns: +- `pkg/agent/loop.go:1001` -- Registry creation in goroutine with recover +- `pkg/seahorse/short_compaction.go:80` -- Async condensed compaction per conversation +- `web/backend/api/gateway.go:758-782` -- Multiple goroutines for gateway process management +- `pkg/agent/loop.go:477` -- `drainBusToSteering` goroutine per message + +**Risk:** Goroutine leaks during error paths or rapid config reloads. + +### 2. Global Mutable State in Gateway Package + +**File:** `web/backend/api/gateway.go:29-43` + +Package-level mutable singleton holding process state, config signatures, and cached tokens. Protected by `sync.Mutex` but creates tight coupling between gateway lifecycle and API handler. + +### 3. Multiple Mutexes in Channel Implementations + +- `pkg/channels/wecom/wecom.go` -- 6 separate mutexes +- `pkg/channels/onebot/onebot.go` -- 3 mutexes +- `pkg/channels/manager.go` -- 1 RWMutex plus per-channel operations + +Increases deadlock risk if lock ordering is not consistent. + +--- + +## Technical Debt + +### 1. config_old.go -- Legacy Config Migration Code + +**File:** `pkg/config/config_old.go` (1001 lines) + +Contains V0 config structs for backward compatibility migration. Will grow as config schema evolves. + +### 2. Media Cleanup Commented Out (TODO) + +**File:** `pkg/agent/loop.go:487-498` + +Known feature regression -- media cleanup disabled due to timing issue. + +### 3. Logger TimeFormat Not Configurable (TODO) + +**File:** `pkg/logger/logger.go:55` + +### 4. MCP Tool Artifact Lifecycle Not Managed (TODO) + +**File:** `pkg/tools/mcp_tool.go:365` + +### 5. GitHub Copilot Provider Incomplete (TODO) + +**File:** `pkg/providers/github_copilot_provider.go:29` + +Only supports HTTP mode, not stdio. + +### 6. ASR Model Restriction Incomplete (TODO) + +**File:** `pkg/audio/asr/asr.go:36` + +--- + +## Error Handling + +### 1. Swallowed Errors in Config Reload + +**File:** `web/backend/api/gateway.go:55-60` + +```go +func refreshPicoTokensLocked(configPath string) { + cfg, err := config.LoadConfig(h.configPath) + if err != nil { + return // Error silently swallowed + } +``` + +If config reload fails, the token is not updated and no error is logged. + +### 2. Ignored `LastInsertId` Error + +**File:** `pkg/seahorse/store.go:57` + +### 3. Panics in Package Initialization + +- `pkg/agent/context_seahorse.go:267` +- `pkg/logger/panic_unix.go:16-19` +- `web/backend/main.go:126-140` + +--- + +## Scalability + +### 1. Single-Process Architecture + +Web backend, agent loop, channel connections, MCP servers, and cron jobs all share one process. + +### 2. In-Memory Rate Limiting + +**File:** `pkg/providers/ratelimiter.go:13` + +All rate limiting is in-memory; state lost on restart. + +### 3. JSONL File-per-Session Storage + +**File:** `pkg/memory/jsonl.go:46-55` + +Each session creates two files. Lock sharding (`numLockShards = 64`) mitigates contention but not file count growth. + +--- + +## Dependencies + +### 1. Large Dependency Surface for Channel Integrations + +15+ messaging platform SDKs compiled in regardless of usage. + +**Fix approach:** Consider build tags to compile only needed channels. + +### 2. WebRTC Dependency for Discord Voice + +**Files:** `pkg/channels/discord/voice.go` + +Pion WebRTC stack (~10 transitive deps) used only for Discord voice. + +--- + +## Maintainability + +### 1. Large Test Files + +- `pkg/agent/loop_test.go` -- 3367 lines +- `pkg/agent/subturn_test.go` -- 2067 lines +- `pkg/agent/steering_test.go` -- 1591 lines +- `pkg/config/config_test.go` -- 1976 lines + +### 2. Duplicate Test Patterns + +Agent test files contain repeated mock structures (`mockProvider`, `mockChannel`, `turnState`) defined inline rather than shared. + +--- + +## Data Integrity + +### 1. Race Condition Window in Conversation Creation + +**File:** `pkg/seahorse/store.go:35-62` + +Classic TOCTOU pattern handled via unique violation detection and retry. Correct for SQLite but fragile. + +### 2. No WAL Mode Configuration for SQLite + +Under concurrent load, this could cause "database is locked" errors. Compaction goroutines (`runCondensedLoop`) can write concurrently with ingestion. + +--- + +## Missing Critical Features + +### 1. No Audit Logging + +No structured audit log for security-sensitive operations. + +### 2. No Health/Metrics Endpoint + +`pkg/health/server.go` provides basic health checking but no Prometheus-compatible metrics. + +### 3. No Graceful Shutdown for All Components + +`AgentLoop.Run()` returns on context cancellation, but sub-goroutines (compaction, media, drains) may outlive the main loop. + +--- + +*Concerns analysis: 2026-04-10* diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md new file mode 100644 index 000000000..5c53fe778 --- /dev/null +++ b/.planning/codebase/CONVENTIONS.md @@ -0,0 +1,229 @@ +# Coding Conventions + +**Analysis Date:** 2026-04-10 + +## Naming Patterns + +**Packages:** +- Lowercase, single-word names (idiomatic Go). Examples: `logger`, `config`, `session`, `providers`, `seahorse`, `tokenizer`, `memory`, `isolation`, `credential`. +- Located under `pkg/` for shared libraries and `cmd/picoclaw/internal/` for CLI-internal packages. +- Internal packages live in `cmd/picoclaw/internal//`, each feature gets its own sub-package: `auth`, `agent`, `gateway`, `cron`, `skills`, `onboard`, `migrate`, `model`, `status`, `version`. + +**Functions:** +- Public: `PascalCase` -- `NewPicoclawCommand`, `SetLevelFromString`, `GetOrCreateConversation`, `RegisterLauncherAuthRoutes`. +- Private: `camelCase` -- `newBenchStore`, `logMessage`, `appendFields`, `getCallerSkip`, `openTestDB`. +- Constructor prefix: `New` for public (`NewAntigravityProvider`, `NewSubagentManager`), `new` for private (`newBenchStore`, `newTestCompactionEngineWithStore`). +- Cobra command constructors: `NewXxxCommand()` in `cmd/picoclaw/internal//` (e.g., `NewAuthCommand()`, `NewAgentCommand()`). Tests call `newXxxCommand()` (lowercase) when the constructor is internal. + +**Variables:** +- `camelCase` for locals and package-level variables: `currentLevel`, `logFile`, `rrCounter`, `consoleWriter`. +- Constants use `PascalCase` or `UPPER_SNAKE_CASE`: `CurrentVersion`, `Component`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`. +- Environment variable constants: `EnvHome`, `EnvConfig` in `pkg/config/`. Actual env var names: `PICOCLAW_HOME`, `PICOCLAW_CONFIG`, `PICOCLAW_LOG_FILE`, `TZ`, `ZONEINFO`. + +**Types/Interfaces:** +- `PascalCase` structs: `Config`, `Store`, `SessionManager`, `AntigravityProvider`, `JSONLBackend`. +- Interface names: `LLMProvider`, `SessionStore`, `CompleteFn` (function type alias), `Tool` (implicit). +- Type aliases for external types: `type LogLevel = zerolog.Level` in `pkg/logger/logger.go`. +- Config sub-types use `XxxConfig` suffix: `IsolationConfig`, `AgentsConfig`, `SessionConfig`, `ChannelsConfig`. + +## Code Formatting + +**Tooling:** `golangci-lint` v2 with formatters configured in `.golangci.yaml`. + +**Formatters enabled:** +- `gci` -- import grouping: `standard` -> `default` -> `localmodule` with custom order. +- `gofmt` -- with rewrite rules: `interface{}` -> `any`, `a[b:len(a)]` -> `a[b:]`. +- `gofumpt` -- strict formatting. +- `goimports` -- auto import management. +- `golines` -- max line length 120. + +**Commands:** +```bash +make fmt # runs golangci-lint fmt +make lint # runs golangci-lint run --build-tags=goolm,stdjson +make fix # runs golangci-lint run --fix --build-tags=goolm,stdjson +``` + +**Lint settings:** +- `default: all` with 35+ linters disabled (see `.golangci.yaml` lines 5-63). +- Line length: 120 chars (`.golangci.yaml` line 85). +- `funlen`: 120 lines / 40 statements. +- `gocognit`: min-complexity 25. +- `gocyclo`: min-complexity 20. +- `lll` exclusions for `//go:generate` lines. +- Test files excluded from `funlen`, `maintidx`, `gocognit`, `gocyclo` (`.golangci.yaml` lines 147-148). +- `testpackage` is disabled -- tests can use the same package name (not `_test`). + +## Import Organization + +**Order (gci config):** +1. Standard library +2. Third-party packages +3. Local module (`github.com/sipeed/picoclaw/...`) + +**Example from `cmd/picoclaw/main.go`:** +```go +import ( + "fmt" + "os" + "time" + + "github.com/spf13/cobra" + + "github.com/sipeed/picoclaw/cmd/picoclaw/internal" + "github.com/sipeed/picoclaw/cmd/picoclaw/internal/agent" + // ... more internal imports + "github.com/sipeed/picoclaw/pkg/config" + "github.com/sipeed/picoclaw/pkg/updater" +) +``` + +**Path aliases:** None used. Full import paths everywhere: `github.com/sipeed/picoclaw/pkg/...`. + +## Error Handling + +**Patterns:** +- `fmt.Errorf("context: %w", err)` for error wrapping with `fmt` and `%w`. Example from `pkg/seahorse/schema.go`: + ```go + return fmt.Errorf("FTS5 check: %w", err) + ``` +- Direct `return nil, fmt.Errorf("antigravity auth: %w", err)` in provider code (`pkg/providers/antigravity_provider.go`). +- Config errors use descriptive messages with `fmt.Errorf("failed to create log directory: %w", err)` (`pkg/logger/logger.go`). +- Tests use `t.Fatalf("operation: %v", err)` for fatal errors and `t.Errorf("description: %v", err)` for non-fatal. +- Some code logs errors rather than returning them (fire-and-forget pattern in `pkg/session/jsonl_backend.go`): + ```go + if err := b.store.AddMessage(...); err != nil { + log.Printf("session: add message: %v", err) + } + ``` +- Sentinel errors not widely used. Most errors are constructed inline with `fmt.Errorf`. + +## Logging + +**Framework:** `github.com/rs/zerolog` wrapped by a custom logger at `pkg/logger/logger.go`. + +**API pattern:** Suffix convention for logging variants: +- `Debug(message)` -- plain message, auto-detected component +- `DebugC(component, message)` -- explicit component +- `Debugf(message, args...)` -- sprintf-style +- `DebugF(message, fields)` -- structured fields as `map[string]any` +- `DebugCF(component, message, fields)` -- component + fields + +**Same pattern for all levels:** `Info/InfoC/Infof/InfoF/InfoCF`, `Warn/WarnC/Warnf/WarnF/WarnCF`, `Error/ErrorC/Errorf/ErrorF/ErrorCF`, `Fatal/FatalC/Fatalf/FatalF/FatalCF`. + +**Log format (TTY):** +``` +15:04:05 WARN component caller message +``` +Component shown in yellow (`\x1b[33m`). Time-only timestamps. + +**Log format (non-TTY):** JSON (zerolog default). + +**Configuration from env:** `PICOCLAW_LOG_FILE` -- if set, enables file logging and disables console. Supports `~/` expansion. + +**Global logger:** Package-level singleton with `sync.RWMutex` for thread safety. Not passed as dependency -- imported directly. + +## Configuration Patterns + +**Main config:** `pkg/config/config.go` -- JSON-based with `json` struct tags. + +**Loading:** +- Config file path via `PICOCLAW_CONFIG` env var or `PICOCLAW_HOME` env var (defaults to `~/.picoclaw`). +- Environment variable overrides via `github.com/caarlos0/env/v11`. +- Schema versioning (`CurrentVersion = 2`) with migration support. +- Config struct uses `json:"-"` on most fields (only `channels`, `model_list`, and build info serialize). + +**Version injection via ldflags:** +``` +-X github.com/sipeed/picoclaw/pkg/config.Version=... +-X github.com/sipeed/picoclaw/pkg/config.GitCommit=... +-X github.com/sipeed/picoclaw/pkg/config.BuildTime=... +-X github.com/sipeed/picoclaw/pkg/config.GoVersion=... +``` +Accessed via `config.GetVersion()`, `config.GetGitCommit()`, etc. + +## Interface Design + +**Style:** Small, focused interfaces. Examples: +- `LLMProvider` -- `Chat(ctx, messages, tools, model, options) (*LLMResponse, error)` + `GetDefaultModel()`, `SupportsTools()`, `GetContextWindow()`. +- `SessionStore` -- `AddMessage`, `AddFullMessage`, `GetHistory`, `GetSummary`, `SetSummary`, `SetHistory`, `TruncateHistory`, `Save`, `Close`. +- `memory.Store` -- file-based session persistence. + +**Function type aliases** used for callbacks: `type CompleteFn func(ctx, prompt, opts) (string, error)` in seahorse compaction. + +**Interface implementations** are created via constructor functions: `NewAntigravityProvider()`, `NewJSONLBackend(store)`, `NewSubagentManager(provider, model, workspace)`. + +## Struct Organization + +**Pattern:** +- Exported structs with public fields for configuration (JSON tags). +- Unexported fields for internal state: `sensitiveCache *SensitiveDataCache`, `store memory.Store`. +- Methods on pointer receivers: `func (c *Config) FilterSensitiveData(...)`. +- Small config sub-structs composed into main `Config`. + +**Example from `pkg/config/config.go`:** +```go +type Config struct { + Version int `json:"version"` + Isolation IsolationConfig `json:"isolation,omitempty"` + Agents AgentsConfig `json:"agents"` + // ... many more sub-configs + sensitiveCache *SensitiveDataCache // unexported, computed cache +} +``` + +## Git Commit Message Style + +**Format:** Conventional Commits. Examples from recent commits: +``` +fix(chat): keep tool-call summary and assistant output in sync (#2449) +fix(seahorse): sanitize user input for FTS5 MATCH queries (#2436) +fix(launcher): align react and react-dom versions (#2467) +build(deps): bump github.com/modelcontextprotocol/go-sdk (#2455) +feat(launcher): standard HTTP login/setup/logout flow for dashboard... +style(lint): satisfy gci and golines for review fixes +fix(agent): gate pico interim publish for internal turns +``` + +**Rules (from CONTRIBUTING.md):** +- English language, imperative mood: "Add retry logic" not "Added retry logic". +- Reference issues: `Fix session leak (#123)`. +- One logical change per commit. +- Squash minor cleanups/typos into a single commit. +- Follow https://www.conventionalcommits.org/zh-hans/v1.0.0/ +- Squash merge is the default strategy. + +## Branch Naming + +**Pattern:** `type/description` -- examples from CONTRIBUTING.md: +- `fix/telegram-timeout` +- `feat/ollama-provider` +- `docs/contributing-guide` + +**Long-lived branches:** `main` (active development), `release/x.y` (stable releases). + +## Documentation Style + +**Package comments:** Minimal. Only the main entry point (`cmd/picoclaw/main.go`) has a header comment block with project description and license. + +**Function comments:** Godoc style when present, mostly on public API: +```go +// ParseLevel converts a case-insensitive level name to a LogLevel. +// Returns the level and true if valid, or (INFO, false) if unrecognized. +func ParseLevel(s string) (LogLevel, bool) { ... } + +// NewAntigravityProvider creates a new Antigravity provider using stored auth credentials. +func NewAntigravityProvider() *AntigravityProvider { ... } +``` + +**Internal code:** Comments explain non-obvious logic, especially in `pkg/seahorse/` (schema, compaction) and complex SQL. Bug-fix tests include detailed BUG comments explaining the issue (see `pkg/seahorse/store_test.go` lines 497-619). + +**nolint directives:** Used sparingly with justification: +```go +//nolint:zerologlint +func getEvent(logger zerolog.Logger, level LogLevel) *zerolog.Event { ... } +``` + +--- + +*Convention analysis: 2026-04-10* diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md new file mode 100644 index 000000000..ed8b5c418 --- /dev/null +++ b/.planning/codebase/INTEGRATIONS.md @@ -0,0 +1,212 @@ +# External Integrations + +**Analysis Date:** 2026-04-10 + +## AI Model Providers + +PicoClaw supports a protocol-based model routing system (`pkg/providers/factory_provider.go`). Models are specified as `protocol/model-id` (e.g., `openai/gpt-4o`, `anthropic/claude-sonnet-4.6`). Default protocol is `openai` when no prefix. + +**Native protocol providers (non-OpenAI-compatible):** +- **Anthropic** (`pkg/providers/anthropic/provider.go`) - Claude models, OAuth or API key auth +- **Anthropic Messages** (`pkg/providers/anthropic_messages/provider.go`) - Messages API variant +- **Claude CLI** (`pkg/providers/claude_cli_provider.go`) - Claude Code CLI via stdio +- **Codex CLI** (`pkg/providers/codex_cli_provider.go`) - OpenAI Codex CLI via stdio +- **AWS Bedrock** (`pkg/providers/bedrock/provider_bedrock.go`) - AWS SDK v2, IAM or credentials +- **Azure OpenAI** (`pkg/providers/azure/provider.go`) - Azure-specific endpoint handling +- **GitHub Copilot** (`pkg/providers/github_copilot_provider.go`) - Copilot SDK integration +- **Antigravity** (`pkg/providers/antigravity_provider.go`) - Custom provider + +**OpenAI-compatible protocols** (`pkg/providers/factory_provider.go`, lines 25-61): +- `openai` (api.openai.com/v1) +- `openrouter` (openrouter.ai/api/v1) +- `groq` (api.groq.com/openai/v1) +- `deepseek` (api.deepseek.com/v1) +- `ollama` (localhost:11434/v1, no API key required) +- `lmstudio` (localhost:1234/v1, no API key required) +- `vllm` (localhost:8000/v1, no API key required) +- `gemini` (generativelanguage.googleapis.com/v1beta) +- `litellm` (localhost:4000/v1) +- `qwen` / `qwen-intl` / `qwen-us` (Dashscope, Alibaba Cloud) +- `moonshot`, `mistral`, `cerebras`, `nvidia`, `volcengine`, `modelscope`, `novita`, `minimax`, `longcat`, `avian`, `zhipu`, `venice`, `vivgrid` +- Coding-specific: `coding-plan`, `alibaba-coding`, `coding-plan-anthropic` + +## Messaging Channels (Chat Platforms) + +Registered via blank imports in `pkg/gateway/gateway.go` (lines 21-37). Each channel lives in `pkg/channels//`. + +| Channel | Package | SDK | Auth | Protocol | +|---------|---------|-----|------|----------| +| Telegram | `pkg/channels/telegram/` | telego v1.8.0 | Bot token | Polling | +| Discord | `pkg/channels/discord/` | discordgo (fork) v0.29.0 | Bot token | WebSocket | +| Feishu/Lark | `pkg/channels/feishu/` | oapi-sdk-go v3.5.3 | App ID + secret | Webhook | +| Slack | `pkg/channels/slack/` | slack-go v0.17.3 | Bot token + App token | WebSocket (Socket Mode) | +| DingTalk | `pkg/channels/dingtalk/` | dingtalk-stream-sdk v0.9.1 | Client ID + secret | Stream | +| QQ | `pkg/channels/qq/` | botgo v0.2.1 | App ID + secret | WebSocket | +| WhatsApp | `pkg/channels/whatsapp/` | Custom bridge | Bridge URL | HTTP bridge | +| WhatsApp Native | `pkg/channels/whatsapp_native/` | whatsmeow | Direct login | WebSocket | +| WeCom | `pkg/channels/wecom/` | Custom | Bot ID + secret | WebSocket | +| Weixin | `pkg/channels/weixin/` | Custom | Token | HTTP webhook | +| Matrix | `pkg/channels/matrix/` | mautrix v0.26.4 | Homeserver + access token | Matrix API | +| IRC | `pkg/channels/irc/` | irc-go v0.6.0 | Nick/password/SASL | IRC | +| LINE | `pkg/channels/line/` | REST client | Channel secret + token | Webhook | +| OneBot | `pkg/channels/onebot/` | WebSocket client | Access token | WebSocket | +| VK | `pkg/channels/vk/` | vksdk v3.3.1 | Group token | Long polling | +| Teams | `pkg/channels/teams_webhook/` | go-teams-notify v2.14.0 | Webhook URL | HTTP POST | +| MaixCam | `pkg/channels/maixcam/` | HTTP client | None | HTTP | +| Pico | `pkg/channels/pico/` | Custom WebSocket | Token | WebSocket (built-in web channel) | +| PicoClient | `pkg/channels/pico/` | Custom WebSocket client | Token | WebSocket (connect to remote Pico) | + +## MCP (Model Context Protocol) Integration + +**Manager:** `pkg/mcp/manager.go` + +- Uses `github.com/modelcontextprotocol/go-sdk v1.5.0` +- Supports **stdio** transport (spawn external process) and **Streamable HTTP** transport +- Reads env files for MCP server configuration (`loadEnvFile()` line 49) +- Custom HTTP headers supported via `headerTransport` wrapper +- MCP tools are wrapped as native PicoClaw tools via `pkg/tools/mcp_tool.go` +- Isolated command transport for sandboxed MCP servers (`pkg/mcp/isolated_command_transport.go`) + +**MCP server configuration** is defined in `ToolsConfig.MCPServers` with fields: `Command`, `Args`, `Env`, `URL`, `Headers`, `EnvFile`. + +## Built-in Tools + +Defined in `pkg/tools/`: + +- **Shell execution** (`shell.go`) - Run shell commands with PTY support, timeout, allow/deny patterns +- **File operations** (`filesystem.go`) - Read, write, edit, list files (workspace-restricted) +- **Edit** (`edit.go`) - Search-and-replace file editing +- **Web search** (`search_tool.go`) - Multiple backends: Brave, Tavily, DuckDuckGo, Perplexity, SearXNG, GLM Search, Baidu Search +- **Web fetch** (`web.go`) - Fetch and extract web page content +- **Subagent** (`subagent.go`) - Spawn sub-agents for delegated tasks +- **Cron** (`cron.go`) - Schedule and manage cron jobs +- **Message tools** (`message.go`) - Send messages, reactions +- **File send** (`send_file.go`) - Send files to chat +- **Image loading** (`load_image.go`) - Load and process images +- **SPI/I2C** (`spi.go`, `i2c.go`) - Hardware bus access (Linux-only, for embedded devices) +- **Skills** (`skills_install.go`, `skills_search.go`) - Install and search skills marketplace +- **TTS send** (`tts_send.go`) - Text-to-speech output +- **Spawn** (`spawn.go`) - Process spawning with status tracking + +## Audio Integrations (ASR/TTS) + +**Speech-to-Text (ASR)** (`pkg/audio/asr/`): +- **OpenAI-compatible Whisper** (`whisper_transcriber.go`) - Via any OpenAI-compatible provider +- **ElevenLabs** (`elevenlabs_transcriber.go`) - ElevenLabs STT API +- **Audio model transcriber** (`audio_model_transcriber.go`) - Generic audio-capable LLM + +**Text-to-Speech (TTS)** (`pkg/audio/tts/`): +- **OpenAI-compatible TTS** (`openai_tts.go`) - Any OpenAI-compatible endpoint +- **Mimo TTS** (`mimo_tts.go`) - Xiaomi Mimo TTS service + +**Audio codecs:** +- OGG Opus encoding (`pkg/audio/ogg.go`) +- Sentence segmentation (`pkg/audio/sentence.go`) + +## Third-Party Library Integrations + +**Data processing:** +- `tidwall/gjson/sjson/pretty/match` - JSON manipulation +- `segmentio/encoding` - High-performance JSON +- `bytedance/sonic` - Fast JSON serialization +- `vmihailenco/msgpack/v5` - MessagePack serialization + +**Security:** +- `cloudflare/circl` - Cryptographic primitives +- `aead.dev/minisign` - Minisign signature verification (for release updates) +- `filippo.io/edwards25519` - Ed25519 curves + +**Observability:** +- `opentelemetry.io/otel` (v1.35.0) - OpenTelemetry tracing (auto-instrumentation) +- `zerolog` - Structured logging with sensitive data filtering + +## Docker/Containerization + +**Images:** +- `docker/Dockerfile` - Minimal Alpine image (`alpine:3.23`), Go 1.25 builder, health check on `:18790/health` +- `docker/Dockerfile.full` - Full image with Node.js 24 + `uv`/`uvx` (Python) for MCP tool support +- `docker/Dockerfile.goreleaser` - Release build variant +- `docker/Dockerfile.goreleaser.launcher` - Launcher-specific variant +- `docker/Dockerfile.heavy` - Heavy variant +- `docker/entrypoint.sh` - First-run entrypoint + +**Docker Compose** (`docker/docker-compose.yml`): +- `picoclaw-agent` - One-shot agent mode (profile: `agent`) +- `picoclaw-gateway` - Long-running bot (profile: `gateway`) +- `picoclaw-launcher` - Web console (profile: `launcher`, ports 18800/18790) + +**Docker Compose Full** (`docker/docker-compose.full.yml`): +- Same services but with full MCP tool support (Node.js runtime) + +**Image registry:** `docker.io/sipeed/picoclaw:latest` / `:full` / `:launcher` + +## CI/CD Configuration + +**GitHub Actions** (`.github/workflows/`): + +| Workflow | File | Purpose | +|----------|------|---------| +| Build | `build.yml` | On push to `main` - `make build-all` on ubuntu-latest | +| PR | `pr.yml` | On pull requests - build + test | +| Release | `release.yml` | On tags - multi-platform release artifacts | +| Nightly | `nightly.yml` | Scheduled nightly builds | +| Docker | `docker-build.yml` | Docker image build and push | +| macOS DMG | `create_dmg.yml` | macOS application bundle | +| TOS upload | `upload-tos.yml` | Terms of service upload | + +**Dependabot** (`.github/dependabot.yml`) - Automated dependency updates + +## Plugin/Extension Systems + +**Skills** - Marketplace-style extension system: +- Skills installed to `~/.picoclaw/workspace/skills/` +- Built-in skills in `skills/` directory at project root +- CLI commands: `picoclaw skills install`, `skills list`, `skills search`, `skills remove`, `skills list-builtin`, `skills show` +- Tools: `pkg/tools/skills_install.go`, `pkg/tools/skills_search.go` +- Skills package: `pkg/skills/` + +**MCP Servers** - External tool providers: +- Configured via `config.json` under `tools.mcp_servers` +- Spawned as child processes (stdio) or connected via HTTP (Streamable HTTP) +- Tools auto-discovered and registered as native tools + +**Process Hooks** (`HooksConfig` in `pkg/config/config.go`): +- Observer/interceptor hooks on agent events +- Built-in hooks (e.g., content moderation) +- Custom process hooks (spawn external process, observe/intercept events) + +**Channel registry** (`pkg/channels/registry.go`): +- Channels auto-register via `init()` functions in their packages +- New channels added by creating a package under `pkg/channels//` with an `init.go` that calls `Register()` + +**Provider factory** (`pkg/providers/factory_provider.go`): +- New OpenAI-compatible providers added by adding a protocol entry to `protocolMetaByName` map +- No code change needed for most new providers - just config + +## Environment Variables + +**Critical configuration:** +- `PICOCLAW_CHANNELS__TOKEN` - Channel authentication tokens +- `PICOCLAW_AGENTS_DEFAULTS_PROVIDER` / `MODEL_NAME` - Default model +- `PICOCLAW_TOOLS_WEB_BRAVE_API_KEYS` / `TAVILY_API_KEYS` - Search API keys +- `PICOCLAW_HOME` - Override home directory (default: `~/.picoclaw`) +- `TZ` / `ZONEINFO` - Timezone configuration + +**Secrets:** Stored in `~/.picoclaw/config.json` (with `SecureString` masking) and `~/.picoclaw/auth.json` (OAuth tokens). Neither file should be committed. + +## Webhooks & Callbacks + +**Incoming:** +- Feishu webhook endpoint (channel-specific) +- LINE webhook (`webhook_host:webhook_port/webhook_path`) +- Weixin webhook +- LINE webhook for callback events +- Teams webhook (output-only, no incoming) + +**Outgoing:** +- Teams webhook notifications (`pkg/channels/teams_webhook/`) +- Web search API calls (Brave, Tavily, Perplexity, etc.) + +--- + +*Integration audit: 2026-04-10* diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md new file mode 100644 index 000000000..d0cb81ae5 --- /dev/null +++ b/.planning/codebase/STACK.md @@ -0,0 +1,164 @@ +# Technology Stack + +**Analysis Date:** 2026-04-10 + +## Languages + +**Primary:** +- **Go 1.25.9** (per `go.mod` line 3) - Backend runtime, all core logic +- **TypeScript ~5.9.3** - Web frontend (`web/frontend/`) + +**Secondary:** +- **JavaScript/Node.js 24** - Full Docker image runtime for MCP tools (`docker/Dockerfile.full`) + +## Runtime + +**Environment:** +- Go 1.25.9 (compiled binary, CGO_ENABLED=0 by default; CGO_ENABLED=1 on macOS for systray) +- Node.js ^20.19.0 || ^22.13.0 || >=24 (frontend via `web/frontend/package.json` engines field) + +**Package Managers:** +- **Go modules** - Lockfile: `go.sum` present +- **pnpm** - Lockfile: `web/frontend/pnpm-lock.yaml` present (inferred from pnpm-lock.yaml) + +## Frameworks + +**Core (Go):** +- **cobra v1.10.2** (`cmd/picoclaw/main.go`) - CLI framework, root command with subcommands: `onboard`, `agent`, `auth`, `gateway`, `status`, `cron`, `migrate`, `skills`, `model`, `version`, `update` +- **modelcontextprotocol/go-sdk v1.5.0** (`pkg/mcp/manager.go`) - MCP (Model Context Protocol) client for external tool servers +- **zerolog v1.35.0** (`pkg/logger/`) - Structured logging +- **modernc.org/sqlite v1.48.2** - SQLite database (pure Go, no CGO) + +**Frontend (web/frontend/):** +- **React 19.2.5** - UI framework +- **Vite 8.0.8** - Build tool and dev server +- **TailwindCSS 4.2.2** - Styling with `@tailwindcss/vite` plugin +- **Radix UI 1.4.3** - Headless component primitives +- **TanStack Router 1.167.0 + React Query 5.97.0** - Routing and data fetching +- **Jotai 2.19.1** - Atomic state management +- **i18next 26.0.3** - Internationalization (en/zh) + +**Backend Web (web/backend/):** +- **net/http** (stdlib) - HTTP server for launcher dashboard + +**Testing:** +- **testify v1.11.1** - Assertion library and mocking (`github.com/stretchr/testify`) + +**Build/Dev:** +- **golangci-lint** - Linting (configured via `.golangci.yaml`) +- **Make** - Build orchestration (`Makefile`, 399 lines) + +## Key Dependencies + +**AI Model Providers (SDK clients):** +- `anthropic-sdk-go v1.26.0` - Anthropic Claude API +- `openai-go/v3 v3.22.0` - OpenAI API +- `aws-sdk-go-v2 + bedrockruntime v1.50.4` - AWS Bedrock +- `github/copilot-sdk/go v0.2.0` - GitHub Copilot CLI + +**Messaging/Chat SDKs (Channels):** +- `slack-go/slack v0.17.3` - Slack +- `bwmarrin/discordgo v0.29.0` (replaced with `yeongaori/discordgo-fork`) - Discord +- `mymmrac/telego v1.8.0` - Telegram +- `larksuite/oapi-sdk-go/v3 v3.5.3` - Feishu/Lark +- `open-dingtalk/dingtalk-stream-sdk-go v0.9.1` - DingTalk +- `tencent-connect/botgo v0.2.1` - QQ +- `ergochat/irc-go v0.6.0` - IRC +- `SevereCloud/vksdk/v3 v3.3.1` - VK +- `atc0005/go-teams-notify/v2 v2.14.0` - Microsoft Teams +- `maunium.net/go/mautrix v0.26.4` - Matrix +- `go.mau.fi/whatsmeow` - WhatsApp (native) +- `gorilla/websocket v1.5.3` - WebSocket (Pico channel) + +**Audio (ASR/TTS):** +- `pion/webrtc/v3 v3.3.6` + `pion/rtp v1.10.1` - WebRTC (Discord voice, audio streaming) +- ElevenLabs transcriber (`pkg/audio/asr/elevenlabs_transcriber.go`) +- OpenAI-compatible TTS (`pkg/audio/tts/openai_tts.go`) +- Mimo TTS (`pkg/audio/tts/mimo_tts.go`) + +**Web/HTTP:** +- `valyala/fasthttp v1.69.0` - High-performance HTTP (Feishu channel) +- `go-resty/resty/v2 v2.17.1` - HTTP client +- `klauspost/compress v1.18.4` - Compression + +**Terminal UI (Launcher TUI):** +- `gdamore/tcell/v2 v2.13.8` - Terminal cell library +- `rivo/tview v0.42.0` - Terminal UI widgets +- `ergochat/readline v0.1.3` - Readline support + +**Utilities:** +- `spf13/cobra v1.10.2` - CLI framework +- `caarlos0/env/v11 v11.4.0` - Environment variable parsing with struct tags +- `BurntSushi/toml v1.6.0` - TOML parsing +- `adhocore/gronx v1.19.6` - Cron expression parsing +- `google/uuid v1.6.0` - UUID generation +- `h2non/filetype v1.1.3` - File type detection +- `mdP/qrterminal/v3 v3.2.1` - QR code terminal output +- `minio/selfupdate v0.6.0` - Binary self-updates +- `creack/pty v1.1.24` - PTY allocation (shell tool) +- `gomarkdown/markdown` - Markdown parsing + +## Database/Storage + +**Primary:** +- **SQLite** via `modernc.org/sqlite v1.48.2` (pure Go, no CGO dependency) - Used by dashboard auth store (`web/backend/dashboardauth/sql.go`) +- **JSONL files** (`pkg/memory/jsonl.go`) - Session memory storage, append-only JSONL format with per-session metadata files +- **JSON files** - Config (`config.json`), auth store (`auth.json`), state (`state/state.json`) + +**No external database server required** - all storage is file-based. + +## Authentication Mechanisms + +1. **API Key auth** - Per-model `api_keys` in config (supports multiple keys for failover, `SecureString` wrapper) +2. **OAuth/PKCE** (`pkg/auth/`) - Anthropic and OpenAI OAuth login flows with token refresh +3. **Platform-specific tokens** - Each channel has its own token/secret (Telegram bot token, Discord token, Feishu app secret, etc.) +4. **Dashboard auth** - Launcher token-based (`PICOCLAW_LAUNCHER_TOKEN` env var) for web console +5. **WeCom/Weixin** - Custom OAuth flows (`pkg/auth/wecom.go`, `pkg/auth/weixin.go`) + +## Configuration + +**Method:** +- JSON config file (`~/.picoclaw/config.json`) +- Environment variables with `env:` struct tags (e.g., `PICOCLAW_CHANNELS_TELEGRAM_TOKEN`) +- `pkg/config/` handles loading, validation, and secure string masking + +**Key directories:** +- `~/.picoclaw/` - Home directory (config, auth, workspace) +- `~/.picoclaw/workspace/` - Workspace (state, skills, session memory) +- `~/.picoclaw/logs/` - Log files + +## Build System + +**Makefile targets** (key ones): +- `make build` - Build for current platform (runs `go generate` first) +- `make build-all` - Cross-compile for 10+ platforms +- `make build-launcher` - Build web console binary +- `make build-launcher-tui` - Build terminal UI binary +- `make build-whatsapp-native` - Build with native WhatsApp support (larger binary) +- `make build-linux-arm` / `build-linux-arm64` / `build-linux-mipsle` / `build-pi-zero` - Embedded targets +- `make test` - Run all tests +- `make lint` / `make fmt` / `make vet` - Code quality +- `make docker-build` / `docker-build-full` - Docker images +- `make install` / `make uninstall` - Local install to `~/.local/bin` + +**Build tags:** `goolm,stdjson` (default); `whatsapp_native` for native WhatsApp + +**Supported platforms:** linux/amd64, linux/arm, linux/arm64, linux/loong64, linux/riscv64, linux/mipsle, darwin/arm64, windows/amd64, netbsd/amd64, netbsd/arm64 + +## Platform Requirements + +**Development:** +- Go 1.25.9+ +- Node.js 20+ (for frontend builds) +- pnpm (for frontend dependencies) +- golangci-lint (for linting) +- Make + +**Production:** +- Single static binary (no runtime dependencies when CGO_ENABLED=0) +- Alpine 3.23 (minimal Docker image) +- Node.js 24 (full Docker image for MCP tool support) + +--- + +*Stack analysis: 2026-04-10* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md new file mode 100644 index 000000000..90c6e8488 --- /dev/null +++ b/.planning/codebase/STRUCTURE.md @@ -0,0 +1,265 @@ +# Codebase Structure + +**Analysis Date:** 2026-04-10 + +## Directory Layout + +``` +picoclaw/ +├── cmd/ # CLI entry points +│ ├── picoclaw/ # Main CLI agent +│ │ ├── main.go # CLI root command +│ │ └── internal/ # CLI subcommand implementations +│ │ ├── agent/ # `picoclaw agent` command +│ │ ├── auth/ # `picoclaw auth` command +│ │ ├── cron/ # `picoclaw cron` subcommands +│ │ ├── gateway/ # `picoclaw gateway` command +│ │ ├── migrate/ # `picoclaw migrate` command +│ │ ├── model/ # `picoclaw model` command +│ │ ├── onboard/ # `picoclaw onboard` command +│ │ ├── skills/ # `picoclaw skills` subcommands +│ │ ├── status/ # `picoclaw status` command +│ │ └── version/ # `picoclaw version` command +│ ├── picoclaw-launcher-tui/ # Terminal UI launcher +│ │ ├── main.go +│ │ ├── ui/ # TUI screen components +│ │ └── config/ # TUI configuration +│ └── membench/ # Memory/performance benchmarking +├── pkg/ # Shared libraries (all reusable packages) +│ ├── agent/ # Agent loop, instances, registry +│ ├── audio/ # Audio processing (ASR/TTS) +│ │ ├── asr/ # Speech recognition (Whisper, ElevenLabs, etc.) +│ │ └── tts/ # Text-to-speech (OpenAI, Mimo, etc.) +│ ├── auth/ # OAuth, PKCE, token management +│ ├── bus/ # Message bus (core async messaging) +│ ├── channels/ # Messaging platform adapters +│ │ ├── dingtalk/ # DingTalk (钉钉) integration +│ │ ├── discord/ # Discord bot + voice +│ │ ├── feishu/ # Feishu/Lark integration +│ │ ├── irc/ # IRC client +│ │ ├── line/ # LINE messaging +│ │ ├── maixcam/ # MaixCam device channel +│ │ ├── matrix/ # Matrix protocol +│ │ ├── onebot/ # OneBot protocol (QQ) +│ │ ├── pico/ # Built-in WebSocket channel (for web UI) +│ │ ├── qq/ # QQ Bot +│ │ ├── slack/ # Slack integration +│ │ ├── teams_webhook/ # Microsoft Teams webhook +│ │ ├── telegram/ # Telegram bot +│ │ ├── vk/ # VK (ВКонтакте) +│ │ ├── wecom/ # WeChat Work (企业微信) +│ │ ├── weixin/ # WeChat (微信) +│ │ ├── whatsapp/ # WhatsApp (web-based) +│ │ └── whatsapp_native/ # WhatsApp native integration +│ ├── commands/ # Command definition and registry +│ ├── config/ # Configuration loading, validation, migration +│ ├── constants/ # Shared constants +│ ├── credential/ # Secure credential storage +│ ├── cron/ # Cron job scheduler +│ ├── devices/ # Device management (hardware I/O) +│ ├── fileutil/ # File system utilities +│ ├── gateway/ # Gateway runtime orchestrator +│ ├── health/ # Health check server +│ ├── heartbeat/ # Heartbeat service +│ ├── identity/ # User identity management +│ ├── isolation/ # Process isolation/sandboxing +│ ├── logger/ # Structured logging +│ ├── mcp/ # MCP (Model Context Protocol) support +│ ├── media/ # Media file storage +│ ├── memory/ # Long-term conversation memory (JSONL) +│ ├── migrate/ # Config/data migration +│ ├── pid/ # PID file management +│ ├── providers/ # LLM provider implementations +│ │ ├── anthropic/ # Anthropic API (Claude) +│ │ ├── anthropic_messages/ # Anthropic Messages API +│ │ ├── azure/ # Azure OpenAI +│ │ ├── bedrock/ # AWS Bedrock +│ │ ├── openai_compat/ # OpenAI-compatible providers +│ │ └── common/ # Shared provider utilities +│ ├── routing/ # Model routing (smart model selection) +│ ├── seahorse/ # Context compression engine (FTS5-based) +│ ├── session/ # Session management (JSONL backend) +│ ├── skills/ # Skill system (agent capabilities) +│ ├── state/ # State management +│ ├── tokenizer/ # Token counting/estimation +│ ├── tools/ # Tool implementations +│ ├── updater/ # Self-update mechanism +│ └── utils/ # General utilities +├── web/ +│ ├── backend/ # Web launcher backend (Go) +│ │ ├── main.go # Launcher entry point +│ │ ├── api/ # REST API handlers +│ │ │ ├── router.go # Route registration +│ │ │ ├── channels.go # Channel CRUD +│ │ │ ├── config.go # Config management +│ │ │ ├── gateway.go # Gateway start/stop/logs +│ │ │ ├── models.go # Model list management +│ │ │ ├── oauth.go # OAuth flow handlers +│ │ │ ├── pico.go # WebSocket chat endpoint +│ │ │ ├── session.go # Session history API +│ │ │ ├── skills.go # Skills management +│ │ │ └── tools.go # Tool actions +│ │ ├── dashboardauth/ # Dashboard authentication +│ │ ├── launcherconfig/ # Launcher-specific config +│ │ ├── middleware/ # HTTP middleware (auth, access control) +│ │ ├── model/ # Status models +│ │ └── utils/ # Backend utilities +│ └── frontend/ # Web dashboard UI (React + TypeScript) +│ ├── src/ +│ │ ├── api/ # API client layer +│ │ ├── components/ # Reusable UI components +│ │ │ ├── agent/ # Agent-related components +│ │ │ │ ├── hub/ # Agent Hub marketplace +│ │ │ │ ├── skills/ # Skills display +│ │ │ │ └── tools/ # Tool configuration +│ │ │ ├── channels/ # Channel management +│ │ │ ├── chat/ # Chat UI components +│ │ │ ├── config/ # Configuration forms +│ │ │ ├── credentials/ # Credential management +│ │ │ ├── logs/ # Log viewer +│ │ │ ├── models/ # Model selector +│ │ │ ├── tour/ # Onboarding tour +│ │ │ └── ui/ # Base UI components (shadcn) +│ │ ├── features/ # Feature modules +│ │ │ └── chat/ # Chat feature (controller, state, protocol) +│ │ ├── hooks/ # React custom hooks +│ │ ├── i18n/ # Internationalization +│ │ │ └── locales/ # Locale JSON files +│ │ ├── lib/ # Utility libraries +│ │ ├── routes/ # TanStack Router routes +│ │ │ ├── agent/ # Agent management page +│ │ │ └── channels/ # Channels management page +│ │ └── store/ # Jotai stores (state management) +│ └── public/ # Static assets +├── workspace/ +│ ├── memory/ # Agent memory templates +│ └── skills/ # Built-in skill definitions +│ ├── agent-browser/ # Browser automation skill +│ ├── github/ # GitHub integration skill +│ ├── hardware/ # Hardware control skill +│ ├── skill-creator/ # Skill creation helper +│ ├── summarize/ # Conversation summarization +│ ├── tmux/ # tmux session management +│ └── weather/ # Weather lookup +├── docs/ # Documentation (multi-language) +│ ├── design/ # Design documents +│ ├── zh/ # Chinese docs +│ ├── ja/ # Japanese docs +│ ├── pt-br/ # Portuguese (BR) docs +│ └── vi/ # Vietnamese docs +└── config/ # Example configuration files +``` + +## Directory Purposes + +**`cmd/`:** CLI binary entry points. Each subdirectory produces a separate binary. + +**`pkg/`:** Shared Go packages. All business logic lives here. Packages are designed for reuse across binaries. + +**`cmd/picoclaw/internal/`:** CLI subcommand implementations. Thin wrappers around `pkg/` packages with Cobra integration. + +**`web/backend/`:** Desktop launcher backend. Embeds frontend assets and provides HTTP API + gateway management. + +**`web/frontend/`:** React + TypeScript dashboard. Built with Vite, TanStack Router, shadcn/ui. Output embedded into Go binary. + +**`workspace/`:** Runtime workspace. Memory templates, skill definitions, agent-specific data. Copied to `~/.picoclaw/` on first run. + +## Key File Locations + +**Entry Points:** +- `cmd/picoclaw/main.go`: CLI root (Cobra-based subcommands) +- `web/backend/main.go`: Desktop launcher (HTTP server + system tray) +- `pkg/gateway/gateway.go`: Core gateway runtime (agent loops, channels, services) +- `cmd/picoclaw-launcher-tui/main.go`: Terminal UI launcher + +**Configuration:** +- `pkg/config/config.go`: Config loading and environment variable integration +- `pkg/config/config_struct.go`: Config type definitions +- `web/backend/launcherconfig/config.go`: Launcher-specific settings + +**Core Logic:** +- `pkg/agent/loop.go`: Main agent event loop +- `pkg/agent/instance.go`: Agent instance with provider, tools, sessions +- `pkg/agent/turn.go`: Turn execution (LLM call + tool loop) +- `pkg/agent/registry.go`: Multi-agent management +- `pkg/bus/bus.go`: Message bus (inbound/outbound/media/audio/voice) +- `pkg/channels/manager.go`: Channel lifecycle and message routing +- `pkg/channels/interfaces.go`: Capability interfaces (streaming, typing, reactions) +- `pkg/tools/registry.go`: Tool registration and execution + +**Routing & API:** +- `web/backend/api/router.go`: All API route registration +- `web/frontend/src/routes/`: TanStack Router route definitions +- `web/frontend/src/routeTree.gen.ts`: Auto-generated route tree + +**Testing:** +- Co-located `*_test.go` files alongside source files throughout `pkg/` and `cmd/` + +## Naming Conventions + +**Files:** +- Go: snake_case for test files (`context_budget_test.go`), CamelCase for implementation files (`context_budget.go`) +- TypeScript: kebab-case for components (`channel-config-fields.ts`), camelCase for hooks (`use-chat-models.ts`) +- Routes: kebab-case (`launcher-login.tsx`) + +**Directories:** +- Go packages: lowercase, single word where possible (`agent`, `channels`, `tools`) +- Frontend features: kebab-case (`features/chat/`) +- Component groups: kebab-case (`components/agent/hub/`) + +**Functions:** +- Go: PascalCase for exported, camelCase for unexported +- Structured logging: `InfoCF`, `ErrorCF`, `DebugCF` (component + field variants) + +## Where to Add New Code + +**New Messaging Channel:** +- Implementation: `pkg/channels//` (with `init.go` for self-registration) +- Register: Import with blank identifier in `pkg/gateway/gateway.go` +- Capability interfaces: `pkg/channels/interfaces.go` + +**New LLM Provider:** +- Implementation: `pkg/providers//` +- Factory: Register in `pkg/providers/factory.go` +- Test: `_test.go` + +**New Tool:** +- Implementation: `pkg/tools/.go` +- Register: Add to tool list in `pkg/gateway/gateway.go` or agent loop setup + +**New Skill:** +- Definition: `workspace/skills//` (SKILL.md + references) + +**New CLI Subcommand:** +- Implementation: `cmd/picoclaw/internal//` +- Register: Add to `cmd/picoclaw/main.go` command list + +**New API Endpoint:** +- Handler: `web/backend/api/.go` +- Route: Register in `web/backend/api/router.go` + +**New Frontend Page:** +- Route: `web/frontend/src/routes/.tsx` +- API client: `web/frontend/src/api/.ts` +- Hook: `web/frontend/src/hooks/use-.ts` + +## Special Directories + +**`web/backend/dist/`:** +- Purpose: Compiled frontend assets (embedded into Go binary) +- Generated: Yes (by `npm run build:backend`) +- Committed: Yes (for self-contained Go builds) + +**`workspace/`:** +- Purpose: Default skill and memory templates copied to user home on first run +- Generated: No (hand-authored) +- Committed: Yes + +**`docs/`:** +- Purpose: Project documentation in multiple languages +- Languages: English (root), zh, ja, pt-br, vi, my +- Committed: Yes + +--- + +*Structure analysis: 2026-04-10* diff --git a/.planning/codebase/TESTING.md b/.planning/codebase/TESTING.md new file mode 100644 index 000000000..b701a2b60 --- /dev/null +++ b/.planning/codebase/TESTING.md @@ -0,0 +1,169 @@ +# Testing Patterns + +**Analysis Date:** 2026-04-10 + +## Test Framework + +**Runner:** Go 标准 `testing` 包。 + +**Assertion libraries:** +- `github.com/stretchr/testify/assert` -- 非致命断言 +- `github.com/stretchr/testify/require` -- 致命断言 + +**无外部测试框架** -- 纯 `testing` + `testify`。 + +## 运行测试 + +```bash +make test # 运行所有测试 +make check # deps + fmt + vet + test +go test -run TestName -v ./pkg/session/ # 运行单个测试 +go test -bench=. -benchmem -run='^$' ./... # 仅运行 benchmark +cd web && make test # Web backend 测试 +go test -tags goolm,stdjson ./... # CI 测试命令 +``` + +**必需的 Build tags:** `goolm,stdjson` + +## 测试文件组织 + +- **位置:** 与源码同目录同包(非 `_test` 包),`.golangci.yaml` 第25行禁用了 `testpackage` +- **示例:** `pkg/seahorse/store_test.go` (`package seahorse`), `cmd/picoclaw/internal/auth/command_test.go` (`package auth`) +- **命名:** `*_test.go` 后缀 +- **数量:** 约 240 个测试文件 + +## 测试结构模式 + +**表驱动测试** (标准模式): +```go +func TestShouldEnableLauncherFileLogging(t *testing.T) { + tests := []struct { + name string + enableConsole bool + debug bool + want bool + }{ + {name: "gui mode", enableConsole: false, debug: false, want: true}, + {name: "console mode", enableConsole: true, debug: false, want: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := fn(tt.enableConsole, tt.debug); got != tt.want { + t.Fatalf("... = %t, want %t", got, tt.want) + } + }) + } +} +``` + +**Cobra 命令测试** (`cmd/picoclaw/internal/*/command_test.go`): +```go +func TestNewAuthCommand(t *testing.T) { + cmd := NewAuthCommand() + require.NotNil(t, cmd) + assert.Equal(t, "auth", cmd.Use) + allowedCommands := []string{"login", "logout", "status", "models", "weixin", "wecom"} + subcommands := cmd.Commands() + assert.Len(t, subcommands, len(allowedCommands)) + for _, subcmd := range subcommands { + found := slices.Contains(allowedCommands, subcmd.Name()) + assert.True(t, found, "unexpected subcommand %q", subcmd.Name()) + } +} +``` + +**HTTP 处理器测试** (`web/backend/api/*_test.go`): +```go +mux := http.NewServeMux() +RegisterLauncherAuthRoutes(mux, LauncherAuthRouteOpts{DashboardToken: tok, SessionCookie: sess}) +t.Run("status_unauthenticated", func(t *testing.T) { + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/auth/status", nil)) + if rec.Code != http.StatusOK { t.Fatalf("status code = %d", rec.Code) } +}) +``` + +**数据库测试** (`pkg/seahorse/store_test.go`): +```go +func openTestStore(t *testing.T) *Store { + t.Helper() + db := openTestDB(t) + if err := runSchema(db); err != nil { t.Fatalf("migration: %v", err) } + return &Store{db: db} +} +``` + +## Mock 策略 + +**手写 mock 结构体**,无 gomock/mockgen/testify/mock。 + +**`pkg/tools/subagent_tool_test.go`:** +```go +type MockLLMProvider struct { lastOptions map[string]any } +func (m *MockLLMProvider) Chat(ctx, messages, tools, model, options) (*providers.LLMResponse, error) { + m.lastOptions = options + return &providers.LLMResponse{Content: "Task completed"}, nil +} +``` + +**回调函数 mock** (`pkg/seahorse/short_compaction_test.go`): +```go +var mockCompleteFn CompleteFn = func(ctx, prompt, opts) (string, error) { + return "Mock summary of the conversation segment.", nil +} +``` + +**Mock 对象:** LLM providers, CompleteFn, 时间 (`l.now = func() time.Time { return t0 }`) +**不 Mock 对象:** 数据库 (使用 in-memory SQLite), HTTP handlers (`httptest`), 文件系统 (`t.TempDir()`) + +## Fixtures 和测试数据 + +- 内存数据库: `openTestDB(t)` + `runSchema(db)`, 每个测试独立 +- 环境变量: `t.Setenv()` 自动清理 +- Testdata: `pkg/channels/telegram/testdata/md2_all_formats.txt` +- 测试常量: 硬编码在测试文件中 + +## Benchmark 测试 + +**位置:** `cmd/membench/` 和 `pkg/seahorse/short_bench_test.go` + +**模式:** +```go +func newBenchStore(b *testing.B) (*Store, func()) { + b.Helper() + db, err := sql.Open("sqlite", ":memory:") + if err != nil { b.Fatalf("open test db: %v", err) } + if err := runSchema(db); err != nil { db.Close(); b.Fatalf("migration: %v", err) } + return &Store{db: db}, func() { db.Close() } +} +func BenchmarkIngest_SingleMessage(b *testing.B) { + s, cleanup := newBenchStore(b); defer cleanup() + b.ResetTimer() + for i := 0; i < b.N; i++ { s.AddMessage(ctx, convID, "user", "Test", 15) } +} +``` + +**运行:** `go test -bench=. -benchmem -run='^$' ./pkg/seahorse/` + +## CI 配置 + +**PR workflow** (`.github/workflows/pr.yml`): +- Lint: `golangci-lint-action@v9` (v2.10.1), `--build-tags=goolm,stdjson` +- 安全检查: `govulncheck` +- 测试: `go test -tags goolm,stdjson ./...` +- 所有任务在 `ubuntu-latest` 上运行 + +**无覆盖率报告** -- CI 中没有 codecov 或 `-coverprofile` + +## 通用模式 + +- `t.Helper()` 在测试工具函数中 +- `t.TempDir()` 用于临时目录 +- `t.Fatalf()` 致命错误, `t.Errorf()` 断言失败 +- OS 特定跳过: `t.Skip("user environment variables only apply on Linux")` +- 并发测试: 直接操作互斥锁 (`manager.mu.Lock()`) +- 错误测试: 检查 `err == nil` 或 `result.IsError` + +--- + +*Testing analysis: 2026-04-10*