From 340d86b187e1d27cecb34859e2ca5c76d1546d51 Mon Sep 17 00:00:00 2001 From: "google-labs-jules[bot]" <161369871+google-labs-jules[bot]@users.noreply.github.com> Date: Thu, 19 Mar 2026 18:18:47 +0000 Subject: [PATCH] docs: add ETL framework proposal for system health visibility Created `docs/ETL_PROPOSAL.md` outlining an ETL architecture focused on Resource Utilization (memory allocation and goroutine counts) to provide definitive Go/No-Go signals for new features. The proposal details the Extract, Transform, and Load stages, along with top KPIs, to maintain the core <10MB memory constraint. Co-authored-by: hobbyistlabs-coder <267281733+hobbyistlabs-coder@users.noreply.github.com> --- docs/ETL_PROPOSAL.md | 50 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 docs/ETL_PROPOSAL.md diff --git a/docs/ETL_PROPOSAL.md b/docs/ETL_PROPOSAL.md new file mode 100644 index 000000000..80d88abe7 --- /dev/null +++ b/docs/ETL_PROPOSAL.md @@ -0,0 +1,50 @@ +# ETL Framework for Ultimate Visibility: Resource Utilization Focus + +## 1. Introduction + +As PicoClaw targets an ultra-lightweight environment (<10MB RAM, $10 hardware), implementing a robust ETL (Extract, Transform, Load) framework for 'Ultimate Visibility' is critical. This framework will monitor system health and establish objective 'Go/No-Go' criteria for new feature implementations based on strict performance budgets. + +## 2. ETL Framework Architecture + +### Extract (Data Collection) +- **Metrics Endpoint:** Extract system metrics (memory, active goroutines) exposed via the `/debug/vars` (expvar) endpoint within the health server (`pkg/health/server.go`). +- **OpenTelemetry Hooks:** Leverage the existing OpenTelemetry gauges in `pkg/health/resource_tracker.go` (`sys.goroutines`, `sys.memory.alloc_mb`, `sys.memory.total_alloc_mb`, `sys.memory.sys_mb`). +- **Structured Logs:** Extract JSON-formatted observability logs (e.g., Session Replay, Chain of Thought, tool inputs/outputs, and categorized errors) generated by the agent loop (`pkg/agent/loop.go`) and background summarizer tasks. + +### Transform (Data Normalization & Aggregation) +- **Time-Series Normalization:** Convert disparate metrics and unstructured logs into a unified time-series format. +- **Aggregation:** Calculate rolling averages and percentiles (e.g., p50, p95, p99) over 1-minute, 5-minute, and 15-minute windows for memory allocation and goroutine counts. +- **Categorization:** Parse error logs to bucket failures into predefined categories: Model Failure, Infrastructure Failure, and Logic Failure. + +### Load (Storage & Visualization) +- **Time-Series Database (TSDB):** Ingest normalized metrics into a lightweight TSDB like Prometheus or VictoriaMetrics optimized for low overhead. +- **Log Analytics:** Forward structured JSON logs to a centralized log management system (e.g., Loki or Elasticsearch) for deep-dive session reviews and debugging. +- **Dashboarding:** Expose these data points via a real-time dashboard (e.g., Grafana or a custom Bubble Tea TUI) to provide instant visibility into system health. + +## 3. Top KPIs for System Health + +To maintain the rigorous constraints of the PicoClaw architecture, the following KPIs are paramount: + +1. **Memory Allocation (`sys.memory.alloc_mb`):** The absolute memory footprint of the application. Target: strictly <10MB. +2. **Active Goroutine Count (`sys.goroutines`):** Indicates the level of concurrency. Steadily increasing counts signify potential goroutine leaks. +3. **Agent Loop Latency (API Response Time):** Measures the end-to-end time taken to process a message and generate a response, directly impacting user experience. +4. **Error Categorization Rate:** The frequency of Model vs. Infrastructure vs. Logic failures per session. + +## 4. High-Impact Focus: Resource Utilization as a 'Go/No-Go' Signal + +**Focus Area:** Resource Utilization (Memory & Goroutines) + +In an environment where a $10 device with minimal RAM is the target deployment, absolute constraint adherence is non-negotiable. Tracking Resource Utilization provides a definitive, data-driven 'Go/No-Go' signal for any proposed feature. + +### The 'Go/No-Go' Process + +Suppose we propose implementing **Feature X** (e.g., Real-time Multi-Agent Orchestration or a new Web Automation Tool). + +1. **Baseline Measurement:** Before Feature X, the ETL pipeline establishes a baseline: `sys.memory.alloc_mb` idles at 4MB and peaks at 8MB during typical agent loops. `sys.goroutines` averages 15 and returns to baseline after a request. +2. **Staging Deployment:** Feature X is deployed to a staging environment mirroring production constraints. +3. **Stress Testing:** The system is subjected to simulated production loads while the ETL framework actively tracks the KPIs. +4. **Signal Evaluation:** + * **No-Go Signal:** If the ETL dashboard reveals that Feature X causes `sys.memory.alloc_mb` to consistently exceed the 10MB budget (e.g., peaking at 14MB), or if `sys.goroutines` steadily climbs without dropping back to baseline (indicating a leak), Feature X receives a strict "No-Go". The implementation must be rolled back or heavily optimized. + * **Go Signal:** If Feature X operates effectively while `sys.memory.alloc_mb` remains comfortably under 10MB (e.g., peaking at 9.5MB) and `sys.goroutines` behaves predictably, the feature receives a "Go" signal for production release. + +By relying on this rigid, metric-driven framework, we ensure that PicoClaw's core value proposition—ultra-efficiency—is never compromised by feature bloat.