docs: add ETL framework proposal for system health visibility

Created `docs/ETL_PROPOSAL.md` outlining an ETL architecture focused
on Resource Utilization (memory allocation and goroutine counts) to
provide definitive Go/No-Go signals for new features. The proposal
details the Extract, Transform, and Load stages, along with top KPIs,
to maintain the core <10MB memory constraint.

Co-authored-by: hobbyistlabs-coder <267281733+hobbyistlabs-coder@users.noreply.github.com>

docs/ETL_PROPOSAL.md

@@ -0,0 +1,50 @@
+# ETL Framework for Ultimate Visibility: Resource Utilization Focus
+
+## 1. Introduction
+
+As PicoClaw targets an ultra-lightweight environment (<10MB RAM, $10 hardware), implementing a robust ETL (Extract, Transform, Load) framework for 'Ultimate Visibility' is critical. This framework will monitor system health and establish objective 'Go/No-Go' criteria for new feature implementations based on strict performance budgets.
+
+## 2. ETL Framework Architecture
+
+### Extract (Data Collection)
+- **Metrics Endpoint:** Extract system metrics (memory, active goroutines) exposed via the `/debug/vars` (expvar) endpoint within the health server (`pkg/health/server.go`).
+- **OpenTelemetry Hooks:** Leverage the existing OpenTelemetry gauges in `pkg/health/resource_tracker.go` (`sys.goroutines`, `sys.memory.alloc_mb`, `sys.memory.total_alloc_mb`, `sys.memory.sys_mb`).
+- **Structured Logs:** Extract JSON-formatted observability logs (e.g., Session Replay, Chain of Thought, tool inputs/outputs, and categorized errors) generated by the agent loop (`pkg/agent/loop.go`) and background summarizer tasks.
+
+### Transform (Data Normalization & Aggregation)
+- **Time-Series Normalization:** Convert disparate metrics and unstructured logs into a unified time-series format.
+- **Aggregation:** Calculate rolling averages and percentiles (e.g., p50, p95, p99) over 1-minute, 5-minute, and 15-minute windows for memory allocation and goroutine counts.
+- **Categorization:** Parse error logs to bucket failures into predefined categories: Model Failure, Infrastructure Failure, and Logic Failure.
+
+### Load (Storage & Visualization)
+- **Time-Series Database (TSDB):** Ingest normalized metrics into a lightweight TSDB like Prometheus or VictoriaMetrics optimized for low overhead.
+- **Log Analytics:** Forward structured JSON logs to a centralized log management system (e.g., Loki or Elasticsearch) for deep-dive session reviews and debugging.
+- **Dashboarding:** Expose these data points via a real-time dashboard (e.g., Grafana or a custom Bubble Tea TUI) to provide instant visibility into system health.
+
+## 3. Top KPIs for System Health
+
+To maintain the rigorous constraints of the PicoClaw architecture, the following KPIs are paramount:
+
+1.  **Memory Allocation (`sys.memory.alloc_mb`):** The absolute memory footprint of the application. Target: strictly <10MB.
+2.  **Active Goroutine Count (`sys.goroutines`):** Indicates the level of concurrency. Steadily increasing counts signify potential goroutine leaks.
+3.  **Agent Loop Latency (API Response Time):** Measures the end-to-end time taken to process a message and generate a response, directly impacting user experience.
+4.  **Error Categorization Rate:** The frequency of Model vs. Infrastructure vs. Logic failures per session.
+
+## 4. High-Impact Focus: Resource Utilization as a 'Go/No-Go' Signal
+
+**Focus Area:** Resource Utilization (Memory & Goroutines)
+
+In an environment where a $10 device with minimal RAM is the target deployment, absolute constraint adherence is non-negotiable. Tracking Resource Utilization provides a definitive, data-driven 'Go/No-Go' signal for any proposed feature.
+
+### The 'Go/No-Go' Process
+
+Suppose we propose implementing **Feature X** (e.g., Real-time Multi-Agent Orchestration or a new Web Automation Tool).
+
+1.  **Baseline Measurement:** Before Feature X, the ETL pipeline establishes a baseline: `sys.memory.alloc_mb` idles at 4MB and peaks at 8MB during typical agent loops. `sys.goroutines` averages 15 and returns to baseline after a request.
+2.  **Staging Deployment:** Feature X is deployed to a staging environment mirroring production constraints.
+3.  **Stress Testing:** The system is subjected to simulated production loads while the ETL framework actively tracks the KPIs.
+4.  **Signal Evaluation:**
+    *   **No-Go Signal:** If the ETL dashboard reveals that Feature X causes `sys.memory.alloc_mb` to consistently exceed the 10MB budget (e.g., peaking at 14MB), or if `sys.goroutines` steadily climbs without dropping back to baseline (indicating a leak), Feature X receives a strict "No-Go". The implementation must be rolled back or heavily optimized.
+    *   **Go Signal:** If Feature X operates effectively while `sys.memory.alloc_mb` remains comfortably under 10MB (e.g., peaking at 9.5MB) and `sys.goroutines` behaves predictably, the feature receives a "Go" signal for production release.
+
+By relying on this rigid, metric-driven framework, we ensure that PicoClaw's core value proposition—ultra-efficiency—is never compromised by feature bloat.