Merge pull request #104 from hobbyistlabs-coder/bolt-web-fetch-optimization-9650873766949108786

⚡ Bolt: Optimize web_fetch HTML detection to prevent large string allocations

.jules/bolt.md

@@ -11,3 +11,7 @@
 ## 2025-03-20 - String Operations Fast Paths and Avoiding Double Searches
 **Learning:** When trying to optimize `strings.ToLower`, ensure you don't introduce regressions with byte-to-rune casting on UTF-8 strings. Also, `strings.Contains(s, sub)` literally calls `strings.Index(s, sub)` under the hood. Using `strings.Contains` followed immediately by `strings.Index` to extract the position is an anti-pattern that searches the string twice, undermining the intended performance optimization.
 **Action:** Always prefer a single `strings.Index` call over `Contains`+`Index`. Stick to one single optimization per PR to reduce risk and review burden.
+
+## 2025-03-25 - Efficient HTTP Response Prefix Checking
+**Learning:** Using `strings.ToLower(string(body))` on large HTTP response payloads (which can be megabytes in size) to check for a small case-insensitive prefix (like `<html` or `<!doctype`) causes massive memory allocation, large garbage collection overhead, and $O(N)$ string iterations.
+**Action:** Use bounded byte slice checks combined with `bytes.EqualFold` (e.g., `bytes.EqualFold(body[:5], []byte("<html"))`) for large payloads. This makes the check $O(1)$ without any string allocations or full-body case conversions.

pkg/tools/web/fetch.go

@@ -1,6 +1,7 @@
 package web
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"errors"
@@ -158,8 +159,9 @@ func (t *WebFetchTool) Execute(ctx context.Context, args map[string]any) *tools.
 			text = string(body)
 			extractor = "raw"
 		}
-	} else if strings.Contains(contentType, "text/html") || len(body) > 0 &&
-		(strings.HasPrefix(string(body), "<!DOCTYPE") || strings.HasPrefix(strings.ToLower(string(body)), "<html")) {
+	} else if strings.Contains(contentType, "text/html") ||
+		(len(body) >= 9 && bytes.EqualFold(body[:9], []byte("<!doctype"))) ||
+		(len(body) >= 5 && bytes.EqualFold(body[:5], []byte("<html"))) {
 		text = t.extractText(string(body))
 		extractor = "text"
 	} else {