* feat(seahorse): implement short-term memory engine of seahorse Add pkg/seahorse/ module implementing a SQLite-backed DAG-based summary hierarchy for context management, ported from lossless-claw's LCM design: - types.go + short_constants.go: core types (Message, Summary, Conversation, ContextItem) and configuration constants (fanout, token targets, thresholds) - migration.go: idempotent DB schema with FTS5 trigram tokenizer for CJK - store.go: full SQLite CRUD (conversations, messages, summaries DAG, context_items with ordinal gap numbering, FTS5 search) - short_engine.go: Engine lifecycle (NewEngine, Ingest, Assemble, Compact), session pattern filtering (ignore/stateless glob→regex compilation), per-session mutex via sync.Map - short_assembler.go: budget-aware context assembly with fresh tail protection (32 messages), oldest-first eviction, summary XML formatting, RebuildContextItems - short_compaction.go: leaf compaction (messages→summary) and condensed compaction (summaries→higher-level summary), 3-level LLM escalation, CompactUntilUnder for emergency overflow - short_retrieval.go: lookupByID, FTS5/LIKE search, recursive expand with token cap - context_seahorse.go: agent.ContextManager adapter, registered as "seahorse", provider↔seahorse message type conversion (ToolCalls, tool_result) * fix(seahorse): correct 3 adapter bugs in context management - TokenCount: use full message (Content+ToolCalls+Media) instead of Content-only - Empty Content: rebuild Content from tool_result Parts when stored empty - Duplicate summaries: summaries only in Summary field, not in History messages - Grep: fix SearchResult.Snippet→Content for summaries - Schema: fix FTS5 SQL uses VIRTUAL TABLE not TEMP TABLE - TestFTS5SQLConstants: verify FTS5 SQL syntax correctness - Test: fix flaky TestCompactLeaf * fix(agent): ingest steering messages into seahorse SQLite Steering messages were only persisted to session JSONL but not ingested into seahorse SQLite, causing them to be missing from context assembly. Added `ts.ingestMessage(turnCtx, al, pm)` call in the steering message injection block alongside the existing JSONL persistence. Test: TestSeahorseSteeringMessageIngested verifies steering messages appear in seahorse SQLite DB after being processed. * fix(seahorse): address 3 blocking bugs from code review - Fix resequenceContextItemsTx scan error handling (store.go:850) Changed `return err` to `return scanErr` to properly propagate scan errors instead of returning nil (which silently corrupts data) - Fix sql.NullString for INTEGER column (store.go:847) Changed `mid` from sql.NullString to sql.NullInt64 since message_id is INTEGER in schema. Removed unnecessary strconv.ParseInt call. - Fix compactCondensed fallback deleting non-candidate items Added ReplaceContextItemsWithSummary method for per-item deletion when candidates are not contiguous in ordinal space. Optimized to use range deletion when candidates are consecutive. * fix(seahorse): pass Budget to Compact for correct condensed threshold Issue #4 from PR review: When Budget was not passed to seahorse.Compact, it defaulted to `tokensBefore * 0.75`, making `tokensBefore > budget` always true and causing condensed compaction to trigger unnecessarily. Changes: - context_seahorse.go: Forward Budget from CompactRequest to CompactInput - loop.go: Pass Budget (ContextWindow) in all 3 Compact calls - Add test verifying condensed is skipped when tokens < threshold - Fix lint issues in store.go and store_test.go * fix(seahorse): add mutex for assembler lazy initialization Issue #5 from PR review: The check-then-create pattern for e.assembler was a data race when multiple goroutines called Assemble() concurrently: if e.assembler == nil { e.assembler = &Assembler{...} } Changes: - Add assemblerMu sync.Mutex to Engine struct - Add initAssemblerOnce() using double-checked locking (same pattern as initCompactionOnce) - Add TestAssemblerLazyInitRace to verify thread-safety * fix(seahorse): handle non-consecutive depths in selectShallowestCondensationCandidate Issue #8 from PR review: the loop iterated depth 0, 1, 2... assuming consecutive keys, but break when key was missing caused deeper depths to never be checked. Fix: collect all existing depth keys, sort, then iterate in order. * fix(seahorse): wrap DeleteMessagesAfterID and appendContextItems in transactions - DeleteMessagesAfterID: wrap all DELETE operations in a transaction for atomicity, remove redundant manual FTS delete (handled by trigger) - appendContextItems: use transaction to fix read-then-write race condition - Add GetMaxOrdinalTx and resolveItemTokenCountTx for transaction-scoped queries - Remove unused resolveItemTokenCount function Fixes PR review issues 6 and 7. * fix(seahorse): derive readable content from Parts and cap CompactUntilUnder iterations - Derive readable content from MessageParts in AddMessageWithParts so FTS5 indexing and summary formatting can access tool call information - formatMessagesForSummary and truncateSummary now fall back to Parts when Content is empty, fixing blank summaries for Part-based messages - Add MaxCompactIterations (20) to prevent CompactUntilUnder infinite loops; exceeded iterations are logged as warnings
144 lines
4.5 KiB
Go
144 lines
4.5 KiB
Go
package seahorse
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// =============================================================================
|
|
// Bug 1: formatMessagesForSummary ignores Parts
|
|
// - formatMessagesForSummary only reads m.Content, empty for Part-based messages
|
|
// - truncateSummary has same issue
|
|
// =============================================================================
|
|
|
|
func TestFormatMessagesForSummaryIncludesParts(t *testing.T) {
|
|
ts := time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC)
|
|
|
|
messages := []Message{
|
|
{ID: 1, Role: "user", Content: "hello world", CreatedAt: ts},
|
|
{
|
|
ID: 2,
|
|
Role: "assistant",
|
|
Content: "", // empty — real content is in Parts
|
|
Parts: []MessagePart{
|
|
{Type: "text", Text: "I will run a command"},
|
|
{Type: "tool_use", Name: "bash", Arguments: `{"command":"ls -la"}`, ToolCallID: "call_1"},
|
|
},
|
|
CreatedAt: ts.Add(time.Minute),
|
|
},
|
|
{
|
|
ID: 3,
|
|
Role: "tool",
|
|
Content: "", // empty — real content is in Parts
|
|
Parts: []MessagePart{
|
|
{Type: "tool_result", Text: "file1.txt\nfile2.txt", ToolCallID: "call_1"},
|
|
},
|
|
CreatedAt: ts.Add(2 * time.Minute),
|
|
},
|
|
}
|
|
|
|
result := formatMessagesForSummary(messages)
|
|
|
|
// Must contain the plain text message
|
|
if !contains(result, "hello world") {
|
|
t.Error("formatMessagesForSummary: missing plain text content")
|
|
}
|
|
|
|
// Must contain tool_use info (not blank)
|
|
if !contains(result, "bash") || !contains(result, "ls -la") {
|
|
t.Errorf("formatMessagesForSummary: tool_use info missing from Parts.\nGot:\n%s", result)
|
|
}
|
|
|
|
// Must contain tool_result info (not blank)
|
|
if !contains(result, "file1.txt") {
|
|
t.Errorf("formatMessagesForSummary: tool_result text missing from Parts.\nGot:\n%s", result)
|
|
}
|
|
}
|
|
|
|
func TestTruncateSummaryIncludesParts(t *testing.T) {
|
|
messages := []Message{
|
|
{ID: 1, Role: "user", Content: "run the tests", CreatedAt: time.Now()},
|
|
{
|
|
ID: 2,
|
|
Role: "assistant",
|
|
Content: "", // empty
|
|
Parts: []MessagePart{
|
|
{Type: "tool_use", Name: "bash", Arguments: `{"command":"go test ./..."}`, ToolCallID: "call_1"},
|
|
},
|
|
CreatedAt: time.Now(),
|
|
},
|
|
{
|
|
ID: 3,
|
|
Role: "tool",
|
|
Content: "", // empty
|
|
Parts: []MessagePart{
|
|
{Type: "tool_result", Text: "PASS\nok 3.2s", ToolCallID: "call_1"},
|
|
},
|
|
CreatedAt: time.Now(),
|
|
},
|
|
}
|
|
|
|
result := truncateSummary(messages)
|
|
|
|
// Must contain plain text
|
|
if !contains(result, "run the tests") {
|
|
t.Error("truncateSummary: missing plain text content")
|
|
}
|
|
|
|
// Must contain tool info from Parts (not blank)
|
|
if !contains(result, "bash") || !contains(result, "go test") {
|
|
t.Errorf("truncateSummary: tool_use info missing from Parts.\nGot:\n%s", result)
|
|
}
|
|
|
|
// Must contain tool_result from Parts
|
|
if !contains(result, "PASS") {
|
|
t.Errorf("truncateSummary: tool_result text missing from Parts.\nGot:\n%s", result)
|
|
}
|
|
}
|
|
|
|
// =============================================================================
|
|
// Bug 2: SearchMessages cannot find Part-based messages
|
|
// - FTS5 indexes empty content, LIKE queries empty content
|
|
// =============================================================================
|
|
|
|
func TestSearchMessagesFindsPartBasedMessages(t *testing.T) {
|
|
s := openTestStore(t)
|
|
ctx := context.Background()
|
|
|
|
conv, _ := s.GetOrCreateConversation(ctx, "agent:search-parts")
|
|
convID := conv.ConversationID
|
|
|
|
// Add a plain message (searchable)
|
|
s.AddMessage(ctx, convID, "user", "list the files please", 5)
|
|
|
|
// Add a Part-based message (tool_use) — currently NOT searchable
|
|
parts := []MessagePart{
|
|
{Type: "tool_use", Name: "bash", Arguments: `{"command":"grep -r TODO ."}`, ToolCallID: "call_1"},
|
|
}
|
|
s.AddMessageWithParts(ctx, convID, "assistant", parts, 10)
|
|
|
|
// Add a Part-based message (tool_result) — currently NOT searchable
|
|
resultParts := []MessagePart{
|
|
{Type: "tool_result", Text: "main.go:42: TODO fix this bug", ToolCallID: "call_1"},
|
|
}
|
|
s.AddMessageWithParts(ctx, convID, "tool", resultParts, 10)
|
|
|
|
// Search for "grep" — should find the tool_use message
|
|
results, err := s.SearchMessages(ctx, SearchInput{Pattern: "grep"})
|
|
if err != nil {
|
|
t.Fatalf("SearchMessages: %v", err)
|
|
}
|
|
if len(results) == 0 {
|
|
t.Error("SearchMessages: 'grep' not found — Part-based messages are invisible to search")
|
|
}
|
|
|
|
// Search for "TODO fix" — should find the tool_result message
|
|
results2, err := s.SearchMessages(ctx, SearchInput{Pattern: "TODO fix"})
|
|
if err != nil {
|
|
t.Fatalf("SearchMessages: %v", err)
|
|
}
|
|
if len(results2) == 0 {
|
|
t.Error("SearchMessages: 'TODO fix' not found — tool_result messages are invisible to search")
|
|
}
|
|
}
|