yao/agent/llm/providers/README.md

LLM Providers Architecture (New)

Overview

This directory contains LLM provider implementations using the Capability Adapters pattern. The new architecture separates API format handling from capability handling.

Architecture Design

┌─────────────────────────────────────────────────┐
│           LLM Provider (API Format)             │
│  - OpenAI-compatible                            │
│  - Claude (TODO)                                │
│  - Custom (TODO)                                │
└──────────────┬──────────────────────────────────┘
               │
               ↓
┌─────────────────────────────────────────────────┐
│        Capability Adapters (Modular)            │
│  - ToolCallAdapter    (native or prompt eng.)   │
│  - VisionAdapter      (native or removal)       │
│  - AudioAdapter       (native or removal)       │
│  - ReasoningAdapter   (o1/R1/GPT-Think)         │
└─────────────────────────────────────────────────┘

Key Concepts

1. Provider = API Format

Providers handle the API communication format:

• OpenAI-compatible API (/v1/chat/completions)
• Claude API (TODO)
• Custom API formats (TODO)

2. Adapters = Capabilities

Adapters handle model capabilities independently:

• ToolCallAdapter: Tool calling (native or prompt engineering)
• VisionAdapter: Image input (native or removal/conversion)
• AudioAdapter: Audio input (native or removal/conversion)
• ReasoningAdapter: Reasoning content (o1/DeepSeek R1/GPT-4o thinking)

Provider Selection

// factory.go
func SelectProvider(conn connector.Connector, options *context.CompletionOptions) (LLM, error) {
    apiFormat := DetectAPIFormat(conn)
    
    switch apiFormat {
    case "openai":
        // Adapters automatically configured based on capabilities
        return openai.New(conn, options.Capabilities), nil
    case "claude":
        return claude.New(conn, options.Capabilities), nil
    default:
        return openai.New(conn, options.Capabilities), nil
    }
}

Directory Structure

providers/
├── factory.go          # Provider selection based on API format
├── base/               # Common functionality
│   └── base.go
├── openai/             # OpenAI-compatible API provider
│   └── openai.go       # Includes adapter integration
└── README.md           # This file

../adapters/            # Capability adapters (separate package)
├── adapter.go          # Base interface
├── toolcall.go         # Tool calling adapter
├── vision.go           # Vision adapter
├── audio.go            # Audio adapter
└── reasoning.go        # Reasoning adapter

OpenAI Provider

The OpenAI provider supports all capabilities through adapters:

type Provider struct {
    *base.Provider
    adapters []adapters.CapabilityAdapter
}

func New(conn connector.Connector, capabilities *context.ModelCapabilities) *Provider {
    return &Provider{
        Provider: base.NewProvider(conn, capabilities),
        adapters: buildAdapters(capabilities), // Auto-configured
    }
}

Adapter Pipeline

Preprocessing (before API call):

Messages → ToolCallAdapter → VisionAdapter → AudioAdapter → API Request

Streaming (during API call):

API Chunk → ReasoningAdapter → ToolCallAdapter → Output

Postprocessing (after API call):

API Response → All Adapters → Final Response

Model Examples

Full-Featured Model (GPT-4o)

# connectors.yml
gpt-4o:
  vision: true
  tool_calls: true
  audio: true
  reasoning: false

Adapters created:

• ToolCallAdapter(native=true)
• VisionAdapter(native=true)
• AudioAdapter(native=true)

Reasoning Model with Tools (OpenAI o1)

o1-preview:
  reasoning: true
  tool_calls: true

Adapters created:

• ToolCallAdapter(native=true)
• ReasoningAdapter(format=openai-o1)

Reasoning Model without Tools (DeepSeek R1)

deepseek-reasoner:
  reasoning: true
  tool_calls: false

Adapters created:

• ToolCallAdapter(native=false) → Uses prompt engineering
• ReasoningAdapter(format=deepseek-r1)

Legacy Model (GPT-3.5-instruct)

gpt-3.5-turbo-instruct:
  tool_calls: false
  vision: false
  audio: false

Adapters created:

• ToolCallAdapter(native=false) → Prompt engineering
• VisionAdapter(native=false) → Removes images
• AudioAdapter(native=false) → Removes audio

Capability Adapters

ToolCallAdapter

When native=true:

• Passes tool definitions to API
• Parses structured tool_calls from response

When native=false:

• Injects tool schemas into system prompt
• Extracts tool calls from text response (JSON parsing)

VisionAdapter

When native=true:

• Passes image URLs/data directly to API

When native=false:

• Removes image content from messages
• Optionally converts to text descriptions

AudioAdapter

When native=true:

• Passes audio data directly to API

When native=false:

• Removes audio content from messages
• Optionally converts to text transcriptions

ReasoningAdapter

Handles different reasoning formats:

OpenAI o1 (reasoning_content field):

{
  "delta": {
    "reasoning_content": "Let me think...",
    "content": "The answer is 42"
  }
}

DeepSeek R1 (may have different format):

{
  "delta": {
    "content": "<think>Let me think...</think>The answer is 42"
  }
}

GPT-4o thinking (future):

{
  "delta": {
    "thinking": "Let me think...",
    "content": "The answer is 42"
  }
}

Adding New Capabilities

1. Create new adapter in ../adapters/:

   type NewCapabilityAdapter struct {
       *BaseAdapter
       nativeSupport bool
   }
   

2. Implement CapabilityAdapter interface

3. Add to buildAdapters() in openai/openai.go:

   if cap.NewCapability != nil {
       result = append(result, adapters.NewNewCapabilityAdapter(*cap.NewCapability))
   }
   

Adding New API Format Provider

1. Create new directory: providers/newapi/

2. Implement LLM interface:

   type Provider struct {
       *base.Provider
       adapters []adapters.CapabilityAdapter
   }
   
   func (p *Provider) Stream(...) (*CompletionResponse, error) {
       // Apply adapter preprocessing
       // Make API call
       // Apply adapter postprocessing
   }
   

3. Update factory.go:

   case "newapi":
       return newapi.New(conn, options.Capabilities), nil
   

Benefits of New Architecture

1. Separation of Concerns:

   • Providers handle API format
   • Adapters handle capabilities

2. Code Reuse:

   • Same adapters work across different providers
   • No duplication of capability logic

3. Easy Extension:

   • Add new capability = add one adapter
   • Add new API = add one provider

4. Flexible Combinations:

   • Any provider can use any adapter combination
   • Capabilities are composable

5. Clear Responsibility:

   • Each adapter handles exactly one capability dimension
   • Easy to test and maintain

Testing Strategy

Unit Tests (per adapter)

• Test preprocessing logic
• Test postprocessing logic
• Test stream chunk processing

Integration Tests (per provider)

• Test with different adapter combinations
• Test full request/response flow
• Test error handling

End-to-End Tests

• Test real API calls with different models
• Verify capability detection
• Verify adapter selection

Migration Notes

Old Architecture → New Architecture

Before:

reasoning.Provider  → Reasoning models (o1, R1)
openai.Provider     → Full-featured models (GPT-4o)
legacy.Provider     → Old models (GPT-3)

After:

openai.Provider + adapters → ALL models

The same OpenAI provider now handles all cases through different adapter combinations.