# Agent Test Framework

A testing framework for Yao AI agents with support for assertions, stability analysis, and CI integration.

## Quick Start

### Agent Tests

```bash
# Test with direct message (auto-detect agent from current directory)
cd assistants/keyword
yao agent test -i "Extract keywords from: AI and machine learning"

# Test with direct message (specify agent explicitly)
yao agent test -i "Hello world" -n workers.system.keyword

# Test with JSONL file (auto-detect agent from path)
yao agent test -i assistants/keyword/tests/inputs.jsonl

# Generate HTML report
yao agent test -i tests/inputs.jsonl -o report.html

# Stability analysis (run each test 5 times)
yao agent test -i tests/inputs.jsonl --runs 5
```

### Script Tests

```bash
# Test agent handler scripts (hooks, tools, setup functions)
yao agent test -i scripts.expense.setup -v

# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady" -v

# Run with custom context (authorization, metadata)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```

## Input Modes

The `-i` flag supports three input modes:

### 1. JSONL File Mode

Load test cases from a file:

```bash
yao agent test -i tests/inputs.jsonl
```

Agent is auto-detected by traversing up from the input file to find `package.yao`.

### 2. Direct Message Mode

Test with a single message:

```bash
# Auto-detect agent from current working directory
cd assistants/keyword
yao agent test -i "Extract keywords from this text"

# Or specify agent explicitly
yao agent test -i "Hello" -n workers.system.keyword
```

Output is printed to stdout (or saved to `-o` if specified).

### 3. Script Test Mode

Test agent handler scripts (hooks, tools, setup functions):

```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v

# Run specific tests with filtering
yao agent test -i scripts.expense.setup --run "TestSystemReady"

# Run with custom context
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```

Script test input format: `scripts.<assistant>.<module>` (e.g., `scripts.expense.setup` → `assistants/expense/src/setup_test.ts`).

**Writing Test Scripts:**

Test scripts should be placed alongside the source files with `_test.ts` or `_test.js` suffix:

```
assistants/expense/src/
├── setup.ts           # Source file
├── setup_test.ts      # Test file
├── tools.ts
└── tools_test.ts
```

Test functions must follow the naming convention `Test*` and accept `(t: testing.T, ctx: agent.Context)`:

```typescript
// assistants/expense/src/setup_test.ts
import { SystemReady } from "./setup";

// Test function signature: function Test*(t: testing.T, ctx: agent.Context)
export function TestSystemReady(t: testing.T, ctx: agent.Context) {
  const result = SystemReady(ctx);

  // Use t.assert for assertions
  t.assert.True(result.success, "SystemReady should succeed");
  t.assert.Equal(result.status, "ready", "Status should be ready");
  t.assert.NotNil(result.data, "Data should not be nil");
}

export function TestSystemReadyError(t: testing.T, ctx: agent.Context) {
  // Access context properties
  console.log("Testing with user:", ctx.authorized.user_id);

  // Test error handling
  const result = SystemReady(ctx);
  t.assert.False(result.error, "Should not have error");
}
```

**Available Assertions:**

| Method                           | Description                    |
| -------------------------------- | ------------------------------ |
| `t.assert.True(value, msg)`      | Assert value is true           |
| `t.assert.False(value, msg)`     | Assert value is false          |
| `t.assert.Equal(a, b, msg)`      | Assert a equals b              |
| `t.assert.NotEqual(a, b, msg)`   | Assert a not equals b          |
| `t.assert.Nil(value, msg)`       | Assert value is null/undefined |
| `t.assert.NotNil(value, msg)`    | Assert value is not nil        |
| `t.assert.Contains(s, sub, msg)` | Assert string contains substr  |
| `t.assert.Len(arr, n, msg)`      | Assert array/string length     |

**Test Control:**

| Method         | Description                  |
| -------------- | ---------------------------- |
| `t.Log(msg)`   | Log a message                |
| `t.Error(msg)` | Mark test as failed with msg |
| `t.Fatal(msg)` | Mark failed and stop test    |
| `t.Skip(msg)`  | Skip this test               |

## Command Line Options

| Flag          | Description                                        | Default                    |
| ------------- | -------------------------------------------------- | -------------------------- |
| `-i`          | Input: JSONL file path, message, or script ID      | (required)                 |
| `-o`          | Output file path                                   | `output-{timestamp}.jsonl` |
| `-n`          | Agent ID (optional, auto-detected)                 | auto-detect                |
| `-c`          | Override connector                                 | agent default              |
| `-u`          | Test user ID                                       | `test-user`                |
| `-t`          | Test team ID                                       | `test-team`                |
| `--ctx`       | Path to context JSON file for custom authorization | -                          |
| `-r`          | Reporter agent ID                                  | built-in                   |
| `--runs`      | Runs per test (stability analysis)                 | 1                          |
| `--run`       | Regex pattern to filter which tests to run         | -                          |
| `--timeout`   | Timeout per test                                   | 5m                         |
| `--parallel`  | Parallel test cases                                | 1                          |
| `-v`          | Verbose output                                     | false                      |
| `--fail-fast` | Stop on first failure                              | false                      |

## Agent Resolution

The agent is resolved in the following priority order:

1. **Explicit `-n` flag**: `yao agent test -i "msg" -n my.agent`
2. **Path-based detection**: Traverse up from input file to find `package.yao`
3. **Current directory**: For direct message mode, look for `package.yao` in cwd

Example directory structure:

```
assistants/workers/system/keyword/
├── package.yao          <- Agent definition (auto-detected)
├── prompts.yml
├── src/
│   └── index.ts
└── tests/
    └── inputs.jsonl     <- Input file
```

## Input Format (JSONL)

Each line is a JSON object:

```jsonl
{"id": "T001", "input": "Simple text"}
{"id": "T002", "input": {"role": "user", "content": "Message with role"}}
{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]}
{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}}
{"id": "T005", "input": "Skip this", "skip": true}
```

### Fields

| Field      | Type                           | Required | Description                    |
| ---------- | ------------------------------ | -------- | ------------------------------ |
| `id`       | string                         | Yes      | Test case ID                   |
| `input`    | string \| Message \| []Message | Yes      | Test input                     |
| `assert`   | Assertion \| []Assertion       | No       | Assertion rules                |
| `expected` | any                            | No       | Expected output (exact match)  |
| `user`     | string                         | No       | Override user ID               |
| `team`     | string                         | No       | Override team ID               |
| `options`  | Options                        | No       | Context options (see below)    |
| `timeout`  | string                         | No       | Override timeout (e.g., "30s") |
| `skip`     | bool                           | No       | Skip this test                 |
| `metadata` | map                            | No       | Additional metadata            |

### Options

The `options` field allows per-test-case configuration that maps to `context.Options`:

| Field                    | Type   | Description                                  |
| ------------------------ | ------ | -------------------------------------------- |
| `connector`              | string | Override connector (e.g., `"deepseek.v3"`)   |
| `mode`                   | string | Agent mode (default: `"chat"`)               |
| `search`                 | bool   | Enable/disable search mode (default: `true`) |
| `disable_global_prompts` | bool   | Temporarily disable global prompts           |
| `metadata`               | map    | Custom data passed to hooks (e.g., scenario) |
| `skip`                   | object | Skip configuration (see below)               |

#### Options.skip

| Field     | Type | Description             |
| --------- | ---- | ----------------------- |
| `history` | bool | Skip history loading    |
| `trace`   | bool | Skip trace logging      |
| `output`  | bool | Skip output to client   |
| `keyword` | bool | Skip keyword extraction |
| `search`  | bool | Skip auto search        |

**Example with options:**

```jsonl
{
  "id": "T001",
  "input": "Query users with status active",
  "options": {
    "connector": "deepseek.v3",
    "metadata": {
      "scenario": "filter"
    },
    "skip": {
      "trace": true
    }
  },
  "assert": {
    "type": "json_path",
    "path": "from",
    "value": "users"
  }
}
```

**Using metadata for hook scenarios:**

The `options.metadata` field is passed to agent hooks. For example, a Create Hook can read `options.metadata.scenario` to select different prompt presets:

```jsonl
{"id": "T001", "input": "...", "options": {"metadata": {"scenario": "aggregation"}}}
{"id": "T002", "input": "...", "options": {"metadata": {"scenario": "join"}}}
```

### Input Types

| Type        | Description          | Example                                               |
| ----------- | -------------------- | ----------------------------------------------------- |
| `string`    | Simple text          | `"Hello world"`                                       |
| `Message`   | Single message       | `{"role": "user", "content": "..."}`                  |
| `[]Message` | Conversation history | `[{"role": "user", ...}, {"role": "assistant", ...}]` |

### File Attachments

Test inputs support file attachments (images, audio, documents) using the `file://` protocol. Files are loaded and converted to appropriate formats for the LLM.

**Supported file types:**

| Type   | Extensions                                                             | Format                         |
| ------ | ---------------------------------------------------------------------- | ------------------------------ |
| Image  | `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.bmp`                       | Base64 data URL in `image_url` |
| Audio  | `.wav`, `.mp3`, `.flac`, `.ogg`, `.m4a`                                | Base64 in `input_audio`        |
| Doc    | `.pdf`, `.doc`, `.docx`, `.xls`, `.xlsx`, `.txt`, `.csv`, `.json`      | Base64 data URL in `file`      |
| Source | `.yao`, `.ts`, `.js`, `.go`, `.py`, `.rs`, `.java`, `.sql`, `.yaml`... | Base64 data URL in `file`      |

**File path resolution:**

- **Relative paths**: Resolved relative to the JSONL input file's directory (for file mode) or current working directory (for message mode)
- **Absolute paths**: Used as-is

**Example with image attachment:**

```jsonl
{
  "id": "T001",
  "input": {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "Please analyze this invoice"
      },
      {
        "type": "image",
        "source": "file://fixtures/invoice.jpg"
      }
    ]
  },
  "assert": {
    "type": "contains",
    "value": "amount"
  }
}
```

**Example with multiple attachments:**

```jsonl
{
  "id": "T002",
  "input": {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "Process these receipts"
      },
      {
        "type": "image",
        "source": "file://fixtures/receipt1.png"
      },
      {
        "type": "image",
        "source": "file://fixtures/receipt2.png"
      },
      {
        "type": "file",
        "source": "file://fixtures/policy.pdf",
        "name": "expense_policy.pdf"
      }
    ]
  }
}
```

**Example with audio:**

```jsonl
{
  "id": "T003",
  "input": {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "Transcribe this audio"
      },
      {
        "type": "audio",
        "source": "file://fixtures/recording.wav"
      }
    ]
  }
}
```

**Content part types:**

| Type        | Fields                                  | Description                      |
| ----------- | --------------------------------------- | -------------------------------- |
| `text`      | `text`                                  | Text content                     |
| `image`     | `source` (file://) or `url`             | Image attachment                 |
| `image_url` | `image_url: {url, detail?}`             | Direct image URL (OpenAI format) |
| `audio`     | `source` (file://) or `data`, `format`  | Audio attachment                 |
| `file`      | `source` (file://) or `url`, `filename` | Document attachment              |
| `data`      | `data: {sources: [...]}`                | Data source references           |

## Assertions

Use `assert` for flexible validation. If `assert` is defined, it takes precedence over `expected`.

### Assertion Types

| Type           | Description                   | Example                                                   |
| -------------- | ----------------------------- | --------------------------------------------------------- |
| `equals`       | Exact match                   | `{"type": "equals", "value": {"key": "val"}}`             |
| `contains`     | Output contains value         | `{"type": "contains", "value": "keyword"}`                |
| `not_contains` | Output does not contain value | `{"type": "not_contains", "value": "error"}`              |
| `json_path`    | Extract JSON path and compare | `{"type": "json_path", "path": "$.field", "value": true}` |
| `regex`        | Match regex pattern           | `{"type": "regex", "value": "\\d+"}`                      |
| `type`         | Check output type             | `{"type": "type", "value": "object"}`                     |
| `script`       | Run custom assertion script   | `{"type": "script", "script": "scripts.test.Check"}`      |

### Assertion Options

| Field     | Type   | Description                 |
| --------- | ------ | --------------------------- |
| `type`    | string | Assertion type (required)   |
| `value`   | any    | Expected value or pattern   |
| `path`    | string | JSON path (for `json_path`) |
| `script`  | string | Script name (for `script`)  |
| `message` | string | Custom failure message      |
| `negate`  | bool   | Invert the result           |

### Examples

**JSON path validation:**

```jsonl
{
  "id": "T001",
  "input": "What's the weather?",
  "assert": {
    "type": "json_path",
    "path": "need_search",
    "value": true
  }
}
```

**Multiple assertions (all must pass):**

```jsonl
{
  "id": "T002",
  "input": "Hello",
  "assert": [
    {
      "type": "json_path",
      "path": "need_search",
      "value": false
    },
    {
      "type": "not_contains",
      "value": "error"
    }
  ]
}
```

**Custom script assertion:**

```jsonl
{
  "id": "T003",
  "input": "Test",
  "assert": {
    "type": "script",
    "script": "scripts.test.Validate"
  }
}
```

Script receives `(output, input, expected)` and returns:

```javascript
// Boolean
return true;

// Or detailed result
return { pass: true, message: "Validation passed" };
```

**Negated assertion:**

```jsonl
{
  "id": "T004",
  "input": "Hello",
  "assert": {
    "type": "contains",
    "value": "error",
    "negate": true
  }
}
```

### JSON Path Notes

- Supports dot-notation: `$.field.subfield` or `field.subfield`
- Supports array indexing: `field[0]`, `field[0].subfield`, `field[0].nested[1]`
- Supports multiple expected values (OR logic): `"value": ["a", "b"]` - passes if actual matches any
- Auto-extracts JSON from markdown code blocks (` ```json ... ``` `)
- Works with both string output and structured objects

**Array index examples:**

```jsonl
{"id": "T001", "assert": {"type": "json_path", "path": "wheres[0].like", "value": "%test%"}}
{"id": "T002", "assert": {"type": "json_path", "path": "wheres[0].in[0]", "value": "pending"}}
{"id": "T003", "assert": {"type": "json_path", "path": "joins[0].from", "value": "users"}}
{"id": "T004", "assert": {"type": "json_path", "path": "groups[0]", "value": "category"}}
```

**Multiple expected values (OR logic):**

```jsonl
{
  "id": "T005",
  "assert": {
    "type": "json_path",
    "path": "error",
    "value": [
      "missing_schema",
      "missing_query"
    ]
  }
}
```

This passes if `error` equals either `"missing_schema"` or `"missing_query"`.

## Output Formats

Determined by `-o` file extension:

| Extension | Format   | Description            |
| --------- | -------- | ---------------------- |
| `.jsonl`  | JSONL    | Streaming (default)    |
| `.json`   | JSON     | Complete structured    |
| `.md`     | Markdown | Human-readable         |
| `.html`   | HTML     | Interactive web report |

### Default Output Path

When `-o` is not specified in file mode:

```
{input_directory}/output-{timestamp}.jsonl
```

Example: `tests/output-20241217100000.jsonl`

In direct message mode without `-o`, output is printed to stdout.

## Stability Analysis

Run each test multiple times to measure consistency:

```bash
yao agent test -i tests/inputs.jsonl --runs 5 -o stability.json
```

Output includes:

- Pass rate per test
- Stability classification (stable, mostly_stable, unstable, highly_unstable)
- Average/min/max duration
- Standard deviation

### Stability Classification

| Pass Rate | Classification  |
| --------- | --------------- |
| 100%      | Stable          |
| 80-99%    | Mostly Stable   |
| 50-79%    | Unstable        |
| < 50%     | Highly Unstable |

## Test Environment

The test framework creates a context with configurable environment:

| Setting    | Flag | Default     |
| ---------- | ---- | ----------- |
| User ID    | `-u` | `test-user` |
| Team ID    | `-t` | `test-team` |
| Locale     | -    | `en-us`     |
| ClientType | -    | `test`      |
| ClientIP   | -    | `127.0.0.1` |

Priority: Command line flags > Test case fields > Defaults

## Custom Reporter Agent

Use `-r` to specify a custom agent for report generation:

```bash
yao agent test -i tests/inputs.jsonl -r report.beautiful -o report.html
```

The reporter agent receives:

```json
{
  "report": { "summary": {...}, "results": [...] },
  "format": "html",
  "options": { "verbose": true }
}
```

## CI Integration

```bash
# Exit code: 0 = all passed, 1 = failures
yao agent test -i tests/inputs.jsonl -o results.jsonl --fail-fast

# Parse JSONL results
cat results.jsonl | jq 'select(.type == "summary")'

# Run script tests
yao agent test -i scripts.expense.setup --fail-fast
```

### GitHub Actions Example

```yaml
- name: Run Agent Tests
  run: |
    yao agent test -i assistants/keyword/tests/inputs.jsonl \
      -u ci-user -t ci-team \
      --runs 3 \
      -o report.json

- name: Run Script Tests
  run: |
    yao agent test -i scripts.expense.setup -v
    yao agent test -i scripts.expense.tools -v

- name: Run Script Tests with Custom Context
  run: |
    yao agent test -i scripts.expense.setup \
      --ctx tests/context.json \
      --run "TestSystem.*" \
      -v

- name: Check Stability
  run: |
    jq -e '.results | all(.pass_rate >= 80)' report.json
```

## Examples

### Agent Tests

```bash
# Quick development test (auto-detect agent)
cd assistants/keyword
yao agent test -i "Extract keywords: AI and ML"

# Quick development test (specify agent)
yao agent test -i "Hello" -n workers.system.keyword

# Full test suite with HTML report
yao agent test -i tests/inputs.jsonl -o report.html -v

# Override connector
yao agent test -i tests/inputs.jsonl -c openai.gpt4

# Stability analysis
yao agent test -i tests/inputs.jsonl --runs 10 -o stability.json

# Parallel execution with timeout
yao agent test -i tests/inputs.jsonl --parallel 4 --timeout 2m

# Custom test environment
yao agent test -i tests/inputs.jsonl -u admin -t prod-team

# Custom reporter agent
yao agent test -i tests/inputs.jsonl -r report.beautiful -o custom-report.md

# Full example with all options
yao agent test -i tests/inputs.jsonl \
  -n keyword.agent \
  -c deepseek.v3 \
  -u test-user \
  -t test-team \
  --runs 3 \
  --timeout 10m \
  --parallel 4 \
  -r report.html \
  -o report.html
```

### Script Tests

```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v

# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady"

# Run tests matching a pattern
yao agent test -i scripts.expense.setup --run "TestSystem.*" -v

# Run with custom context (authorization, metadata, etc.)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v

# Run with specific user/team
yao agent test -i scripts.expense.setup -u admin -t ops-team -v

# Combine options
yao agent test -i scripts.expense.setup \
  --ctx tests/context.json \
  --run "TestSystem.*" \
  --timeout 30s \
  -v
```

## Exit Codes

| Code | Description                                         |
| ---- | --------------------------------------------------- |
| 0    | All tests passed                                    |
| 1    | Tests failed, configuration error, or runtime error |