Enhance Test Case Structure with Custom Assertions
- Introduced an `assert` field in the test case structure to allow for custom assertion rules, providing flexibility in output validation. - Defined various assertion types, including `equals`, `contains`, `not_contains`, `json_path`, `regex`, and `script`, to cater to different validation needs. - Updated the test runner to utilize the new assertion mechanism, replacing the previous expected output validation with a more robust asserter. - Enhanced documentation in DESIGN.md to include detailed examples and explanations of the new assertion capabilities, improving clarity for users.
This commit is contained in:
Max
parent
d21f9c3769
commit
9e4febf782
5 changed files with 1064 additions and 15 deletions
|
|
@ -357,13 +357,143 @@ Each line in the input file is a JSON object with the following structure:
|
|||
| ---------- | ------------------------------ | -------- | ---------------------------------------------------- |
|
||||
| `id` | string | Yes | Unique test case identifier (e.g., "T001") |
|
||||
| `input` | string \| Message \| []Message | Yes | Test input |
|
||||
| `expected` | any | No | Expected output for validation |
|
||||
| `expected` | any | No | Expected output for exact match validation |
|
||||
| `assert` | Assertion \| []Assertion | No | Custom assertion rules (see Assertions section) |
|
||||
| `user` | string | No | User ID for this test case (overridden by `-u` flag) |
|
||||
| `team` | string | No | Team ID for this test case (overridden by `-t` flag) |
|
||||
| `metadata` | map | No | Additional metadata for the test case |
|
||||
| `skip` | bool | No | Skip this test case |
|
||||
| `timeout` | string | No | Override timeout (e.g., "30s", "1m") |
|
||||
|
||||
### Assertions
|
||||
|
||||
The `assert` field allows flexible validation of agent output. If `assert` is defined, it takes precedence over `expected`.
|
||||
|
||||
#### Assertion Types
|
||||
|
||||
| Type | Description | Example |
|
||||
| -------------- | ----------------------------------------------- | ---------------------------------------------------------------- |
|
||||
| `equals` | Exact match (default if only `expected` is set) | `{"type": "equals", "value": {"need_search": false}}` |
|
||||
| `contains` | Output contains the expected string/value | `{"type": "contains", "value": "keyword"}` |
|
||||
| `not_contains` | Output does not contain the string/value | `{"type": "not_contains", "value": "error"}` |
|
||||
| `json_path` | Extract value using JSON path and compare | `{"type": "json_path", "path": "$.need_search", "value": false}` |
|
||||
| `regex` | Match output against regex pattern | `{"type": "regex", "value": "\\d{3}-\\d{4}"}` |
|
||||
| `type` | Check output type (string, object, array, etc.) | `{"type": "type", "value": "object"}` |
|
||||
| `script` | Run a custom assertion script | `{"type": "script", "script": "scripts.test.Assert"}` |
|
||||
|
||||
#### Assertion Structure
|
||||
|
||||
```typescript
|
||||
interface Assertion {
|
||||
type: string; // Assertion type (required)
|
||||
value?: any; // Expected value or pattern
|
||||
path?: string; // JSON path for json_path assertions
|
||||
script?: string; // Script name for script assertions
|
||||
message?: string; // Custom failure message
|
||||
negate?: boolean; // Invert the assertion result
|
||||
}
|
||||
```
|
||||
|
||||
#### Examples
|
||||
|
||||
**Simple contains check:**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T001",
|
||||
"input": "Hello",
|
||||
"assert": {
|
||||
"type": "contains",
|
||||
"value": "need_search"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**JSON path validation (for agents returning JSON):**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T002",
|
||||
"input": "What's the weather?",
|
||||
"assert": {
|
||||
"type": "json_path",
|
||||
"path": "$.need_search",
|
||||
"value": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Multiple assertions (all must pass):**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T003",
|
||||
"input": "Calculate 2+2",
|
||||
"assert": [
|
||||
{
|
||||
"type": "json_path",
|
||||
"path": "$.need_search",
|
||||
"value": false
|
||||
},
|
||||
{
|
||||
"type": "json_path",
|
||||
"path": "$.confidence",
|
||||
"value": 0.99
|
||||
},
|
||||
{
|
||||
"type": "not_contains",
|
||||
"value": "error"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Custom script assertion:**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T004",
|
||||
"input": "Complex test",
|
||||
"assert": {
|
||||
"type": "script",
|
||||
"script": "scripts.test.ValidateOutput"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The script receives `(output, input, expected)` and should return:
|
||||
|
||||
```typescript
|
||||
// Simple boolean
|
||||
return true; // or false
|
||||
|
||||
// Or detailed result
|
||||
return {
|
||||
pass: true,
|
||||
message: "Validation passed: output contains expected keywords",
|
||||
};
|
||||
```
|
||||
|
||||
**Negated assertion:**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T005",
|
||||
"input": "Hello",
|
||||
"assert": {
|
||||
"type": "contains",
|
||||
"value": "error",
|
||||
"negate": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### JSON Path Notes
|
||||
|
||||
- Supports simple dot-notation paths: `$.field.subfield` or `field.subfield`
|
||||
- Automatically extracts JSON from markdown code blocks (e.g., ` ```json ... ``` `)
|
||||
- Works with both string output and structured objects
|
||||
|
||||
### Environment Override Priority
|
||||
|
||||
The test environment (user/team) is determined by the following priority (highest first):
|
||||
|
|
|
|||
381
agent/test/README.md
Normal file
381
agent/test/README.md
Normal file
|
|
@ -0,0 +1,381 @@
|
|||
# Agent Test Framework
|
||||
|
||||
A testing framework for Yao AI agents with support for assertions, stability analysis, and CI integration.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Test with direct message (auto-detect agent from current directory)
|
||||
cd assistants/keyword
|
||||
yao agent test -i "Extract keywords from: AI and machine learning"
|
||||
|
||||
# Test with direct message (specify agent explicitly)
|
||||
yao agent test -i "Hello world" -n workers.system.keyword
|
||||
|
||||
# Test with JSONL file (auto-detect agent from path)
|
||||
yao agent test -i assistants/keyword/tests/inputs.jsonl
|
||||
|
||||
# Generate HTML report
|
||||
yao agent test -i tests/inputs.jsonl -o report.html
|
||||
|
||||
# Stability analysis (run each test 5 times)
|
||||
yao agent test -i tests/inputs.jsonl --runs 5
|
||||
```
|
||||
|
||||
## Input Modes
|
||||
|
||||
The `-i` flag supports two input modes:
|
||||
|
||||
### 1. JSONL File Mode
|
||||
|
||||
Load test cases from a file:
|
||||
|
||||
```bash
|
||||
yao agent test -i tests/inputs.jsonl
|
||||
```
|
||||
|
||||
Agent is auto-detected by traversing up from the input file to find `package.yao`.
|
||||
|
||||
### 2. Direct Message Mode
|
||||
|
||||
Test with a single message:
|
||||
|
||||
```bash
|
||||
# Auto-detect agent from current working directory
|
||||
cd assistants/keyword
|
||||
yao agent test -i "Extract keywords from this text"
|
||||
|
||||
# Or specify agent explicitly
|
||||
yao agent test -i "Hello" -n workers.system.keyword
|
||||
```
|
||||
|
||||
Output is printed to stdout (or saved to `-o` if specified).
|
||||
|
||||
## Command Line Options
|
||||
|
||||
| Flag | Description | Default |
|
||||
| ------------- | ---------------------------------------- | -------------------------- |
|
||||
| `-i` | Input: JSONL file path or direct message | (required) |
|
||||
| `-o` | Output file path | `output-{timestamp}.jsonl` |
|
||||
| `-n` | Agent ID (optional, auto-detected) | auto-detect |
|
||||
| `-c` | Override connector | agent default |
|
||||
| `-u` | Test user ID | `test-user` |
|
||||
| `-t` | Test team ID | `test-team` |
|
||||
| `-r` | Reporter agent ID | built-in |
|
||||
| `--runs` | Runs per test (stability analysis) | 1 |
|
||||
| `--timeout` | Timeout per test | 5m |
|
||||
| `--parallel` | Parallel test cases | 1 |
|
||||
| `-v` | Verbose output | false |
|
||||
| `--fail-fast` | Stop on first failure | false |
|
||||
|
||||
## Agent Resolution
|
||||
|
||||
The agent is resolved in the following priority order:
|
||||
|
||||
1. **Explicit `-n` flag**: `yao agent test -i "msg" -n my.agent`
|
||||
2. **Path-based detection**: Traverse up from input file to find `package.yao`
|
||||
3. **Current directory**: For direct message mode, look for `package.yao` in cwd
|
||||
|
||||
Example directory structure:
|
||||
|
||||
```
|
||||
assistants/workers/system/keyword/
|
||||
├── package.yao <- Agent definition (auto-detected)
|
||||
├── prompts.yml
|
||||
├── src/
|
||||
│ └── index.ts
|
||||
└── tests/
|
||||
└── inputs.jsonl <- Input file
|
||||
```
|
||||
|
||||
## Input Format (JSONL)
|
||||
|
||||
Each line is a JSON object:
|
||||
|
||||
```jsonl
|
||||
{"id": "T001", "input": "Simple text"}
|
||||
{"id": "T002", "input": {"role": "user", "content": "Message with role"}}
|
||||
{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]}
|
||||
{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}}
|
||||
{"id": "T005", "input": "Skip this", "skip": true}
|
||||
```
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
| ---------- | ------------------------------ | -------- | ------------------------------ |
|
||||
| `id` | string | Yes | Test case ID |
|
||||
| `input` | string \| Message \| []Message | Yes | Test input |
|
||||
| `assert` | Assertion \| []Assertion | No | Assertion rules |
|
||||
| `expected` | any | No | Expected output (exact match) |
|
||||
| `user` | string | No | Override user ID |
|
||||
| `team` | string | No | Override team ID |
|
||||
| `timeout` | string | No | Override timeout (e.g., "30s") |
|
||||
| `skip` | bool | No | Skip this test |
|
||||
| `metadata` | map | No | Additional metadata |
|
||||
|
||||
### Input Types
|
||||
|
||||
| Type | Description | Example |
|
||||
| ----------- | -------------------- | ----------------------------------------------------- |
|
||||
| `string` | Simple text | `"Hello world"` |
|
||||
| `Message` | Single message | `{"role": "user", "content": "..."}` |
|
||||
| `[]Message` | Conversation history | `[{"role": "user", ...}, {"role": "assistant", ...}]` |
|
||||
|
||||
## Assertions
|
||||
|
||||
Use `assert` for flexible validation. If `assert` is defined, it takes precedence over `expected`.
|
||||
|
||||
### Assertion Types
|
||||
|
||||
| Type | Description | Example |
|
||||
| -------------- | ----------------------------- | --------------------------------------------------------- |
|
||||
| `equals` | Exact match | `{"type": "equals", "value": {"key": "val"}}` |
|
||||
| `contains` | Output contains value | `{"type": "contains", "value": "keyword"}` |
|
||||
| `not_contains` | Output does not contain value | `{"type": "not_contains", "value": "error"}` |
|
||||
| `json_path` | Extract JSON path and compare | `{"type": "json_path", "path": "$.field", "value": true}` |
|
||||
| `regex` | Match regex pattern | `{"type": "regex", "value": "\\d+"}` |
|
||||
| `type` | Check output type | `{"type": "type", "value": "object"}` |
|
||||
| `script` | Run custom assertion script | `{"type": "script", "script": "scripts.test.Check"}` |
|
||||
|
||||
### Assertion Options
|
||||
|
||||
| Field | Type | Description |
|
||||
| --------- | ------ | --------------------------- |
|
||||
| `type` | string | Assertion type (required) |
|
||||
| `value` | any | Expected value or pattern |
|
||||
| `path` | string | JSON path (for `json_path`) |
|
||||
| `script` | string | Script name (for `script`) |
|
||||
| `message` | string | Custom failure message |
|
||||
| `negate` | bool | Invert the result |
|
||||
|
||||
### Examples
|
||||
|
||||
**JSON path validation:**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T001",
|
||||
"input": "What's the weather?",
|
||||
"assert": {
|
||||
"type": "json_path",
|
||||
"path": "need_search",
|
||||
"value": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Multiple assertions (all must pass):**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T002",
|
||||
"input": "Hello",
|
||||
"assert": [
|
||||
{
|
||||
"type": "json_path",
|
||||
"path": "need_search",
|
||||
"value": false
|
||||
},
|
||||
{
|
||||
"type": "not_contains",
|
||||
"value": "error"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Custom script assertion:**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T003",
|
||||
"input": "Test",
|
||||
"assert": {
|
||||
"type": "script",
|
||||
"script": "scripts.test.Validate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Script receives `(output, input, expected)` and returns:
|
||||
|
||||
```javascript
|
||||
// Boolean
|
||||
return true;
|
||||
|
||||
// Or detailed result
|
||||
return { pass: true, message: "Validation passed" };
|
||||
```
|
||||
|
||||
**Negated assertion:**
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T004",
|
||||
"input": "Hello",
|
||||
"assert": {
|
||||
"type": "contains",
|
||||
"value": "error",
|
||||
"negate": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### JSON Path Notes
|
||||
|
||||
- Supports dot-notation: `$.field.subfield` or `field.subfield`
|
||||
- Auto-extracts JSON from markdown code blocks (` ```json ... ``` `)
|
||||
- Works with both string output and structured objects
|
||||
|
||||
## Output Formats
|
||||
|
||||
Determined by `-o` file extension:
|
||||
|
||||
| Extension | Format | Description |
|
||||
| --------- | -------- | ---------------------- |
|
||||
| `.jsonl` | JSONL | Streaming (default) |
|
||||
| `.json` | JSON | Complete structured |
|
||||
| `.md` | Markdown | Human-readable |
|
||||
| `.html` | HTML | Interactive web report |
|
||||
|
||||
### Default Output Path
|
||||
|
||||
When `-o` is not specified in file mode:
|
||||
|
||||
```
|
||||
{input_directory}/output-{timestamp}.jsonl
|
||||
```
|
||||
|
||||
Example: `tests/output-20241217100000.jsonl`
|
||||
|
||||
In direct message mode without `-o`, output is printed to stdout.
|
||||
|
||||
## Stability Analysis
|
||||
|
||||
Run each test multiple times to measure consistency:
|
||||
|
||||
```bash
|
||||
yao agent test -i tests/inputs.jsonl --runs 5 -o stability.json
|
||||
```
|
||||
|
||||
Output includes:
|
||||
|
||||
- Pass rate per test
|
||||
- Stability classification (stable, mostly_stable, unstable, highly_unstable)
|
||||
- Average/min/max duration
|
||||
- Standard deviation
|
||||
|
||||
### Stability Classification
|
||||
|
||||
| Pass Rate | Classification |
|
||||
| --------- | --------------- |
|
||||
| 100% | Stable |
|
||||
| 80-99% | Mostly Stable |
|
||||
| 50-79% | Unstable |
|
||||
| < 50% | Highly Unstable |
|
||||
|
||||
## Test Environment
|
||||
|
||||
The test framework creates a context with configurable environment:
|
||||
|
||||
| Setting | Flag | Default |
|
||||
| ---------- | ---- | ----------- |
|
||||
| User ID | `-u` | `test-user` |
|
||||
| Team ID | `-t` | `test-team` |
|
||||
| Locale | - | `en-us` |
|
||||
| ClientType | - | `test` |
|
||||
| ClientIP | - | `127.0.0.1` |
|
||||
|
||||
Priority: Command line flags > Test case fields > Defaults
|
||||
|
||||
## Custom Reporter Agent
|
||||
|
||||
Use `-r` to specify a custom agent for report generation:
|
||||
|
||||
```bash
|
||||
yao agent test -i tests/inputs.jsonl -r report.beautiful -o report.html
|
||||
```
|
||||
|
||||
The reporter agent receives:
|
||||
|
||||
```json
|
||||
{
|
||||
"report": { "summary": {...}, "results": [...] },
|
||||
"format": "html",
|
||||
"options": { "verbose": true }
|
||||
}
|
||||
```
|
||||
|
||||
## CI Integration
|
||||
|
||||
```bash
|
||||
# Exit code: 0 = all passed, 1 = failures
|
||||
yao agent test -i tests/inputs.jsonl -o results.jsonl --fail-fast
|
||||
|
||||
# Parse JSONL results
|
||||
cat results.jsonl | jq 'select(.type == "summary")'
|
||||
```
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
- name: Run Agent Tests
|
||||
run: |
|
||||
yao agent test -i assistants/keyword/tests/inputs.jsonl \
|
||||
-u ci-user -t ci-team \
|
||||
--runs 3 \
|
||||
-o report.json
|
||||
|
||||
- name: Check Stability
|
||||
run: |
|
||||
jq -e '.results | all(.pass_rate >= 80)' report.json
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Quick development test (auto-detect agent)
|
||||
cd assistants/keyword
|
||||
yao agent test -i "Extract keywords: AI and ML"
|
||||
|
||||
# Quick development test (specify agent)
|
||||
yao agent test -i "Hello" -n workers.system.keyword
|
||||
|
||||
# Full test suite with HTML report
|
||||
yao agent test -i tests/inputs.jsonl -o report.html -v
|
||||
|
||||
# Override connector
|
||||
yao agent test -i tests/inputs.jsonl -c openai.gpt4
|
||||
|
||||
# Stability analysis
|
||||
yao agent test -i tests/inputs.jsonl --runs 10 -o stability.json
|
||||
|
||||
# Parallel execution with timeout
|
||||
yao agent test -i tests/inputs.jsonl --parallel 4 --timeout 2m
|
||||
|
||||
# Custom test environment
|
||||
yao agent test -i tests/inputs.jsonl -u admin -t prod-team
|
||||
|
||||
# Custom reporter agent
|
||||
yao agent test -i tests/inputs.jsonl -r report.beautiful -o custom-report.md
|
||||
|
||||
# Full example with all options
|
||||
yao agent test -i tests/inputs.jsonl \
|
||||
-n keyword.agent \
|
||||
-c deepseek.v3 \
|
||||
-u test-user \
|
||||
-t test-team \
|
||||
--runs 3 \
|
||||
--timeout 10m \
|
||||
--parallel 4 \
|
||||
-r report.html \
|
||||
-o report.html
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Description |
|
||||
| ---- | --------------------------------------------------- |
|
||||
| 0 | All tests passed |
|
||||
| 1 | Tests failed, configuration error, or runtime error |
|
||||
480
agent/test/assert.go
Normal file
480
agent/test/assert.go
Normal file
|
|
@ -0,0 +1,480 @@
|
|||
package test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
jsoniter "github.com/json-iterator/go"
|
||||
"github.com/yaoapp/gou/process"
|
||||
)
|
||||
|
||||
// Asserter handles test assertions
|
||||
type Asserter struct{}
|
||||
|
||||
// NewAsserter creates a new asserter
|
||||
func NewAsserter() *Asserter {
|
||||
return &Asserter{}
|
||||
}
|
||||
|
||||
// Validate validates the output against the test case's assertions
|
||||
// Returns (passed, error message)
|
||||
func (a *Asserter) Validate(tc *Case, output interface{}) (bool, string) {
|
||||
// If assert is defined, use assertion rules
|
||||
if tc.Assert != nil {
|
||||
return a.validateAssertions(tc, output)
|
||||
}
|
||||
|
||||
// If expected is defined, use simple comparison
|
||||
if tc.Expected != nil {
|
||||
if validateOutput(output, tc.Expected) {
|
||||
return true, ""
|
||||
}
|
||||
return false, "output does not match expected"
|
||||
}
|
||||
|
||||
// No assertions defined - pass if we got output without error
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// validateAssertions validates output against assertion rules
|
||||
func (a *Asserter) validateAssertions(tc *Case, output interface{}) (bool, string) {
|
||||
assertions := a.parseAssertions(tc.Assert)
|
||||
if len(assertions) == 0 {
|
||||
return true, ""
|
||||
}
|
||||
|
||||
var failures []string
|
||||
for _, assertion := range assertions {
|
||||
result := a.evaluateAssertion(assertion, output, tc.Input)
|
||||
if !result.Passed {
|
||||
msg := result.Message
|
||||
if assertion.Message != "" {
|
||||
msg = assertion.Message
|
||||
}
|
||||
failures = append(failures, msg)
|
||||
}
|
||||
}
|
||||
|
||||
if len(failures) > 0 {
|
||||
return false, strings.Join(failures, "; ")
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// parseAssertions parses the assert field into a list of assertions
|
||||
func (a *Asserter) parseAssertions(assert interface{}) []*Assertion {
|
||||
if assert == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
var assertions []*Assertion
|
||||
|
||||
switch v := assert.(type) {
|
||||
case map[string]interface{}:
|
||||
// Single assertion object
|
||||
assertion := a.mapToAssertion(v)
|
||||
if assertion != nil {
|
||||
assertions = append(assertions, assertion)
|
||||
}
|
||||
|
||||
case []interface{}:
|
||||
// Array of assertions
|
||||
for _, item := range v {
|
||||
if m, ok := item.(map[string]interface{}); ok {
|
||||
assertion := a.mapToAssertion(m)
|
||||
if assertion != nil {
|
||||
assertions = append(assertions, assertion)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
case string:
|
||||
// Shorthand: just a type name (e.g., "contains")
|
||||
assertions = append(assertions, &Assertion{Type: v})
|
||||
}
|
||||
|
||||
return assertions
|
||||
}
|
||||
|
||||
// mapToAssertion converts a map to an Assertion
|
||||
func (a *Asserter) mapToAssertion(m map[string]interface{}) *Assertion {
|
||||
assertion := &Assertion{}
|
||||
|
||||
if t, ok := m["type"].(string); ok {
|
||||
assertion.Type = t
|
||||
}
|
||||
if v, ok := m["value"]; ok {
|
||||
assertion.Value = v
|
||||
}
|
||||
if p, ok := m["path"].(string); ok {
|
||||
assertion.Path = p
|
||||
}
|
||||
if s, ok := m["script"].(string); ok {
|
||||
assertion.Script = s
|
||||
}
|
||||
if msg, ok := m["message"].(string); ok {
|
||||
assertion.Message = msg
|
||||
}
|
||||
if n, ok := m["negate"].(bool); ok {
|
||||
assertion.Negate = n
|
||||
}
|
||||
|
||||
return assertion
|
||||
}
|
||||
|
||||
// evaluateAssertion evaluates a single assertion
|
||||
func (a *Asserter) evaluateAssertion(assertion *Assertion, output, input interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Expected: assertion.Value,
|
||||
}
|
||||
|
||||
switch assertion.Type {
|
||||
case "equals", "":
|
||||
result = a.assertEquals(assertion, output)
|
||||
case "contains":
|
||||
result = a.assertContains(assertion, output)
|
||||
case "not_contains":
|
||||
result = a.assertNotContains(assertion, output)
|
||||
case "json_path":
|
||||
result = a.assertJSONPath(assertion, output)
|
||||
case "regex":
|
||||
result = a.assertRegex(assertion, output)
|
||||
case "type":
|
||||
result = a.assertType(assertion, output)
|
||||
case "script":
|
||||
result = a.assertScript(assertion, output, input)
|
||||
default:
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("unknown assertion type: %s", assertion.Type)
|
||||
}
|
||||
|
||||
// Apply negate
|
||||
if assertion.Negate {
|
||||
result.Passed = !result.Passed
|
||||
if result.Passed {
|
||||
result.Message = "negated assertion passed"
|
||||
} else {
|
||||
result.Message = "negated: " + result.Message
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// assertEquals checks for exact equality
|
||||
func (a *Asserter) assertEquals(assertion *Assertion, output interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Actual: output,
|
||||
Expected: assertion.Value,
|
||||
}
|
||||
|
||||
if validateOutput(output, assertion.Value) {
|
||||
result.Passed = true
|
||||
result.Message = "values are equal"
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("expected %v, got %v", assertion.Value, output)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// assertContains checks if output contains the expected value
|
||||
func (a *Asserter) assertContains(assertion *Assertion, output interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Actual: output,
|
||||
Expected: assertion.Value,
|
||||
}
|
||||
|
||||
outputStr := a.toString(output)
|
||||
expectedStr := a.toString(assertion.Value)
|
||||
|
||||
if strings.Contains(outputStr, expectedStr) {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("output contains '%s'", expectedStr)
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("output does not contain '%s'", expectedStr)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// assertNotContains checks if output does not contain the expected value
|
||||
func (a *Asserter) assertNotContains(assertion *Assertion, output interface{}) *AssertionResult {
|
||||
result := a.assertContains(assertion, output)
|
||||
result.Passed = !result.Passed
|
||||
if result.Passed {
|
||||
result.Message = fmt.Sprintf("output does not contain '%s'", a.toString(assertion.Value))
|
||||
} else {
|
||||
result.Message = fmt.Sprintf("output should not contain '%s'", a.toString(assertion.Value))
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// assertJSONPath extracts a value using JSON path and compares
|
||||
func (a *Asserter) assertJSONPath(assertion *Assertion, output interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Expected: assertion.Value,
|
||||
}
|
||||
|
||||
// Convert output to JSON if needed
|
||||
var jsonData interface{}
|
||||
switch v := output.(type) {
|
||||
case string:
|
||||
// Try to parse as JSON
|
||||
if err := jsoniter.Unmarshal([]byte(v), &jsonData); err != nil {
|
||||
// Try to extract JSON from markdown code blocks
|
||||
extracted := extractJSONFromText(v)
|
||||
if extracted != nil {
|
||||
jsonData = extracted
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("output is not valid JSON: %s", err.Error())
|
||||
return result
|
||||
}
|
||||
}
|
||||
case map[string]interface{}, []interface{}:
|
||||
jsonData = v
|
||||
default:
|
||||
result.Passed = false
|
||||
result.Message = "output is not a JSON object or array"
|
||||
return result
|
||||
}
|
||||
|
||||
// Extract value using simple path (e.g., "$.need_search" or "need_search")
|
||||
path := strings.TrimPrefix(assertion.Path, "$.")
|
||||
actual := a.extractPath(jsonData, path)
|
||||
result.Actual = actual
|
||||
|
||||
if validateOutput(actual, assertion.Value) {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("path '%s' equals expected value", assertion.Path)
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("path '%s': expected %v, got %v", assertion.Path, assertion.Value, actual)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// extractPath extracts a value from JSON using a simple dot-notation path
|
||||
func (a *Asserter) extractPath(data interface{}, path string) interface{} {
|
||||
parts := strings.Split(path, ".")
|
||||
current := data
|
||||
|
||||
for _, part := range parts {
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
switch v := current.(type) {
|
||||
case map[string]interface{}:
|
||||
current = v[part]
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
return current
|
||||
}
|
||||
|
||||
// assertRegex checks if output matches a regex pattern
|
||||
func (a *Asserter) assertRegex(assertion *Assertion, output interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Actual: output,
|
||||
Expected: assertion.Value,
|
||||
}
|
||||
|
||||
pattern, ok := assertion.Value.(string)
|
||||
if !ok {
|
||||
result.Passed = false
|
||||
result.Message = "regex pattern must be a string"
|
||||
return result
|
||||
}
|
||||
|
||||
re, err := regexp.Compile(pattern)
|
||||
if err != nil {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("invalid regex pattern: %s", err.Error())
|
||||
return result
|
||||
}
|
||||
|
||||
outputStr := a.toString(output)
|
||||
if re.MatchString(outputStr) {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("output matches pattern '%s'", pattern)
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("output does not match pattern '%s'", pattern)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// assertType checks the type of the output
|
||||
func (a *Asserter) assertType(assertion *Assertion, output interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Actual: output,
|
||||
Expected: assertion.Value,
|
||||
}
|
||||
|
||||
expectedType, ok := assertion.Value.(string)
|
||||
if !ok {
|
||||
result.Passed = false
|
||||
result.Message = "type assertion value must be a string"
|
||||
return result
|
||||
}
|
||||
|
||||
actualType := a.getType(output)
|
||||
result.Actual = actualType
|
||||
|
||||
if actualType == expectedType {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("output is of type '%s'", expectedType)
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("expected type '%s', got '%s'", expectedType, actualType)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// getType returns the type name of a value
|
||||
func (a *Asserter) getType(v interface{}) string {
|
||||
if v == nil {
|
||||
return "null"
|
||||
}
|
||||
|
||||
switch v.(type) {
|
||||
case string:
|
||||
return "string"
|
||||
case float64, float32, int, int64, int32:
|
||||
return "number"
|
||||
case bool:
|
||||
return "boolean"
|
||||
case []interface{}:
|
||||
return "array"
|
||||
case map[string]interface{}:
|
||||
return "object"
|
||||
default:
|
||||
return fmt.Sprintf("%T", v)
|
||||
}
|
||||
}
|
||||
|
||||
// assertScript runs a custom assertion script
|
||||
func (a *Asserter) assertScript(assertion *Assertion, output, input interface{}) *AssertionResult {
|
||||
result := &AssertionResult{
|
||||
Assertion: assertion,
|
||||
Actual: output,
|
||||
}
|
||||
|
||||
if assertion.Script == "" {
|
||||
result.Passed = false
|
||||
result.Message = "script assertion requires a script name"
|
||||
return result
|
||||
}
|
||||
|
||||
// Build script arguments
|
||||
args := []interface{}{
|
||||
output,
|
||||
input,
|
||||
assertion.Value,
|
||||
}
|
||||
|
||||
// Run the script as a process
|
||||
p, err := process.Of(assertion.Script, args...)
|
||||
if err != nil {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("failed to create process: %s", err.Error())
|
||||
return result
|
||||
}
|
||||
|
||||
res, err := p.Exec()
|
||||
if err != nil {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("script execution failed: %s", err.Error())
|
||||
return result
|
||||
}
|
||||
|
||||
// Parse script result
|
||||
// Expected format: { "pass": bool, "message": string }
|
||||
switch v := res.(type) {
|
||||
case bool:
|
||||
result.Passed = v
|
||||
if v {
|
||||
result.Message = "script assertion passed"
|
||||
} else {
|
||||
result.Message = "script assertion failed"
|
||||
}
|
||||
|
||||
case map[string]interface{}:
|
||||
if pass, ok := v["pass"].(bool); ok {
|
||||
result.Passed = pass
|
||||
}
|
||||
if msg, ok := v["message"].(string); ok {
|
||||
result.Message = msg
|
||||
}
|
||||
|
||||
default:
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("script returned unexpected type: %T", res)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// toString converts a value to string for comparison
|
||||
func (a *Asserter) toString(v interface{}) string {
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
|
||||
switch val := v.(type) {
|
||||
case string:
|
||||
return val
|
||||
case []byte:
|
||||
return string(val)
|
||||
default:
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("%v", v)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
}
|
||||
|
||||
// extractJSONFromText tries to extract JSON from text (e.g., markdown code blocks)
|
||||
func extractJSONFromText(text string) interface{} {
|
||||
// Try to find JSON in code blocks
|
||||
patterns := []string{
|
||||
"```json\n",
|
||||
"```\n",
|
||||
}
|
||||
|
||||
for _, start := range patterns {
|
||||
if idx := strings.Index(text, start); idx >= 0 {
|
||||
text = text[idx+len(start):]
|
||||
if endIdx := strings.Index(text, "```"); endIdx >= 0 {
|
||||
text = text[:endIdx]
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Try to parse
|
||||
var result interface{}
|
||||
if err := jsoniter.Unmarshal([]byte(strings.TrimSpace(text)), &result); err == nil {
|
||||
return result
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
|
@ -64,8 +64,12 @@ func (r *Executor) RunDirect() (*Report, error) {
|
|||
ctx := NewTestContextFromOptions(chatID, agentInfo.ID, r.opts, tc)
|
||||
defer ctx.Release()
|
||||
|
||||
// Set connector override if specified
|
||||
opts := &context.Options{}
|
||||
// Set options: skip history (input already contains conversation), connector override
|
||||
opts := &context.Options{
|
||||
Skip: &context.Skip{
|
||||
History: true, // Skip history loading - input already contains full conversation
|
||||
},
|
||||
}
|
||||
if r.opts.Connector != "" {
|
||||
opts.Connector = r.opts.Connector
|
||||
}
|
||||
|
|
@ -293,8 +297,12 @@ func (r *Executor) runSingleTest(ast *assistant.Assistant, tc *Case, agentID str
|
|||
ctx := NewTestContextFromOptions(chatID, agentID, r.opts, tc)
|
||||
defer ctx.Release()
|
||||
|
||||
// Set connector override if specified
|
||||
opts := &context.Options{}
|
||||
// Set options: skip history (input already contains conversation), connector override
|
||||
opts := &context.Options{
|
||||
Skip: &context.Skip{
|
||||
History: true, // Skip history loading - input already contains full conversation
|
||||
},
|
||||
}
|
||||
if r.opts.Connector != "" {
|
||||
opts.Connector = r.opts.Connector
|
||||
}
|
||||
|
|
@ -332,17 +340,14 @@ func (r *Executor) runSingleTest(ast *assistant.Assistant, tc *Case, agentID str
|
|||
// Extract output
|
||||
result.Output = extractOutput(response)
|
||||
|
||||
// Validate result if expected is set
|
||||
if tc.Expected != nil {
|
||||
if validateOutput(result.Output, tc.Expected) {
|
||||
result.Status = StatusPassed
|
||||
} else {
|
||||
result.Status = StatusFailed
|
||||
result.Error = "output does not match expected"
|
||||
}
|
||||
} else {
|
||||
// No expected value - pass if no error
|
||||
// Validate result using asserter
|
||||
asserter := NewAsserter()
|
||||
passed, errMsg := asserter.Validate(tc, result.Output)
|
||||
if passed {
|
||||
result.Status = StatusPassed
|
||||
} else {
|
||||
result.Status = StatusFailed
|
||||
result.Error = errMsg
|
||||
}
|
||||
|
||||
r.output.TestResult(result.Status, duration)
|
||||
|
|
|
|||
|
|
@ -190,6 +190,11 @@ type Case struct {
|
|||
// If set, the actual output will be compared against this
|
||||
Expected interface{} `json:"expected,omitempty"`
|
||||
|
||||
// Assert defines custom assertion rules (optional)
|
||||
// If set, these rules will be used instead of simple expected comparison
|
||||
// Can be a single assertion or an array of assertions
|
||||
Assert interface{} `json:"assert,omitempty"`
|
||||
|
||||
// Environment (per-test case, can be overridden by command line flags)
|
||||
// ===============================
|
||||
|
||||
|
|
@ -210,6 +215,54 @@ type Case struct {
|
|||
Timeout string `json:"timeout,omitempty"`
|
||||
}
|
||||
|
||||
// Assertion represents a single assertion rule
|
||||
type Assertion struct {
|
||||
// Type is the assertion type:
|
||||
// - "equals": exact match (default if expected is set)
|
||||
// - "contains": output contains the expected string/value
|
||||
// - "not_contains": output does not contain the string/value
|
||||
// - "json_path": extract value using JSON path and compare
|
||||
// - "regex": match output against regex pattern
|
||||
// - "script": run a custom assertion script
|
||||
// - "type": check output type (string, object, array, number, boolean)
|
||||
// - "schema": validate against JSON schema
|
||||
Type string `json:"type"`
|
||||
|
||||
// Value is the expected value or pattern (depends on type)
|
||||
Value interface{} `json:"value,omitempty"`
|
||||
|
||||
// Path is the JSON path for json_path assertions (e.g., "$.need_search")
|
||||
Path string `json:"path,omitempty"`
|
||||
|
||||
// Script is the assertion script name for script assertions
|
||||
// The script receives (output, input, expected) and returns {pass: bool, message: string}
|
||||
Script string `json:"script,omitempty"`
|
||||
|
||||
// Message is a custom failure message
|
||||
Message string `json:"message,omitempty"`
|
||||
|
||||
// Negate inverts the assertion result
|
||||
Negate bool `json:"negate,omitempty"`
|
||||
}
|
||||
|
||||
// AssertionResult represents the result of an assertion
|
||||
type AssertionResult struct {
|
||||
// Passed indicates whether the assertion passed
|
||||
Passed bool `json:"passed"`
|
||||
|
||||
// Message describes the assertion result
|
||||
Message string `json:"message,omitempty"`
|
||||
|
||||
// Assertion is the original assertion that was evaluated
|
||||
Assertion *Assertion `json:"assertion,omitempty"`
|
||||
|
||||
// Actual is the actual value that was compared
|
||||
Actual interface{} `json:"actual,omitempty"`
|
||||
|
||||
// Expected is the expected value
|
||||
Expected interface{} `json:"expected,omitempty"`
|
||||
}
|
||||
|
||||
// GetEnvironment returns the effective test environment for this test case
|
||||
// Priority: command line flags > test case fields > defaults
|
||||
func (tc *Case) GetEnvironment(opts *Options) *Environment {
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue