Enhance Test Case Structure with Custom Assertions
- Introduced an `assert` field in the test case structure to allow for custom assertion rules, providing flexibility in output validation. - Defined various assertion types, including `equals`, `contains`, `not_contains`, `json_path`, `regex`, and `script`, to cater to different validation needs. - Updated the test runner to utilize the new assertion mechanism, replacing the previous expected output validation with a more robust asserter. - Enhanced documentation in DESIGN.md to include detailed examples and explanations of the new assertion capabilities, improving clarity for users.
This commit is contained in:
Max
parent
d21f9c3769
commit
9e4febf782
5 changed files with 1064 additions and 15 deletions
|
|
@ -357,13 +357,143 @@ Each line in the input file is a JSON object with the following structure:
|
||||||
| ---------- | ------------------------------ | -------- | ---------------------------------------------------- |
|
| ---------- | ------------------------------ | -------- | ---------------------------------------------------- |
|
||||||
| `id` | string | Yes | Unique test case identifier (e.g., "T001") |
|
| `id` | string | Yes | Unique test case identifier (e.g., "T001") |
|
||||||
| `input` | string \| Message \| []Message | Yes | Test input |
|
| `input` | string \| Message \| []Message | Yes | Test input |
|
||||||
| `expected` | any | No | Expected output for validation |
|
| `expected` | any | No | Expected output for exact match validation |
|
||||||
|
| `assert` | Assertion \| []Assertion | No | Custom assertion rules (see Assertions section) |
|
||||||
| `user` | string | No | User ID for this test case (overridden by `-u` flag) |
|
| `user` | string | No | User ID for this test case (overridden by `-u` flag) |
|
||||||
| `team` | string | No | Team ID for this test case (overridden by `-t` flag) |
|
| `team` | string | No | Team ID for this test case (overridden by `-t` flag) |
|
||||||
| `metadata` | map | No | Additional metadata for the test case |
|
| `metadata` | map | No | Additional metadata for the test case |
|
||||||
| `skip` | bool | No | Skip this test case |
|
| `skip` | bool | No | Skip this test case |
|
||||||
| `timeout` | string | No | Override timeout (e.g., "30s", "1m") |
|
| `timeout` | string | No | Override timeout (e.g., "30s", "1m") |
|
||||||
|
|
||||||
|
### Assertions
|
||||||
|
|
||||||
|
The `assert` field allows flexible validation of agent output. If `assert` is defined, it takes precedence over `expected`.
|
||||||
|
|
||||||
|
#### Assertion Types
|
||||||
|
|
||||||
|
| Type | Description | Example |
|
||||||
|
| -------------- | ----------------------------------------------- | ---------------------------------------------------------------- |
|
||||||
|
| `equals` | Exact match (default if only `expected` is set) | `{"type": "equals", "value": {"need_search": false}}` |
|
||||||
|
| `contains` | Output contains the expected string/value | `{"type": "contains", "value": "keyword"}` |
|
||||||
|
| `not_contains` | Output does not contain the string/value | `{"type": "not_contains", "value": "error"}` |
|
||||||
|
| `json_path` | Extract value using JSON path and compare | `{"type": "json_path", "path": "$.need_search", "value": false}` |
|
||||||
|
| `regex` | Match output against regex pattern | `{"type": "regex", "value": "\\d{3}-\\d{4}"}` |
|
||||||
|
| `type` | Check output type (string, object, array, etc.) | `{"type": "type", "value": "object"}` |
|
||||||
|
| `script` | Run a custom assertion script | `{"type": "script", "script": "scripts.test.Assert"}` |
|
||||||
|
|
||||||
|
#### Assertion Structure
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface Assertion {
|
||||||
|
type: string; // Assertion type (required)
|
||||||
|
value?: any; // Expected value or pattern
|
||||||
|
path?: string; // JSON path for json_path assertions
|
||||||
|
script?: string; // Script name for script assertions
|
||||||
|
message?: string; // Custom failure message
|
||||||
|
negate?: boolean; // Invert the assertion result
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Examples
|
||||||
|
|
||||||
|
**Simple contains check:**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T001",
|
||||||
|
"input": "Hello",
|
||||||
|
"assert": {
|
||||||
|
"type": "contains",
|
||||||
|
"value": "need_search"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**JSON path validation (for agents returning JSON):**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T002",
|
||||||
|
"input": "What's the weather?",
|
||||||
|
"assert": {
|
||||||
|
"type": "json_path",
|
||||||
|
"path": "$.need_search",
|
||||||
|
"value": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Multiple assertions (all must pass):**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T003",
|
||||||
|
"input": "Calculate 2+2",
|
||||||
|
"assert": [
|
||||||
|
{
|
||||||
|
"type": "json_path",
|
||||||
|
"path": "$.need_search",
|
||||||
|
"value": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "json_path",
|
||||||
|
"path": "$.confidence",
|
||||||
|
"value": 0.99
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "not_contains",
|
||||||
|
"value": "error"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Custom script assertion:**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T004",
|
||||||
|
"input": "Complex test",
|
||||||
|
"assert": {
|
||||||
|
"type": "script",
|
||||||
|
"script": "scripts.test.ValidateOutput"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The script receives `(output, input, expected)` and should return:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Simple boolean
|
||||||
|
return true; // or false
|
||||||
|
|
||||||
|
// Or detailed result
|
||||||
|
return {
|
||||||
|
pass: true,
|
||||||
|
message: "Validation passed: output contains expected keywords",
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
**Negated assertion:**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T005",
|
||||||
|
"input": "Hello",
|
||||||
|
"assert": {
|
||||||
|
"type": "contains",
|
||||||
|
"value": "error",
|
||||||
|
"negate": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JSON Path Notes
|
||||||
|
|
||||||
|
- Supports simple dot-notation paths: `$.field.subfield` or `field.subfield`
|
||||||
|
- Automatically extracts JSON from markdown code blocks (e.g., ` ```json ... ``` `)
|
||||||
|
- Works with both string output and structured objects
|
||||||
|
|
||||||
### Environment Override Priority
|
### Environment Override Priority
|
||||||
|
|
||||||
The test environment (user/team) is determined by the following priority (highest first):
|
The test environment (user/team) is determined by the following priority (highest first):
|
||||||
|
|
|
||||||
381
agent/test/README.md
Normal file
381
agent/test/README.md
Normal file
|
|
@ -0,0 +1,381 @@
|
||||||
|
# Agent Test Framework
|
||||||
|
|
||||||
|
A testing framework for Yao AI agents with support for assertions, stability analysis, and CI integration.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test with direct message (auto-detect agent from current directory)
|
||||||
|
cd assistants/keyword
|
||||||
|
yao agent test -i "Extract keywords from: AI and machine learning"
|
||||||
|
|
||||||
|
# Test with direct message (specify agent explicitly)
|
||||||
|
yao agent test -i "Hello world" -n workers.system.keyword
|
||||||
|
|
||||||
|
# Test with JSONL file (auto-detect agent from path)
|
||||||
|
yao agent test -i assistants/keyword/tests/inputs.jsonl
|
||||||
|
|
||||||
|
# Generate HTML report
|
||||||
|
yao agent test -i tests/inputs.jsonl -o report.html
|
||||||
|
|
||||||
|
# Stability analysis (run each test 5 times)
|
||||||
|
yao agent test -i tests/inputs.jsonl --runs 5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Input Modes
|
||||||
|
|
||||||
|
The `-i` flag supports two input modes:
|
||||||
|
|
||||||
|
### 1. JSONL File Mode
|
||||||
|
|
||||||
|
Load test cases from a file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yao agent test -i tests/inputs.jsonl
|
||||||
|
```
|
||||||
|
|
||||||
|
Agent is auto-detected by traversing up from the input file to find `package.yao`.
|
||||||
|
|
||||||
|
### 2. Direct Message Mode
|
||||||
|
|
||||||
|
Test with a single message:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Auto-detect agent from current working directory
|
||||||
|
cd assistants/keyword
|
||||||
|
yao agent test -i "Extract keywords from this text"
|
||||||
|
|
||||||
|
# Or specify agent explicitly
|
||||||
|
yao agent test -i "Hello" -n workers.system.keyword
|
||||||
|
```
|
||||||
|
|
||||||
|
Output is printed to stdout (or saved to `-o` if specified).
|
||||||
|
|
||||||
|
## Command Line Options
|
||||||
|
|
||||||
|
| Flag | Description | Default |
|
||||||
|
| ------------- | ---------------------------------------- | -------------------------- |
|
||||||
|
| `-i` | Input: JSONL file path or direct message | (required) |
|
||||||
|
| `-o` | Output file path | `output-{timestamp}.jsonl` |
|
||||||
|
| `-n` | Agent ID (optional, auto-detected) | auto-detect |
|
||||||
|
| `-c` | Override connector | agent default |
|
||||||
|
| `-u` | Test user ID | `test-user` |
|
||||||
|
| `-t` | Test team ID | `test-team` |
|
||||||
|
| `-r` | Reporter agent ID | built-in |
|
||||||
|
| `--runs` | Runs per test (stability analysis) | 1 |
|
||||||
|
| `--timeout` | Timeout per test | 5m |
|
||||||
|
| `--parallel` | Parallel test cases | 1 |
|
||||||
|
| `-v` | Verbose output | false |
|
||||||
|
| `--fail-fast` | Stop on first failure | false |
|
||||||
|
|
||||||
|
## Agent Resolution
|
||||||
|
|
||||||
|
The agent is resolved in the following priority order:
|
||||||
|
|
||||||
|
1. **Explicit `-n` flag**: `yao agent test -i "msg" -n my.agent`
|
||||||
|
2. **Path-based detection**: Traverse up from input file to find `package.yao`
|
||||||
|
3. **Current directory**: For direct message mode, look for `package.yao` in cwd
|
||||||
|
|
||||||
|
Example directory structure:
|
||||||
|
|
||||||
|
```
|
||||||
|
assistants/workers/system/keyword/
|
||||||
|
├── package.yao <- Agent definition (auto-detected)
|
||||||
|
├── prompts.yml
|
||||||
|
├── src/
|
||||||
|
│ └── index.ts
|
||||||
|
└── tests/
|
||||||
|
└── inputs.jsonl <- Input file
|
||||||
|
```
|
||||||
|
|
||||||
|
## Input Format (JSONL)
|
||||||
|
|
||||||
|
Each line is a JSON object:
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{"id": "T001", "input": "Simple text"}
|
||||||
|
{"id": "T002", "input": {"role": "user", "content": "Message with role"}}
|
||||||
|
{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]}
|
||||||
|
{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}}
|
||||||
|
{"id": "T005", "input": "Skip this", "skip": true}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fields
|
||||||
|
|
||||||
|
| Field | Type | Required | Description |
|
||||||
|
| ---------- | ------------------------------ | -------- | ------------------------------ |
|
||||||
|
| `id` | string | Yes | Test case ID |
|
||||||
|
| `input` | string \| Message \| []Message | Yes | Test input |
|
||||||
|
| `assert` | Assertion \| []Assertion | No | Assertion rules |
|
||||||
|
| `expected` | any | No | Expected output (exact match) |
|
||||||
|
| `user` | string | No | Override user ID |
|
||||||
|
| `team` | string | No | Override team ID |
|
||||||
|
| `timeout` | string | No | Override timeout (e.g., "30s") |
|
||||||
|
| `skip` | bool | No | Skip this test |
|
||||||
|
| `metadata` | map | No | Additional metadata |
|
||||||
|
|
||||||
|
### Input Types
|
||||||
|
|
||||||
|
| Type | Description | Example |
|
||||||
|
| ----------- | -------------------- | ----------------------------------------------------- |
|
||||||
|
| `string` | Simple text | `"Hello world"` |
|
||||||
|
| `Message` | Single message | `{"role": "user", "content": "..."}` |
|
||||||
|
| `[]Message` | Conversation history | `[{"role": "user", ...}, {"role": "assistant", ...}]` |
|
||||||
|
|
||||||
|
## Assertions
|
||||||
|
|
||||||
|
Use `assert` for flexible validation. If `assert` is defined, it takes precedence over `expected`.
|
||||||
|
|
||||||
|
### Assertion Types
|
||||||
|
|
||||||
|
| Type | Description | Example |
|
||||||
|
| -------------- | ----------------------------- | --------------------------------------------------------- |
|
||||||
|
| `equals` | Exact match | `{"type": "equals", "value": {"key": "val"}}` |
|
||||||
|
| `contains` | Output contains value | `{"type": "contains", "value": "keyword"}` |
|
||||||
|
| `not_contains` | Output does not contain value | `{"type": "not_contains", "value": "error"}` |
|
||||||
|
| `json_path` | Extract JSON path and compare | `{"type": "json_path", "path": "$.field", "value": true}` |
|
||||||
|
| `regex` | Match regex pattern | `{"type": "regex", "value": "\\d+"}` |
|
||||||
|
| `type` | Check output type | `{"type": "type", "value": "object"}` |
|
||||||
|
| `script` | Run custom assertion script | `{"type": "script", "script": "scripts.test.Check"}` |
|
||||||
|
|
||||||
|
### Assertion Options
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
| --------- | ------ | --------------------------- |
|
||||||
|
| `type` | string | Assertion type (required) |
|
||||||
|
| `value` | any | Expected value or pattern |
|
||||||
|
| `path` | string | JSON path (for `json_path`) |
|
||||||
|
| `script` | string | Script name (for `script`) |
|
||||||
|
| `message` | string | Custom failure message |
|
||||||
|
| `negate` | bool | Invert the result |
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
**JSON path validation:**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T001",
|
||||||
|
"input": "What's the weather?",
|
||||||
|
"assert": {
|
||||||
|
"type": "json_path",
|
||||||
|
"path": "need_search",
|
||||||
|
"value": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Multiple assertions (all must pass):**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T002",
|
||||||
|
"input": "Hello",
|
||||||
|
"assert": [
|
||||||
|
{
|
||||||
|
"type": "json_path",
|
||||||
|
"path": "need_search",
|
||||||
|
"value": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "not_contains",
|
||||||
|
"value": "error"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Custom script assertion:**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T003",
|
||||||
|
"input": "Test",
|
||||||
|
"assert": {
|
||||||
|
"type": "script",
|
||||||
|
"script": "scripts.test.Validate"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Script receives `(output, input, expected)` and returns:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Boolean
|
||||||
|
return true;
|
||||||
|
|
||||||
|
// Or detailed result
|
||||||
|
return { pass: true, message: "Validation passed" };
|
||||||
|
```
|
||||||
|
|
||||||
|
**Negated assertion:**
|
||||||
|
|
||||||
|
```jsonl
|
||||||
|
{
|
||||||
|
"id": "T004",
|
||||||
|
"input": "Hello",
|
||||||
|
"assert": {
|
||||||
|
"type": "contains",
|
||||||
|
"value": "error",
|
||||||
|
"negate": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### JSON Path Notes
|
||||||
|
|
||||||
|
- Supports dot-notation: `$.field.subfield` or `field.subfield`
|
||||||
|
- Auto-extracts JSON from markdown code blocks (` ```json ... ``` `)
|
||||||
|
- Works with both string output and structured objects
|
||||||
|
|
||||||
|
## Output Formats
|
||||||
|
|
||||||
|
Determined by `-o` file extension:
|
||||||
|
|
||||||
|
| Extension | Format | Description |
|
||||||
|
| --------- | -------- | ---------------------- |
|
||||||
|
| `.jsonl` | JSONL | Streaming (default) |
|
||||||
|
| `.json` | JSON | Complete structured |
|
||||||
|
| `.md` | Markdown | Human-readable |
|
||||||
|
| `.html` | HTML | Interactive web report |
|
||||||
|
|
||||||
|
### Default Output Path
|
||||||
|
|
||||||
|
When `-o` is not specified in file mode:
|
||||||
|
|
||||||
|
```
|
||||||
|
{input_directory}/output-{timestamp}.jsonl
|
||||||
|
```
|
||||||
|
|
||||||
|
Example: `tests/output-20241217100000.jsonl`
|
||||||
|
|
||||||
|
In direct message mode without `-o`, output is printed to stdout.
|
||||||
|
|
||||||
|
## Stability Analysis
|
||||||
|
|
||||||
|
Run each test multiple times to measure consistency:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yao agent test -i tests/inputs.jsonl --runs 5 -o stability.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Output includes:
|
||||||
|
|
||||||
|
- Pass rate per test
|
||||||
|
- Stability classification (stable, mostly_stable, unstable, highly_unstable)
|
||||||
|
- Average/min/max duration
|
||||||
|
- Standard deviation
|
||||||
|
|
||||||
|
### Stability Classification
|
||||||
|
|
||||||
|
| Pass Rate | Classification |
|
||||||
|
| --------- | --------------- |
|
||||||
|
| 100% | Stable |
|
||||||
|
| 80-99% | Mostly Stable |
|
||||||
|
| 50-79% | Unstable |
|
||||||
|
| < 50% | Highly Unstable |
|
||||||
|
|
||||||
|
## Test Environment
|
||||||
|
|
||||||
|
The test framework creates a context with configurable environment:
|
||||||
|
|
||||||
|
| Setting | Flag | Default |
|
||||||
|
| ---------- | ---- | ----------- |
|
||||||
|
| User ID | `-u` | `test-user` |
|
||||||
|
| Team ID | `-t` | `test-team` |
|
||||||
|
| Locale | - | `en-us` |
|
||||||
|
| ClientType | - | `test` |
|
||||||
|
| ClientIP | - | `127.0.0.1` |
|
||||||
|
|
||||||
|
Priority: Command line flags > Test case fields > Defaults
|
||||||
|
|
||||||
|
## Custom Reporter Agent
|
||||||
|
|
||||||
|
Use `-r` to specify a custom agent for report generation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yao agent test -i tests/inputs.jsonl -r report.beautiful -o report.html
|
||||||
|
```
|
||||||
|
|
||||||
|
The reporter agent receives:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"report": { "summary": {...}, "results": [...] },
|
||||||
|
"format": "html",
|
||||||
|
"options": { "verbose": true }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## CI Integration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Exit code: 0 = all passed, 1 = failures
|
||||||
|
yao agent test -i tests/inputs.jsonl -o results.jsonl --fail-fast
|
||||||
|
|
||||||
|
# Parse JSONL results
|
||||||
|
cat results.jsonl | jq 'select(.type == "summary")'
|
||||||
|
```
|
||||||
|
|
||||||
|
### GitHub Actions Example
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: Run Agent Tests
|
||||||
|
run: |
|
||||||
|
yao agent test -i assistants/keyword/tests/inputs.jsonl \
|
||||||
|
-u ci-user -t ci-team \
|
||||||
|
--runs 3 \
|
||||||
|
-o report.json
|
||||||
|
|
||||||
|
- name: Check Stability
|
||||||
|
run: |
|
||||||
|
jq -e '.results | all(.pass_rate >= 80)' report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Quick development test (auto-detect agent)
|
||||||
|
cd assistants/keyword
|
||||||
|
yao agent test -i "Extract keywords: AI and ML"
|
||||||
|
|
||||||
|
# Quick development test (specify agent)
|
||||||
|
yao agent test -i "Hello" -n workers.system.keyword
|
||||||
|
|
||||||
|
# Full test suite with HTML report
|
||||||
|
yao agent test -i tests/inputs.jsonl -o report.html -v
|
||||||
|
|
||||||
|
# Override connector
|
||||||
|
yao agent test -i tests/inputs.jsonl -c openai.gpt4
|
||||||
|
|
||||||
|
# Stability analysis
|
||||||
|
yao agent test -i tests/inputs.jsonl --runs 10 -o stability.json
|
||||||
|
|
||||||
|
# Parallel execution with timeout
|
||||||
|
yao agent test -i tests/inputs.jsonl --parallel 4 --timeout 2m
|
||||||
|
|
||||||
|
# Custom test environment
|
||||||
|
yao agent test -i tests/inputs.jsonl -u admin -t prod-team
|
||||||
|
|
||||||
|
# Custom reporter agent
|
||||||
|
yao agent test -i tests/inputs.jsonl -r report.beautiful -o custom-report.md
|
||||||
|
|
||||||
|
# Full example with all options
|
||||||
|
yao agent test -i tests/inputs.jsonl \
|
||||||
|
-n keyword.agent \
|
||||||
|
-c deepseek.v3 \
|
||||||
|
-u test-user \
|
||||||
|
-t test-team \
|
||||||
|
--runs 3 \
|
||||||
|
--timeout 10m \
|
||||||
|
--parallel 4 \
|
||||||
|
-r report.html \
|
||||||
|
-o report.html
|
||||||
|
```
|
||||||
|
|
||||||
|
## Exit Codes
|
||||||
|
|
||||||
|
| Code | Description |
|
||||||
|
| ---- | --------------------------------------------------- |
|
||||||
|
| 0 | All tests passed |
|
||||||
|
| 1 | Tests failed, configuration error, or runtime error |
|
||||||
480
agent/test/assert.go
Normal file
480
agent/test/assert.go
Normal file
|
|
@ -0,0 +1,480 @@
|
||||||
|
package test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
jsoniter "github.com/json-iterator/go"
|
||||||
|
"github.com/yaoapp/gou/process"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Asserter handles test assertions
|
||||||
|
type Asserter struct{}
|
||||||
|
|
||||||
|
// NewAsserter creates a new asserter
|
||||||
|
func NewAsserter() *Asserter {
|
||||||
|
return &Asserter{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate validates the output against the test case's assertions
|
||||||
|
// Returns (passed, error message)
|
||||||
|
func (a *Asserter) Validate(tc *Case, output interface{}) (bool, string) {
|
||||||
|
// If assert is defined, use assertion rules
|
||||||
|
if tc.Assert != nil {
|
||||||
|
return a.validateAssertions(tc, output)
|
||||||
|
}
|
||||||
|
|
||||||
|
// If expected is defined, use simple comparison
|
||||||
|
if tc.Expected != nil {
|
||||||
|
if validateOutput(output, tc.Expected) {
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
return false, "output does not match expected"
|
||||||
|
}
|
||||||
|
|
||||||
|
// No assertions defined - pass if we got output without error
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// validateAssertions validates output against assertion rules
|
||||||
|
func (a *Asserter) validateAssertions(tc *Case, output interface{}) (bool, string) {
|
||||||
|
assertions := a.parseAssertions(tc.Assert)
|
||||||
|
if len(assertions) == 0 {
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
var failures []string
|
||||||
|
for _, assertion := range assertions {
|
||||||
|
result := a.evaluateAssertion(assertion, output, tc.Input)
|
||||||
|
if !result.Passed {
|
||||||
|
msg := result.Message
|
||||||
|
if assertion.Message != "" {
|
||||||
|
msg = assertion.Message
|
||||||
|
}
|
||||||
|
failures = append(failures, msg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(failures) > 0 {
|
||||||
|
return false, strings.Join(failures, "; ")
|
||||||
|
}
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseAssertions parses the assert field into a list of assertions
|
||||||
|
func (a *Asserter) parseAssertions(assert interface{}) []*Assertion {
|
||||||
|
if assert == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var assertions []*Assertion
|
||||||
|
|
||||||
|
switch v := assert.(type) {
|
||||||
|
case map[string]interface{}:
|
||||||
|
// Single assertion object
|
||||||
|
assertion := a.mapToAssertion(v)
|
||||||
|
if assertion != nil {
|
||||||
|
assertions = append(assertions, assertion)
|
||||||
|
}
|
||||||
|
|
||||||
|
case []interface{}:
|
||||||
|
// Array of assertions
|
||||||
|
for _, item := range v {
|
||||||
|
if m, ok := item.(map[string]interface{}); ok {
|
||||||
|
assertion := a.mapToAssertion(m)
|
||||||
|
if assertion != nil {
|
||||||
|
assertions = append(assertions, assertion)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
case string:
|
||||||
|
// Shorthand: just a type name (e.g., "contains")
|
||||||
|
assertions = append(assertions, &Assertion{Type: v})
|
||||||
|
}
|
||||||
|
|
||||||
|
return assertions
|
||||||
|
}
|
||||||
|
|
||||||
|
// mapToAssertion converts a map to an Assertion
|
||||||
|
func (a *Asserter) mapToAssertion(m map[string]interface{}) *Assertion {
|
||||||
|
assertion := &Assertion{}
|
||||||
|
|
||||||
|
if t, ok := m["type"].(string); ok {
|
||||||
|
assertion.Type = t
|
||||||
|
}
|
||||||
|
if v, ok := m["value"]; ok {
|
||||||
|
assertion.Value = v
|
||||||
|
}
|
||||||
|
if p, ok := m["path"].(string); ok {
|
||||||
|
assertion.Path = p
|
||||||
|
}
|
||||||
|
if s, ok := m["script"].(string); ok {
|
||||||
|
assertion.Script = s
|
||||||
|
}
|
||||||
|
if msg, ok := m["message"].(string); ok {
|
||||||
|
assertion.Message = msg
|
||||||
|
}
|
||||||
|
if n, ok := m["negate"].(bool); ok {
|
||||||
|
assertion.Negate = n
|
||||||
|
}
|
||||||
|
|
||||||
|
return assertion
|
||||||
|
}
|
||||||
|
|
||||||
|
// evaluateAssertion evaluates a single assertion
|
||||||
|
func (a *Asserter) evaluateAssertion(assertion *Assertion, output, input interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Expected: assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
switch assertion.Type {
|
||||||
|
case "equals", "":
|
||||||
|
result = a.assertEquals(assertion, output)
|
||||||
|
case "contains":
|
||||||
|
result = a.assertContains(assertion, output)
|
||||||
|
case "not_contains":
|
||||||
|
result = a.assertNotContains(assertion, output)
|
||||||
|
case "json_path":
|
||||||
|
result = a.assertJSONPath(assertion, output)
|
||||||
|
case "regex":
|
||||||
|
result = a.assertRegex(assertion, output)
|
||||||
|
case "type":
|
||||||
|
result = a.assertType(assertion, output)
|
||||||
|
case "script":
|
||||||
|
result = a.assertScript(assertion, output, input)
|
||||||
|
default:
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("unknown assertion type: %s", assertion.Type)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply negate
|
||||||
|
if assertion.Negate {
|
||||||
|
result.Passed = !result.Passed
|
||||||
|
if result.Passed {
|
||||||
|
result.Message = "negated assertion passed"
|
||||||
|
} else {
|
||||||
|
result.Message = "negated: " + result.Message
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertEquals checks for exact equality
|
||||||
|
func (a *Asserter) assertEquals(assertion *Assertion, output interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Actual: output,
|
||||||
|
Expected: assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
if validateOutput(output, assertion.Value) {
|
||||||
|
result.Passed = true
|
||||||
|
result.Message = "values are equal"
|
||||||
|
} else {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("expected %v, got %v", assertion.Value, output)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertContains checks if output contains the expected value
|
||||||
|
func (a *Asserter) assertContains(assertion *Assertion, output interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Actual: output,
|
||||||
|
Expected: assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
outputStr := a.toString(output)
|
||||||
|
expectedStr := a.toString(assertion.Value)
|
||||||
|
|
||||||
|
if strings.Contains(outputStr, expectedStr) {
|
||||||
|
result.Passed = true
|
||||||
|
result.Message = fmt.Sprintf("output contains '%s'", expectedStr)
|
||||||
|
} else {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("output does not contain '%s'", expectedStr)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertNotContains checks if output does not contain the expected value
|
||||||
|
func (a *Asserter) assertNotContains(assertion *Assertion, output interface{}) *AssertionResult {
|
||||||
|
result := a.assertContains(assertion, output)
|
||||||
|
result.Passed = !result.Passed
|
||||||
|
if result.Passed {
|
||||||
|
result.Message = fmt.Sprintf("output does not contain '%s'", a.toString(assertion.Value))
|
||||||
|
} else {
|
||||||
|
result.Message = fmt.Sprintf("output should not contain '%s'", a.toString(assertion.Value))
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertJSONPath extracts a value using JSON path and compares
|
||||||
|
func (a *Asserter) assertJSONPath(assertion *Assertion, output interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Expected: assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert output to JSON if needed
|
||||||
|
var jsonData interface{}
|
||||||
|
switch v := output.(type) {
|
||||||
|
case string:
|
||||||
|
// Try to parse as JSON
|
||||||
|
if err := jsoniter.Unmarshal([]byte(v), &jsonData); err != nil {
|
||||||
|
// Try to extract JSON from markdown code blocks
|
||||||
|
extracted := extractJSONFromText(v)
|
||||||
|
if extracted != nil {
|
||||||
|
jsonData = extracted
|
||||||
|
} else {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("output is not valid JSON: %s", err.Error())
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case map[string]interface{}, []interface{}:
|
||||||
|
jsonData = v
|
||||||
|
default:
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = "output is not a JSON object or array"
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract value using simple path (e.g., "$.need_search" or "need_search")
|
||||||
|
path := strings.TrimPrefix(assertion.Path, "$.")
|
||||||
|
actual := a.extractPath(jsonData, path)
|
||||||
|
result.Actual = actual
|
||||||
|
|
||||||
|
if validateOutput(actual, assertion.Value) {
|
||||||
|
result.Passed = true
|
||||||
|
result.Message = fmt.Sprintf("path '%s' equals expected value", assertion.Path)
|
||||||
|
} else {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("path '%s': expected %v, got %v", assertion.Path, assertion.Value, actual)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// extractPath extracts a value from JSON using a simple dot-notation path
|
||||||
|
func (a *Asserter) extractPath(data interface{}, path string) interface{} {
|
||||||
|
parts := strings.Split(path, ".")
|
||||||
|
current := data
|
||||||
|
|
||||||
|
for _, part := range parts {
|
||||||
|
if part == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
switch v := current.(type) {
|
||||||
|
case map[string]interface{}:
|
||||||
|
current = v[part]
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return current
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertRegex checks if output matches a regex pattern
|
||||||
|
func (a *Asserter) assertRegex(assertion *Assertion, output interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Actual: output,
|
||||||
|
Expected: assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
pattern, ok := assertion.Value.(string)
|
||||||
|
if !ok {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = "regex pattern must be a string"
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
re, err := regexp.Compile(pattern)
|
||||||
|
if err != nil {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("invalid regex pattern: %s", err.Error())
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
outputStr := a.toString(output)
|
||||||
|
if re.MatchString(outputStr) {
|
||||||
|
result.Passed = true
|
||||||
|
result.Message = fmt.Sprintf("output matches pattern '%s'", pattern)
|
||||||
|
} else {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("output does not match pattern '%s'", pattern)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertType checks the type of the output
|
||||||
|
func (a *Asserter) assertType(assertion *Assertion, output interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Actual: output,
|
||||||
|
Expected: assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
expectedType, ok := assertion.Value.(string)
|
||||||
|
if !ok {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = "type assertion value must be a string"
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
actualType := a.getType(output)
|
||||||
|
result.Actual = actualType
|
||||||
|
|
||||||
|
if actualType == expectedType {
|
||||||
|
result.Passed = true
|
||||||
|
result.Message = fmt.Sprintf("output is of type '%s'", expectedType)
|
||||||
|
} else {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("expected type '%s', got '%s'", expectedType, actualType)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// getType returns the type name of a value
|
||||||
|
func (a *Asserter) getType(v interface{}) string {
|
||||||
|
if v == nil {
|
||||||
|
return "null"
|
||||||
|
}
|
||||||
|
|
||||||
|
switch v.(type) {
|
||||||
|
case string:
|
||||||
|
return "string"
|
||||||
|
case float64, float32, int, int64, int32:
|
||||||
|
return "number"
|
||||||
|
case bool:
|
||||||
|
return "boolean"
|
||||||
|
case []interface{}:
|
||||||
|
return "array"
|
||||||
|
case map[string]interface{}:
|
||||||
|
return "object"
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("%T", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertScript runs a custom assertion script
|
||||||
|
func (a *Asserter) assertScript(assertion *Assertion, output, input interface{}) *AssertionResult {
|
||||||
|
result := &AssertionResult{
|
||||||
|
Assertion: assertion,
|
||||||
|
Actual: output,
|
||||||
|
}
|
||||||
|
|
||||||
|
if assertion.Script == "" {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = "script assertion requires a script name"
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build script arguments
|
||||||
|
args := []interface{}{
|
||||||
|
output,
|
||||||
|
input,
|
||||||
|
assertion.Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run the script as a process
|
||||||
|
p, err := process.Of(assertion.Script, args...)
|
||||||
|
if err != nil {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("failed to create process: %s", err.Error())
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := p.Exec()
|
||||||
|
if err != nil {
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("script execution failed: %s", err.Error())
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse script result
|
||||||
|
// Expected format: { "pass": bool, "message": string }
|
||||||
|
switch v := res.(type) {
|
||||||
|
case bool:
|
||||||
|
result.Passed = v
|
||||||
|
if v {
|
||||||
|
result.Message = "script assertion passed"
|
||||||
|
} else {
|
||||||
|
result.Message = "script assertion failed"
|
||||||
|
}
|
||||||
|
|
||||||
|
case map[string]interface{}:
|
||||||
|
if pass, ok := v["pass"].(bool); ok {
|
||||||
|
result.Passed = pass
|
||||||
|
}
|
||||||
|
if msg, ok := v["message"].(string); ok {
|
||||||
|
result.Message = msg
|
||||||
|
}
|
||||||
|
|
||||||
|
default:
|
||||||
|
result.Passed = false
|
||||||
|
result.Message = fmt.Sprintf("script returned unexpected type: %T", res)
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// toString converts a value to string for comparison
|
||||||
|
func (a *Asserter) toString(v interface{}) string {
|
||||||
|
if v == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
switch val := v.(type) {
|
||||||
|
case string:
|
||||||
|
return val
|
||||||
|
case []byte:
|
||||||
|
return string(val)
|
||||||
|
default:
|
||||||
|
b, err := json.Marshal(v)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Sprintf("%v", v)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// extractJSONFromText tries to extract JSON from text (e.g., markdown code blocks)
|
||||||
|
func extractJSONFromText(text string) interface{} {
|
||||||
|
// Try to find JSON in code blocks
|
||||||
|
patterns := []string{
|
||||||
|
"```json\n",
|
||||||
|
"```\n",
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, start := range patterns {
|
||||||
|
if idx := strings.Index(text, start); idx >= 0 {
|
||||||
|
text = text[idx+len(start):]
|
||||||
|
if endIdx := strings.Index(text, "```"); endIdx >= 0 {
|
||||||
|
text = text[:endIdx]
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to parse
|
||||||
|
var result interface{}
|
||||||
|
if err := jsoniter.Unmarshal([]byte(strings.TrimSpace(text)), &result); err == nil {
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
@ -64,8 +64,12 @@ func (r *Executor) RunDirect() (*Report, error) {
|
||||||
ctx := NewTestContextFromOptions(chatID, agentInfo.ID, r.opts, tc)
|
ctx := NewTestContextFromOptions(chatID, agentInfo.ID, r.opts, tc)
|
||||||
defer ctx.Release()
|
defer ctx.Release()
|
||||||
|
|
||||||
// Set connector override if specified
|
// Set options: skip history (input already contains conversation), connector override
|
||||||
opts := &context.Options{}
|
opts := &context.Options{
|
||||||
|
Skip: &context.Skip{
|
||||||
|
History: true, // Skip history loading - input already contains full conversation
|
||||||
|
},
|
||||||
|
}
|
||||||
if r.opts.Connector != "" {
|
if r.opts.Connector != "" {
|
||||||
opts.Connector = r.opts.Connector
|
opts.Connector = r.opts.Connector
|
||||||
}
|
}
|
||||||
|
|
@ -293,8 +297,12 @@ func (r *Executor) runSingleTest(ast *assistant.Assistant, tc *Case, agentID str
|
||||||
ctx := NewTestContextFromOptions(chatID, agentID, r.opts, tc)
|
ctx := NewTestContextFromOptions(chatID, agentID, r.opts, tc)
|
||||||
defer ctx.Release()
|
defer ctx.Release()
|
||||||
|
|
||||||
// Set connector override if specified
|
// Set options: skip history (input already contains conversation), connector override
|
||||||
opts := &context.Options{}
|
opts := &context.Options{
|
||||||
|
Skip: &context.Skip{
|
||||||
|
History: true, // Skip history loading - input already contains full conversation
|
||||||
|
},
|
||||||
|
}
|
||||||
if r.opts.Connector != "" {
|
if r.opts.Connector != "" {
|
||||||
opts.Connector = r.opts.Connector
|
opts.Connector = r.opts.Connector
|
||||||
}
|
}
|
||||||
|
|
@ -332,17 +340,14 @@ func (r *Executor) runSingleTest(ast *assistant.Assistant, tc *Case, agentID str
|
||||||
// Extract output
|
// Extract output
|
||||||
result.Output = extractOutput(response)
|
result.Output = extractOutput(response)
|
||||||
|
|
||||||
// Validate result if expected is set
|
// Validate result using asserter
|
||||||
if tc.Expected != nil {
|
asserter := NewAsserter()
|
||||||
if validateOutput(result.Output, tc.Expected) {
|
passed, errMsg := asserter.Validate(tc, result.Output)
|
||||||
result.Status = StatusPassed
|
if passed {
|
||||||
} else {
|
|
||||||
result.Status = StatusFailed
|
|
||||||
result.Error = "output does not match expected"
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// No expected value - pass if no error
|
|
||||||
result.Status = StatusPassed
|
result.Status = StatusPassed
|
||||||
|
} else {
|
||||||
|
result.Status = StatusFailed
|
||||||
|
result.Error = errMsg
|
||||||
}
|
}
|
||||||
|
|
||||||
r.output.TestResult(result.Status, duration)
|
r.output.TestResult(result.Status, duration)
|
||||||
|
|
|
||||||
|
|
@ -190,6 +190,11 @@ type Case struct {
|
||||||
// If set, the actual output will be compared against this
|
// If set, the actual output will be compared against this
|
||||||
Expected interface{} `json:"expected,omitempty"`
|
Expected interface{} `json:"expected,omitempty"`
|
||||||
|
|
||||||
|
// Assert defines custom assertion rules (optional)
|
||||||
|
// If set, these rules will be used instead of simple expected comparison
|
||||||
|
// Can be a single assertion or an array of assertions
|
||||||
|
Assert interface{} `json:"assert,omitempty"`
|
||||||
|
|
||||||
// Environment (per-test case, can be overridden by command line flags)
|
// Environment (per-test case, can be overridden by command line flags)
|
||||||
// ===============================
|
// ===============================
|
||||||
|
|
||||||
|
|
@ -210,6 +215,54 @@ type Case struct {
|
||||||
Timeout string `json:"timeout,omitempty"`
|
Timeout string `json:"timeout,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Assertion represents a single assertion rule
|
||||||
|
type Assertion struct {
|
||||||
|
// Type is the assertion type:
|
||||||
|
// - "equals": exact match (default if expected is set)
|
||||||
|
// - "contains": output contains the expected string/value
|
||||||
|
// - "not_contains": output does not contain the string/value
|
||||||
|
// - "json_path": extract value using JSON path and compare
|
||||||
|
// - "regex": match output against regex pattern
|
||||||
|
// - "script": run a custom assertion script
|
||||||
|
// - "type": check output type (string, object, array, number, boolean)
|
||||||
|
// - "schema": validate against JSON schema
|
||||||
|
Type string `json:"type"`
|
||||||
|
|
||||||
|
// Value is the expected value or pattern (depends on type)
|
||||||
|
Value interface{} `json:"value,omitempty"`
|
||||||
|
|
||||||
|
// Path is the JSON path for json_path assertions (e.g., "$.need_search")
|
||||||
|
Path string `json:"path,omitempty"`
|
||||||
|
|
||||||
|
// Script is the assertion script name for script assertions
|
||||||
|
// The script receives (output, input, expected) and returns {pass: bool, message: string}
|
||||||
|
Script string `json:"script,omitempty"`
|
||||||
|
|
||||||
|
// Message is a custom failure message
|
||||||
|
Message string `json:"message,omitempty"`
|
||||||
|
|
||||||
|
// Negate inverts the assertion result
|
||||||
|
Negate bool `json:"negate,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// AssertionResult represents the result of an assertion
|
||||||
|
type AssertionResult struct {
|
||||||
|
// Passed indicates whether the assertion passed
|
||||||
|
Passed bool `json:"passed"`
|
||||||
|
|
||||||
|
// Message describes the assertion result
|
||||||
|
Message string `json:"message,omitempty"`
|
||||||
|
|
||||||
|
// Assertion is the original assertion that was evaluated
|
||||||
|
Assertion *Assertion `json:"assertion,omitempty"`
|
||||||
|
|
||||||
|
// Actual is the actual value that was compared
|
||||||
|
Actual interface{} `json:"actual,omitempty"`
|
||||||
|
|
||||||
|
// Expected is the expected value
|
||||||
|
Expected interface{} `json:"expected,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// GetEnvironment returns the effective test environment for this test case
|
// GetEnvironment returns the effective test environment for this test case
|
||||||
// Priority: command line flags > test case fields > defaults
|
// Priority: command line flags > test case fields > defaults
|
||||||
func (tc *Case) GetEnvironment(opts *Options) *Environment {
|
func (tc *Case) GetEnvironment(opts *Options) *Environment {
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue