Implement Script Testing Framework and Enhance Test Context Management
- Introduced a new script testing mode to allow testing of agent handler scripts (hooks, tools, etc.) using a Go-like interface, enabling better unit testing of TypeScript/JavaScript code. - Enhanced the `LoadScripts` function to skip test files during script loading, ensuring only relevant scripts are processed. - Refactored the test context creation to support custom context configurations via a JSON file, allowing for flexible authorization and metadata management during tests. - Updated the test runner to handle script tests, including the ability to filter tests using regex patterns and manage custom context data. - Improved documentation to include details on script testing usage, input formats, and available assertions, enhancing developer experience and clarity.
This commit is contained in:
parent
aacb81fb52
commit
5e64c78435
15 changed files with 3048 additions and 119 deletions
|
|
@ -75,6 +75,11 @@ func LoadScripts(srcDir string) (*hook.Script, map[string]*Script, error) {
|
|||
// Get relative path for determining if it's index
|
||||
relPath := strings.TrimPrefix(file, root+"/")
|
||||
|
||||
// Skip test files (*_test.ts, *_test.js)
|
||||
if strings.HasSuffix(relPath, "_test.ts") || strings.HasSuffix(relPath, "_test.js") {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check if it's the root index.ts/js (hook script)
|
||||
// Only src/index.ts is the hook script, not src/foo/index.ts
|
||||
isRootIndex := relPath == "index.ts" || relPath == "index.js"
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@
|
|||
Agent Test Package provides a framework for testing AI agents with structured test cases.
|
||||
It supports batch testing, report generation, stability analysis, and CI integration.
|
||||
|
||||
Additionally, it supports **Script Testing** for testing Agent handler scripts (hooks, tools, etc.) with a Go-like testing interface.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
|
|
@ -23,6 +25,12 @@ yao agent test -i assistants/keyword/tests/inputs.jsonl --runs 5
|
|||
|
||||
# Generate HTML report
|
||||
yao agent test -i assistants/keyword/tests/inputs.jsonl -r report.html -o report.html
|
||||
|
||||
# Run script tests (test agent handler scripts)
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Run script tests with specific user/team context
|
||||
yao agent test -i scripts.expense.tools -u admin -t ops-team -v
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
|
@ -77,7 +85,7 @@ yao agent test -i tests/inputs.jsonl \
|
|||
|
||||
### Input Modes
|
||||
|
||||
The `-i` flag supports two input modes:
|
||||
The `-i` flag supports three input modes:
|
||||
|
||||
**1. JSONL File Mode** - Load test cases from a file:
|
||||
|
||||
|
|
@ -104,6 +112,33 @@ When using direct message mode:
|
|||
- Output is printed to stdout (or saved to `-o` if specified)
|
||||
- Useful for quick testing and debugging
|
||||
|
||||
**3. Script Test Mode** - Test agent handler scripts:
|
||||
|
||||
```bash
|
||||
# Run all tests in a script module
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Run with specific user/team context
|
||||
yao agent test -i scripts.expense.tools -u admin -t ops-team
|
||||
|
||||
# Run with timeout
|
||||
yao agent test -i scripts.expense.setup --timeout 30s
|
||||
|
||||
# Run specific tests by pattern (like go test -run)
|
||||
yao agent test -i scripts.expense.setup -run TestSystemReady
|
||||
|
||||
# Run tests matching a regex pattern
|
||||
yao agent test -i scripts.expense.setup -run "TestSystem.*"
|
||||
```
|
||||
|
||||
When using script test mode:
|
||||
|
||||
- Input starts with `scripts.` prefix to indicate script testing
|
||||
- Maps to the script file (e.g., `scripts.expense.setup` → `expense/src/setup_test.ts`)
|
||||
- Automatically discovers and runs all `Test*` functions in the script
|
||||
- Uses Go-like testing interface with assertions
|
||||
- See [Script Testing](#script-testing) section for details
|
||||
|
||||
### Default Output Path
|
||||
|
||||
When `-o` is not specified and using JSONL file mode, the output file is automatically generated in the same directory as the input file:
|
||||
|
|
@ -131,8 +166,10 @@ When using direct message mode without `-o`, output is printed to stdout.
|
|||
| `-c` | `--connector` | Override connector | agent default | `-c openai.gpt4` |
|
||||
| `-u` | `--user` | Test user ID (global override) | "test-user" | `-u admin` |
|
||||
| `-t` | `--team` | Test team ID (global override) | "test-team" | `-t ops-team` |
|
||||
| | `--ctx` | Path to context JSON file | - | `--ctx tests/context.json` |
|
||||
| `-r` | `--reporter` | Custom reporter agent ID | - (use built-in) | `-r report.beautiful` |
|
||||
| | `--runs` | Number of runs for stability analysis | 1 | `--runs 5` |
|
||||
| | `--run` | Regex pattern to filter tests | - | `--run "TestSystem.*"` |
|
||||
| | `--timeout` | Default timeout per test case | 5m | `--timeout 10m` |
|
||||
| | `--parallel` | Number of parallel test cases | 1 | `--parallel 4` |
|
||||
| `-v` | `--verbose` | Verbose output | false | `-v` |
|
||||
|
|
@ -330,6 +367,437 @@ Create a custom reporter agent at `assistants/reporters/my-reporter/`:
|
|||
- Make it printable
|
||||
```
|
||||
|
||||
## Script Testing
|
||||
|
||||
Script Testing allows you to test Agent handler scripts (hooks, tools, setup functions, etc.) using a Go-like testing interface. This is useful for unit testing individual functions in your agent's TypeScript/JavaScript code.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Run script tests
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# With user/team context
|
||||
yao agent test -i scripts.expense.setup -u admin -t ops-team -v
|
||||
|
||||
# With timeout
|
||||
yao agent test -i scripts.expense.setup --timeout 30s -v
|
||||
```
|
||||
|
||||
### Script Resolution
|
||||
|
||||
The `scripts.` prefix indicates script test mode. The script is resolved as follows:
|
||||
|
||||
| Input | Script Path | Test File |
|
||||
| ----------------------- | ---------------------- | --------------------------- |
|
||||
| `scripts.expense.setup` | `expense/src/setup.ts` | `expense/src/setup_test.ts` |
|
||||
| `scripts.expense.tools` | `expense/src/tools.ts` | `expense/src/tools_test.ts` |
|
||||
| `scripts.keyword.index` | `keyword/src/index.ts` | `keyword/src/index_test.ts` |
|
||||
|
||||
The test file naming convention is `{module}_test.ts` (similar to Go's `_test.go` convention).
|
||||
|
||||
### Test Function Signature
|
||||
|
||||
Test functions must follow this signature:
|
||||
|
||||
```typescript
|
||||
function TestFunctionName(t: testing.T, ctx: agent.Context) {
|
||||
// Test logic here
|
||||
}
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
|
||||
- Function name must start with `Test` (case-sensitive)
|
||||
- First parameter `t` is the testing object with assertions
|
||||
- Second parameter `ctx` is the agent context (same as used in hooks/tools)
|
||||
- Functions not starting with `Test` are ignored (can be used as helpers)
|
||||
|
||||
### Example Test File
|
||||
|
||||
```typescript
|
||||
// setup_test.ts
|
||||
// @ts-nocheck
|
||||
|
||||
// Test the SystemReady function
|
||||
function TestSystemReady(t: testing.T, ctx: agent.Context) {
|
||||
const { assert } = t;
|
||||
|
||||
// Call the function being tested
|
||||
const result = SystemReady(ctx);
|
||||
|
||||
// Assert the result
|
||||
assert.True(result, "SystemReady should return true");
|
||||
}
|
||||
|
||||
// Test error case
|
||||
function TestSystemReadyWithInvalidContext(t: testing.T, ctx: agent.Context) {
|
||||
const { assert } = t;
|
||||
|
||||
// Modify context to simulate error condition
|
||||
ctx.User = null;
|
||||
|
||||
const result = SystemReady(ctx);
|
||||
assert.False(result, "SystemReady should return false when user is null");
|
||||
}
|
||||
|
||||
// Helper function (not a test - doesn't start with "Test")
|
||||
function createMockData() {
|
||||
return { id: 1, name: "test" };
|
||||
}
|
||||
|
||||
// Test with helper
|
||||
function TestSetupWithMockData(t: testing.T, ctx: agent.Context) {
|
||||
const { assert } = t;
|
||||
const mockData = createMockData();
|
||||
|
||||
const result = Setup(ctx, mockData);
|
||||
assert.NotNil(result, "Setup should return a result");
|
||||
assert.Equal(result.id, 1, "Result ID should match");
|
||||
}
|
||||
```
|
||||
|
||||
### Testing Object (`t`)
|
||||
|
||||
The `t` parameter provides the testing interface:
|
||||
|
||||
```typescript
|
||||
interface testing.T {
|
||||
// Assertions object
|
||||
assert: testing.Assert;
|
||||
|
||||
// Test metadata
|
||||
name: string; // Current test function name
|
||||
failed: boolean; // Whether the test has failed
|
||||
|
||||
// Logging (output appears in test report)
|
||||
log(...args: any[]): void; // Log info message
|
||||
error(...args: any[]): void; // Log error message
|
||||
|
||||
// Control flow
|
||||
skip(reason?: string): void; // Skip this test
|
||||
fail(reason?: string): void; // Mark test as failed
|
||||
fatal(reason?: string): void; // Mark as failed and stop execution
|
||||
}
|
||||
```
|
||||
|
||||
### Assertions (`t.assert`)
|
||||
|
||||
The `assert` object provides assertion methods:
|
||||
|
||||
| Method | Description |
|
||||
| ----------------------------------- | ---------------------------------- |
|
||||
| `True(value, message?)` | Assert value is true |
|
||||
| `False(value, message?)` | Assert value is false |
|
||||
| `Equal(actual, expected, message?)` | Assert deep equality |
|
||||
| `NotEqual(actual, expected, msg?)` | Assert not equal |
|
||||
| `Nil(value, message?)` | Assert value is null/undefined |
|
||||
| `NotNil(value, message?)` | Assert value is not null/undefined |
|
||||
| `Contains(str, substr, message?)` | Assert string contains substring |
|
||||
| `NotContains(str, substr, msg?)` | Assert string does not contain |
|
||||
| `Len(value, length, message?)` | Assert array/string length |
|
||||
| `Greater(a, b, message?)` | Assert a > b |
|
||||
| `GreaterOrEqual(a, b, message?)` | Assert a >= b |
|
||||
| `Less(a, b, message?)` | Assert a < b |
|
||||
| `LessOrEqual(a, b, message?)` | Assert a <= b |
|
||||
| `Error(err, message?)` | Assert err is an error |
|
||||
| `NoError(err, message?)` | Assert err is null/undefined |
|
||||
| `Panic(fn, message?)` | Assert function throws |
|
||||
| `NoPanic(fn, message?)` | Assert function does not throw |
|
||||
| `Match(value, pattern, message?)` | Assert value matches regex |
|
||||
| `NotMatch(value, pattern, msg?)` | Assert value does not match regex |
|
||||
| `JSONPath(obj, path, expected, m?)` | Assert JSON path value |
|
||||
| `Type(value, typeName, message?)` | Assert value type |
|
||||
|
||||
### Agent Context (`ctx`)
|
||||
|
||||
The `ctx` parameter is the same `agent.Context` used in agent hooks and tools:
|
||||
|
||||
```typescript
|
||||
interface agent.Context {
|
||||
// User information (from -u flag or default)
|
||||
User: {
|
||||
ID: string;
|
||||
Name?: string;
|
||||
};
|
||||
|
||||
// Team information (from -t flag or default)
|
||||
Team: {
|
||||
ID: string;
|
||||
Name?: string;
|
||||
};
|
||||
|
||||
// Locale (default: "en-us")
|
||||
Locale: string;
|
||||
|
||||
// Client information
|
||||
Client: {
|
||||
Type: string; // "test"
|
||||
IP: string; // "127.0.0.1"
|
||||
};
|
||||
|
||||
// Metadata (can be set via test case)
|
||||
Metadata: Record<string, any>;
|
||||
|
||||
// Chat/Session ID
|
||||
ChatID: string;
|
||||
|
||||
// Assistant ID (resolved from script path)
|
||||
AssistantID: string;
|
||||
}
|
||||
```
|
||||
|
||||
### Script Test Output
|
||||
|
||||
Script test results are reported in the same format as agent tests:
|
||||
|
||||
```
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
Script Test: scripts.expense.setup
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
Script: expense/src/setup_test.ts
|
||||
Tests: 3 functions
|
||||
User: test-user
|
||||
Team: test-team
|
||||
───────────────────────────────────────────────────────────────────────────────
|
||||
Running Tests
|
||||
───────────────────────────────────────────────────────────────────────────────
|
||||
► [TestSystemReady] ...
|
||||
✓ PASSED (12ms)
|
||||
|
||||
► [TestSystemReadyWithInvalidContext] ...
|
||||
✓ PASSED (8ms)
|
||||
|
||||
► [TestSetupWithMockData] ...
|
||||
✗ FAILED (15ms)
|
||||
└─ assertion failed: Result ID should match
|
||||
expected: 1
|
||||
actual: 2
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
Summary: 2 passed, 1 failed, 0 skipped (35ms)
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
```
|
||||
|
||||
### Script Test Options
|
||||
|
||||
Script tests support the following command line options:
|
||||
|
||||
| Flag | Description | Default | Example |
|
||||
| ------------- | -------------------------------- | ----------- | -------------------- |
|
||||
| `-u` | User ID for context | "test-user" | `-u admin` |
|
||||
| `-t` | Team ID for context | "test-team" | `-t ops-team` |
|
||||
| `--ctx` | Path to context JSON file | - | `--ctx context.json` |
|
||||
| `-v` | Verbose output | false | `-v` |
|
||||
| `--run` | Regex to filter tests | - | `--run "TestSystem"` |
|
||||
| `--timeout` | Timeout per test function | 30s | `--timeout 1m` |
|
||||
| `--fail-fast` | Stop on first failure | false | `--fail-fast` |
|
||||
| `-o` | Output file for report | stdout | `-o report.json` |
|
||||
| `-r` | Reporter agent for custom report | - | `-r report.html` |
|
||||
|
||||
The `--run` flag accepts a Go-style regex pattern to filter which tests to run:
|
||||
|
||||
```bash
|
||||
# Run only TestSystemReady
|
||||
yao agent test -i scripts.expense.setup --run TestSystemReady
|
||||
|
||||
# Run all tests starting with "TestSystem"
|
||||
yao agent test -i scripts.expense.setup --run "TestSystem.*"
|
||||
|
||||
# Run tests containing "Error"
|
||||
yao agent test -i scripts.expense.setup --run ".*Error.*"
|
||||
```
|
||||
|
||||
### Custom Context Configuration
|
||||
|
||||
The `--ctx` flag allows you to provide a JSON file with custom context configuration, giving full control over authorization data, metadata, and client information:
|
||||
|
||||
```bash
|
||||
# Use custom context file
|
||||
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
|
||||
```
|
||||
|
||||
**Context JSON Format:**
|
||||
|
||||
```json
|
||||
{
|
||||
"authorized": {
|
||||
"sub": "user-12345",
|
||||
"client_id": "my-app",
|
||||
"scope": "read write",
|
||||
"session_id": "sess-abc123",
|
||||
"user_id": "admin",
|
||||
"team_id": "team-001",
|
||||
"tenant_id": "acme-corp",
|
||||
"remember_me": false,
|
||||
"constraints": {
|
||||
"owner_only": false,
|
||||
"creator_only": false,
|
||||
"editor_only": false,
|
||||
"team_only": true,
|
||||
"extra": {
|
||||
"department": "engineering",
|
||||
"region": "us-west"
|
||||
}
|
||||
}
|
||||
},
|
||||
"metadata": {
|
||||
"request_id": "req-123",
|
||||
"trace_id": "trace-456",
|
||||
"custom_field": "custom_value"
|
||||
},
|
||||
"client": {
|
||||
"type": "web",
|
||||
"user_agent": "Mozilla/5.0",
|
||||
"ip": "192.168.1.100"
|
||||
},
|
||||
"locale": "zh-cn",
|
||||
"referer": "https://example.com/dashboard"
|
||||
}
|
||||
```
|
||||
|
||||
**Field Descriptions:**
|
||||
|
||||
| Field | Description |
|
||||
| -------------------------- | --------------------------------------------------- |
|
||||
| `authorized.sub` | Subject identifier (JWT sub claim) |
|
||||
| `authorized.client_id` | OAuth client ID |
|
||||
| `authorized.scope` | Access scope |
|
||||
| `authorized.session_id` | Session identifier |
|
||||
| `authorized.user_id` | User identifier (overrides -u flag) |
|
||||
| `authorized.team_id` | Team identifier (overrides -t flag) |
|
||||
| `authorized.tenant_id` | Tenant identifier |
|
||||
| `authorized.remember_me` | Remember me flag |
|
||||
| `authorized.constraints` | Data access constraints (set by ACL enforcement) |
|
||||
| `constraints.owner_only` | Only access owner's data |
|
||||
| `constraints.creator_only` | Only access creator's data |
|
||||
| `constraints.editor_only` | Only access editor's data |
|
||||
| `constraints.team_only` | Only access team's data (filter by team_id) |
|
||||
| `constraints.extra` | User-defined constraints (department, region, etc.) |
|
||||
| `metadata` | Custom metadata passed to context |
|
||||
| `client.type` | Client type (web, mobile, test, etc.) |
|
||||
| `client.user_agent` | Client user agent string |
|
||||
| `client.ip` | Client IP address |
|
||||
| `locale` | Locale setting (e.g., "en-us", "zh-cn") |
|
||||
| `referer` | Request referer URL |
|
||||
|
||||
**Priority:** When both `-u`/`-t` flags and `--ctx` file are provided, the context file values take precedence.
|
||||
|
||||
### Script Test Report Format
|
||||
|
||||
When using `-o` to save results:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "script_test",
|
||||
"script": "scripts.expense.setup",
|
||||
"script_path": "expense/src/setup_test.ts",
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"passed": 2,
|
||||
"failed": 1,
|
||||
"skipped": 0,
|
||||
"duration_ms": 35
|
||||
},
|
||||
"environment": {
|
||||
"user_id": "test-user",
|
||||
"team_id": "test-team",
|
||||
"locale": "en-us"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"name": "TestSystemReady",
|
||||
"status": "passed",
|
||||
"duration_ms": 12,
|
||||
"logs": []
|
||||
},
|
||||
{
|
||||
"name": "TestSystemReadyWithInvalidContext",
|
||||
"status": "passed",
|
||||
"duration_ms": 8,
|
||||
"logs": []
|
||||
},
|
||||
{
|
||||
"name": "TestSetupWithMockData",
|
||||
"status": "failed",
|
||||
"duration_ms": 15,
|
||||
"error": "assertion failed: Result ID should match",
|
||||
"assertion": {
|
||||
"type": "Equal",
|
||||
"expected": 1,
|
||||
"actual": 2,
|
||||
"message": "Result ID should match"
|
||||
},
|
||||
"logs": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"started_at": "2024-12-17T10:00:00Z",
|
||||
"completed_at": "2024-12-17T10:00:00Z",
|
||||
"version": "0.10.5"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Naming Convention**: Use descriptive test names that explain what's being tested
|
||||
|
||||
- Good: `TestSystemReadyWithValidUser`, `TestSetupReturnsErrorOnMissingConfig`
|
||||
- Bad: `Test1`, `TestIt`
|
||||
|
||||
2. **One Assertion Per Concept**: Each test should verify one behavior
|
||||
|
||||
```typescript
|
||||
// Good: Focused tests
|
||||
function TestSetupCreatesDatabase(t, ctx) { ... }
|
||||
function TestSetupInitializesCache(t, ctx) { ... }
|
||||
|
||||
// Bad: Testing too many things
|
||||
function TestSetup(t, ctx) {
|
||||
// tests database, cache, config, etc.
|
||||
}
|
||||
```
|
||||
|
||||
3. **Use Helper Functions**: Extract common setup logic
|
||||
|
||||
```typescript
|
||||
function setupTestContext(ctx) {
|
||||
ctx.Metadata.testMode = true;
|
||||
return ctx;
|
||||
}
|
||||
|
||||
function TestFeatureA(t, ctx) {
|
||||
ctx = setupTestContext(ctx);
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
4. **Test Error Cases**: Don't just test happy paths
|
||||
|
||||
```typescript
|
||||
function TestSetupWithMissingConfig(t, ctx) {
|
||||
const { assert } = t;
|
||||
ctx.Metadata.config = null;
|
||||
|
||||
const result = Setup(ctx);
|
||||
assert.Error(result.error, "Should return error for missing config");
|
||||
}
|
||||
```
|
||||
|
||||
5. **Clean Up**: If your test modifies global state, clean up after
|
||||
```typescript
|
||||
function TestWithGlobalState(t, ctx) {
|
||||
const originalValue = GlobalConfig.value;
|
||||
try {
|
||||
GlobalConfig.value = "test";
|
||||
// ... test logic
|
||||
} finally {
|
||||
GlobalConfig.value = originalValue;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Input Format (JSONL)
|
||||
|
||||
Each line in the input file is a JSON object with the following structure:
|
||||
|
|
@ -632,8 +1100,13 @@ agent/test/
|
|||
├── runner.go # Test runner implementation
|
||||
├── loader.go # Test case loader
|
||||
├── resolver.go # Agent resolver
|
||||
├── environment.go # Test environment setup
|
||||
├── stability.go # Stability analysis
|
||||
├── context.go # Test context creation
|
||||
├── assert.go # Assertion implementation
|
||||
├── input.go # Input parsing
|
||||
├── output.go # Output formatting
|
||||
├── script.go # Script test runner (NEW)
|
||||
├── script_types.go # Script test types (NEW)
|
||||
├── script_assert.go # Script assertion bindings (NEW)
|
||||
└── reporter/
|
||||
├── json.go # JSON reporter
|
||||
├── html.go # HTML reporter
|
||||
|
|
@ -665,7 +1138,74 @@ Executes test cases against an agent:
|
|||
- Executes each test case (optionally multiple runs)
|
||||
- Collects results and stability metrics
|
||||
|
||||
### 5. Reporter
|
||||
### 5. ScriptRunner (NEW)
|
||||
|
||||
Executes script tests for agent handler scripts:
|
||||
|
||||
- Resolves script path from `scripts.` prefix
|
||||
- Discovers `Test*` functions in the script
|
||||
- Creates test context with environment
|
||||
- Executes each test function with testing object and context
|
||||
- Collects results and generates report
|
||||
|
||||
### 6. ScriptTestCase (NEW)
|
||||
|
||||
Represents a single script test function:
|
||||
|
||||
```go
|
||||
type ScriptTestCase struct {
|
||||
Name string // Function name (e.g., "TestSystemReady")
|
||||
Function string // Full function reference
|
||||
}
|
||||
```
|
||||
|
||||
### 7. ScriptTestResult (NEW)
|
||||
|
||||
Represents the result of running a script test function:
|
||||
|
||||
```go
|
||||
type ScriptTestResult struct {
|
||||
Name string `json:"name"`
|
||||
Status Status `json:"status"`
|
||||
DurationMs int64 `json:"duration_ms"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Assertion *AssertionInfo `json:"assertion,omitempty"`
|
||||
Logs []string `json:"logs,omitempty"`
|
||||
}
|
||||
|
||||
type AssertionInfo struct {
|
||||
Type string `json:"type"`
|
||||
Expected interface{} `json:"expected,omitempty"`
|
||||
Actual interface{} `json:"actual,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
### 8. ScriptTestReport (NEW)
|
||||
|
||||
Represents the complete script test report:
|
||||
|
||||
```go
|
||||
type ScriptTestReport struct {
|
||||
Type string `json:"type"` // "script_test"
|
||||
Script string `json:"script"`
|
||||
ScriptPath string `json:"script_path"`
|
||||
Summary *ScriptTestSummary `json:"summary"`
|
||||
Environment *Environment `json:"environment"`
|
||||
Results []*ScriptTestResult `json:"results"`
|
||||
Metadata *ReportMetadata `json:"metadata"`
|
||||
}
|
||||
|
||||
type ScriptTestSummary struct {
|
||||
Total int `json:"total"`
|
||||
Passed int `json:"passed"`
|
||||
Failed int `json:"failed"`
|
||||
Skipped int `json:"skipped"`
|
||||
DurationMs int64 `json:"duration_ms"`
|
||||
}
|
||||
```
|
||||
|
||||
### 9. Reporter
|
||||
|
||||
Generates reports in various formats. The format is determined by the `-o` file extension:
|
||||
|
||||
|
|
@ -713,7 +1253,8 @@ This allows for fully customizable report generation using AI agents
|
|||
```go
|
||||
type Options struct {
|
||||
// Input/Output
|
||||
InputFile string // Path to inputs.jsonl
|
||||
Input string // Input source: file path, message, or scripts.xxx
|
||||
InputMode InputMode // Auto-detected: file, message, or script
|
||||
OutputFile string // Path to output report
|
||||
|
||||
// Agent Selection
|
||||
|
|
@ -737,6 +1278,186 @@ type Options struct {
|
|||
Verbose bool // Verbose output
|
||||
FailFast bool // Stop on first failure
|
||||
}
|
||||
|
||||
// InputMode represents the input mode for test cases
|
||||
type InputMode string
|
||||
|
||||
const (
|
||||
InputModeFile InputMode = "file" // JSONL file input
|
||||
InputModeMessage InputMode = "message" // Direct message input
|
||||
InputModeScript InputMode = "script" // Script test mode (NEW)
|
||||
)
|
||||
```
|
||||
|
||||
### Input Mode Detection
|
||||
|
||||
The input mode is automatically detected based on the input value:
|
||||
|
||||
| Input Pattern | Mode | Description |
|
||||
| ----------------- | --------- | -------------------------- |
|
||||
| `scripts.xxx.yyy` | `script` | Script test mode |
|
||||
| `*.jsonl` | `file` | JSONL file mode |
|
||||
| `path/to/file` | `file` | File path (if file exists) |
|
||||
| `"any text"` | `message` | Direct message mode |
|
||||
|
||||
```go
|
||||
func DetectInputMode(input string) InputMode {
|
||||
// Check for script test prefix
|
||||
if strings.HasPrefix(input, "scripts.") {
|
||||
return InputModeScript
|
||||
}
|
||||
|
||||
// Check if it's a file path
|
||||
if strings.HasSuffix(input, ".jsonl") || fileExists(input) {
|
||||
return InputModeFile
|
||||
}
|
||||
|
||||
// Default to message mode
|
||||
return InputModeMessage
|
||||
}
|
||||
```
|
||||
|
||||
## Script Testing Implementation
|
||||
|
||||
### Script Resolution
|
||||
|
||||
```go
|
||||
// ResolveScript resolves the script path from scripts.xxx.yyy format
|
||||
func ResolveScript(input string) (*ScriptInfo, error) {
|
||||
// Remove "scripts." prefix
|
||||
path := strings.TrimPrefix(input, "scripts.")
|
||||
|
||||
// Split into parts: "expense.setup" -> ["expense", "setup"]
|
||||
parts := strings.Split(path, ".")
|
||||
if len(parts) < 2 {
|
||||
return nil, fmt.Errorf("invalid script path: %s", input)
|
||||
}
|
||||
|
||||
// Build paths
|
||||
// assistantDir: expense
|
||||
// moduleName: setup
|
||||
// scriptPath: expense/src/setup.ts
|
||||
// testPath: expense/src/setup_test.ts
|
||||
assistantDir := parts[0]
|
||||
moduleName := parts[1]
|
||||
|
||||
return &ScriptInfo{
|
||||
ID: input,
|
||||
Assistant: assistantDir,
|
||||
Module: moduleName,
|
||||
ScriptPath: filepath.Join(assistantDir, "src", moduleName+".ts"),
|
||||
TestPath: filepath.Join(assistantDir, "src", moduleName+"_test.ts"),
|
||||
}, nil
|
||||
}
|
||||
```
|
||||
|
||||
### Test Function Discovery
|
||||
|
||||
Test functions are discovered by scanning the script for functions starting with `Test`:
|
||||
|
||||
```go
|
||||
// DiscoverTests finds all Test* functions in the script
|
||||
func DiscoverTests(scriptPath string) ([]*ScriptTestCase, error) {
|
||||
// Use the JavaScript runtime to list exported functions
|
||||
// Filter for functions starting with "Test"
|
||||
// Return list of test cases
|
||||
}
|
||||
```
|
||||
|
||||
### Testing Object Binding
|
||||
|
||||
The `testing.T` object is provided to test functions via JavaScript runtime binding:
|
||||
|
||||
```go
|
||||
// TestingT represents the testing object passed to test functions
|
||||
type TestingT struct {
|
||||
name string
|
||||
failed bool
|
||||
skipped bool
|
||||
logs []string
|
||||
assert *AssertObject
|
||||
}
|
||||
|
||||
// AssertObject provides assertion methods
|
||||
type AssertObject struct {
|
||||
t *TestingT
|
||||
}
|
||||
|
||||
func (a *AssertObject) True(value bool, message ...string) {
|
||||
if !value {
|
||||
a.t.fail(formatMessage("expected true, got false", message))
|
||||
}
|
||||
}
|
||||
|
||||
func (a *AssertObject) Equal(actual, expected interface{}, message ...string) {
|
||||
if !reflect.DeepEqual(actual, expected) {
|
||||
a.t.fail(formatMessage(
|
||||
fmt.Sprintf("expected %v, got %v", expected, actual),
|
||||
message,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
// ... other assertion methods
|
||||
```
|
||||
|
||||
### Script Execution Flow
|
||||
|
||||
```
|
||||
1. Parse input: "scripts.expense.setup"
|
||||
2. Resolve script info:
|
||||
- TestPath: expense/src/setup_test.ts
|
||||
- ScriptPath: expense/src/setup.ts
|
||||
3. Discover test functions: [TestSystemReady, TestSetupWithMockData, ...]
|
||||
4. For each test function:
|
||||
a. Create testing.T object
|
||||
b. Create agent.Context with environment
|
||||
c. Execute: TestFunction(t, ctx)
|
||||
d. Collect result (passed/failed/skipped)
|
||||
5. Generate report
|
||||
```
|
||||
|
||||
### Integration with Existing Runner
|
||||
|
||||
```go
|
||||
func (r *Executor) Run() (*Report, error) {
|
||||
switch r.opts.InputMode {
|
||||
case InputModeScript:
|
||||
return r.RunScriptTests()
|
||||
case InputModeMessage:
|
||||
return r.RunDirect()
|
||||
default:
|
||||
return r.RunTests()
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Executor) RunScriptTests() (*Report, error) {
|
||||
// 1. Resolve script
|
||||
scriptInfo, err := ResolveScript(r.opts.Input)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 2. Discover tests
|
||||
tests, err := DiscoverTests(scriptInfo.TestPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 3. Run each test
|
||||
results := make([]*ScriptTestResult, 0, len(tests))
|
||||
for _, tc := range tests {
|
||||
result := r.runScriptTest(tc, scriptInfo)
|
||||
results = append(results, result)
|
||||
|
||||
if r.opts.FailFast && result.Status == StatusFailed {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Generate report
|
||||
return r.buildScriptReport(scriptInfo, results), nil
|
||||
}
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
|
@ -785,3 +1506,8 @@ The command exits with code 1 if any tests fail, making it easy to integrate wit
|
|||
5. **Diff Reports**: Compare results between runs
|
||||
6. **Flaky Test Detection**: Automatic identification of unstable tests
|
||||
7. **Test Prioritization**: Run most important/failing tests first
|
||||
8. **Script Test Enhancements**:
|
||||
- Parallel script test execution
|
||||
- Setup/Teardown hooks (`TestMain`, `BeforeEach`, `AfterEach`)
|
||||
- Mocking utilities for external dependencies
|
||||
- Code coverage for TypeScript/JavaScript scripts
|
||||
|
|
|
|||
|
|
@ -4,6 +4,8 @@ A testing framework for Yao AI agents with support for assertions, stability ana
|
|||
|
||||
## Quick Start
|
||||
|
||||
### Agent Tests
|
||||
|
||||
```bash
|
||||
# Test with direct message (auto-detect agent from current directory)
|
||||
cd assistants/keyword
|
||||
|
|
@ -22,9 +24,22 @@ yao agent test -i tests/inputs.jsonl -o report.html
|
|||
yao agent test -i tests/inputs.jsonl --runs 5
|
||||
```
|
||||
|
||||
### Script Tests
|
||||
|
||||
```bash
|
||||
# Test agent handler scripts (hooks, tools, setup functions)
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Run specific tests with regex filter
|
||||
yao agent test -i scripts.expense.setup --run "TestSystemReady" -v
|
||||
|
||||
# Run with custom context (authorization, metadata)
|
||||
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
|
||||
```
|
||||
|
||||
## Input Modes
|
||||
|
||||
The `-i` flag supports two input modes:
|
||||
The `-i` flag supports three input modes:
|
||||
|
||||
### 1. JSONL File Mode
|
||||
|
||||
|
|
@ -51,22 +66,101 @@ yao agent test -i "Hello" -n workers.system.keyword
|
|||
|
||||
Output is printed to stdout (or saved to `-o` if specified).
|
||||
|
||||
### 3. Script Test Mode
|
||||
|
||||
Test agent handler scripts (hooks, tools, setup functions):
|
||||
|
||||
```bash
|
||||
# Run all tests in a script module
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Run specific tests with filtering
|
||||
yao agent test -i scripts.expense.setup --run "TestSystemReady"
|
||||
|
||||
# Run with custom context
|
||||
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
|
||||
```
|
||||
|
||||
Script test input format: `scripts.<assistant>.<module>` (e.g., `scripts.expense.setup` → `assistants/expense/src/setup_test.ts`).
|
||||
|
||||
**Writing Test Scripts:**
|
||||
|
||||
Test scripts should be placed alongside the source files with `_test.ts` or `_test.js` suffix:
|
||||
|
||||
```
|
||||
assistants/expense/src/
|
||||
├── setup.ts # Source file
|
||||
├── setup_test.ts # Test file
|
||||
├── tools.ts
|
||||
└── tools_test.ts
|
||||
```
|
||||
|
||||
Test functions must follow the naming convention `Test*` and accept `(t: testing.T, ctx: agent.Context)`:
|
||||
|
||||
```typescript
|
||||
// assistants/expense/src/setup_test.ts
|
||||
import { SystemReady } from "./setup";
|
||||
|
||||
// Test function signature: function Test*(t: testing.T, ctx: agent.Context)
|
||||
export function TestSystemReady(t: testing.T, ctx: agent.Context) {
|
||||
const result = SystemReady(ctx);
|
||||
|
||||
// Use t.assert for assertions
|
||||
t.assert.True(result.success, "SystemReady should succeed");
|
||||
t.assert.Equal(result.status, "ready", "Status should be ready");
|
||||
t.assert.NotNil(result.data, "Data should not be nil");
|
||||
}
|
||||
|
||||
export function TestSystemReadyError(t: testing.T, ctx: agent.Context) {
|
||||
// Access context properties
|
||||
console.log("Testing with user:", ctx.authorized.user_id);
|
||||
|
||||
// Test error handling
|
||||
const result = SystemReady(ctx);
|
||||
t.assert.False(result.error, "Should not have error");
|
||||
}
|
||||
```
|
||||
|
||||
**Available Assertions:**
|
||||
|
||||
| Method | Description |
|
||||
| -------------------------------- | ------------------------------ |
|
||||
| `t.assert.True(value, msg)` | Assert value is true |
|
||||
| `t.assert.False(value, msg)` | Assert value is false |
|
||||
| `t.assert.Equal(a, b, msg)` | Assert a equals b |
|
||||
| `t.assert.NotEqual(a, b, msg)` | Assert a not equals b |
|
||||
| `t.assert.Nil(value, msg)` | Assert value is null/undefined |
|
||||
| `t.assert.NotNil(value, msg)` | Assert value is not nil |
|
||||
| `t.assert.Contains(s, sub, msg)` | Assert string contains substr |
|
||||
| `t.assert.Len(arr, n, msg)` | Assert array/string length |
|
||||
|
||||
**Test Control:**
|
||||
|
||||
| Method | Description |
|
||||
| -------------- | ---------------------------- |
|
||||
| `t.Log(msg)` | Log a message |
|
||||
| `t.Error(msg)` | Mark test as failed with msg |
|
||||
| `t.Fatal(msg)` | Mark failed and stop test |
|
||||
| `t.Skip(msg)` | Skip this test |
|
||||
|
||||
## Command Line Options
|
||||
|
||||
| Flag | Description | Default |
|
||||
| ------------- | ---------------------------------------- | -------------------------- |
|
||||
| `-i` | Input: JSONL file path or direct message | (required) |
|
||||
| `-o` | Output file path | `output-{timestamp}.jsonl` |
|
||||
| `-n` | Agent ID (optional, auto-detected) | auto-detect |
|
||||
| `-c` | Override connector | agent default |
|
||||
| `-u` | Test user ID | `test-user` |
|
||||
| `-t` | Test team ID | `test-team` |
|
||||
| `-r` | Reporter agent ID | built-in |
|
||||
| `--runs` | Runs per test (stability analysis) | 1 |
|
||||
| `--timeout` | Timeout per test | 5m |
|
||||
| `--parallel` | Parallel test cases | 1 |
|
||||
| `-v` | Verbose output | false |
|
||||
| `--fail-fast` | Stop on first failure | false |
|
||||
| Flag | Description | Default |
|
||||
| ------------- | -------------------------------------------------- | -------------------------- |
|
||||
| `-i` | Input: JSONL file path, message, or script ID | (required) |
|
||||
| `-o` | Output file path | `output-{timestamp}.jsonl` |
|
||||
| `-n` | Agent ID (optional, auto-detected) | auto-detect |
|
||||
| `-c` | Override connector | agent default |
|
||||
| `-u` | Test user ID | `test-user` |
|
||||
| `-t` | Test team ID | `test-team` |
|
||||
| `--ctx` | Path to context JSON file for custom authorization | - |
|
||||
| `-r` | Reporter agent ID | built-in |
|
||||
| `--runs` | Runs per test (stability analysis) | 1 |
|
||||
| `--run` | Regex pattern to filter which tests to run | - |
|
||||
| `--timeout` | Timeout per test | 5m |
|
||||
| `--parallel` | Parallel test cases | 1 |
|
||||
| `-v` | Verbose output | false |
|
||||
| `--fail-fast` | Stop on first failure | false |
|
||||
|
||||
## Agent Resolution
|
||||
|
||||
|
|
@ -119,24 +213,24 @@ Each line is a JSON object:
|
|||
|
||||
The `options` field allows per-test-case configuration that maps to `context.Options`:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------------------ | ------- | -------------------------------------------------- |
|
||||
| `connector` | string | Override connector (e.g., `"deepseek.v3"`) |
|
||||
| `mode` | string | Agent mode (default: `"chat"`) |
|
||||
| `search` | bool | Enable/disable search mode (default: `true`) |
|
||||
| `disable_global_prompts` | bool | Temporarily disable global prompts |
|
||||
| `metadata` | map | Custom data passed to hooks (e.g., scenario) |
|
||||
| `skip` | object | Skip configuration (see below) |
|
||||
| Field | Type | Description |
|
||||
| ------------------------ | ------ | -------------------------------------------- |
|
||||
| `connector` | string | Override connector (e.g., `"deepseek.v3"`) |
|
||||
| `mode` | string | Agent mode (default: `"chat"`) |
|
||||
| `search` | bool | Enable/disable search mode (default: `true`) |
|
||||
| `disable_global_prompts` | bool | Temporarily disable global prompts |
|
||||
| `metadata` | map | Custom data passed to hooks (e.g., scenario) |
|
||||
| `skip` | object | Skip configuration (see below) |
|
||||
|
||||
#### Options.skip
|
||||
|
||||
| Field | Type | Description |
|
||||
| --------- | ---- | ------------------------ |
|
||||
| `history` | bool | Skip history loading |
|
||||
| `trace` | bool | Skip trace logging |
|
||||
| `output` | bool | Skip output to client |
|
||||
| `keyword` | bool | Skip keyword extraction |
|
||||
| `search` | bool | Skip auto search |
|
||||
| Field | Type | Description |
|
||||
| --------- | ---- | ----------------------- |
|
||||
| `history` | bool | Skip history loading |
|
||||
| `trace` | bool | Skip trace logging |
|
||||
| `output` | bool | Skip output to client |
|
||||
| `keyword` | bool | Skip keyword extraction |
|
||||
| `search` | bool | Skip auto search |
|
||||
|
||||
**Example with options:**
|
||||
|
||||
|
|
@ -146,10 +240,18 @@ The `options` field allows per-test-case configuration that maps to `context.Opt
|
|||
"input": "Query users with status active",
|
||||
"options": {
|
||||
"connector": "deepseek.v3",
|
||||
"metadata": {"scenario": "filter"},
|
||||
"skip": {"trace": true}
|
||||
"metadata": {
|
||||
"scenario": "filter"
|
||||
},
|
||||
"skip": {
|
||||
"trace": true
|
||||
}
|
||||
},
|
||||
"assert": {"type": "json_path", "path": "from", "value": "users"}
|
||||
"assert": {
|
||||
"type": "json_path",
|
||||
"path": "from",
|
||||
"value": "users"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -290,7 +392,17 @@ return { pass: true, message: "Validation passed" };
|
|||
**Multiple expected values (OR logic):**
|
||||
|
||||
```jsonl
|
||||
{"id": "T005", "assert": {"type": "json_path", "path": "error", "value": ["missing_schema", "missing_query"]}}
|
||||
{
|
||||
"id": "T005",
|
||||
"assert": {
|
||||
"type": "json_path",
|
||||
"path": "error",
|
||||
"value": [
|
||||
"missing_schema",
|
||||
"missing_query"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This passes if `error` equals either `"missing_schema"` or `"missing_query"`.
|
||||
|
|
@ -382,6 +494,9 @@ yao agent test -i tests/inputs.jsonl -o results.jsonl --fail-fast
|
|||
|
||||
# Parse JSONL results
|
||||
cat results.jsonl | jq 'select(.type == "summary")'
|
||||
|
||||
# Run script tests
|
||||
yao agent test -i scripts.expense.setup --fail-fast
|
||||
```
|
||||
|
||||
### GitHub Actions Example
|
||||
|
|
@ -394,6 +509,18 @@ cat results.jsonl | jq 'select(.type == "summary")'
|
|||
--runs 3 \
|
||||
-o report.json
|
||||
|
||||
- name: Run Script Tests
|
||||
run: |
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
yao agent test -i scripts.expense.tools -v
|
||||
|
||||
- name: Run Script Tests with Custom Context
|
||||
run: |
|
||||
yao agent test -i scripts.expense.setup \
|
||||
--ctx tests/context.json \
|
||||
--run "TestSystem.*" \
|
||||
-v
|
||||
|
||||
- name: Check Stability
|
||||
run: |
|
||||
jq -e '.results | all(.pass_rate >= 80)' report.json
|
||||
|
|
@ -401,6 +528,8 @@ cat results.jsonl | jq 'select(.type == "summary")'
|
|||
|
||||
## Examples
|
||||
|
||||
### Agent Tests
|
||||
|
||||
```bash
|
||||
# Quick development test (auto-detect agent)
|
||||
cd assistants/keyword
|
||||
|
|
@ -440,6 +569,32 @@ yao agent test -i tests/inputs.jsonl \
|
|||
-o report.html
|
||||
```
|
||||
|
||||
### Script Tests
|
||||
|
||||
```bash
|
||||
# Run all tests in a script module
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Run specific tests with regex filter
|
||||
yao agent test -i scripts.expense.setup --run "TestSystemReady"
|
||||
|
||||
# Run tests matching a pattern
|
||||
yao agent test -i scripts.expense.setup --run "TestSystem.*" -v
|
||||
|
||||
# Run with custom context (authorization, metadata, etc.)
|
||||
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
|
||||
|
||||
# Run with specific user/team
|
||||
yao agent test -i scripts.expense.setup -u admin -t ops-team -v
|
||||
|
||||
# Combine options
|
||||
yao agent test -i scripts.expense.setup \
|
||||
--ctx tests/context.json \
|
||||
--run "TestSystem.*" \
|
||||
--timeout 30s \
|
||||
-v
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Description |
|
||||
|
|
|
|||
|
|
@ -255,13 +255,27 @@ func (a *Asserter) assertJSONPath(assertion *Assertion, output interface{}) *Ass
|
|||
result.Actual = actual
|
||||
|
||||
// Compare expected value with actual value
|
||||
// First, try direct comparison (handles both primitive values and arrays)
|
||||
if validateOutput(actual, assertion.Value) {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("path '%s' equals expected value", assertion.Path)
|
||||
} else {
|
||||
// If expected is an array, check if actual matches ANY element (IN semantics)
|
||||
if expectedArr, ok := assertion.Value.([]interface{}); ok {
|
||||
// Check if actual is one of the expected values
|
||||
for _, expectedItem := range expectedArr {
|
||||
if validateOutput(actual, expectedItem) {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("path '%s' equals one of expected values", assertion.Path)
|
||||
return result
|
||||
}
|
||||
}
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("path '%s': expected %v, got %v", assertion.Path, assertion.Value, actual)
|
||||
result.Message = fmt.Sprintf("path '%s': expected one of %v, got %v", assertion.Path, assertion.Value, actual)
|
||||
} else {
|
||||
// Direct comparison for non-array expected values
|
||||
if validateOutput(actual, assertion.Value) {
|
||||
result.Passed = true
|
||||
result.Message = fmt.Sprintf("path '%s' equals expected value", assertion.Path)
|
||||
} else {
|
||||
result.Passed = false
|
||||
result.Message = fmt.Sprintf("path '%s': expected %v, got %v", assertion.Path, assertion.Value, actual)
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -13,11 +13,7 @@ import (
|
|||
// but configurable via Environment
|
||||
func NewTestContext(chatID, assistantID string, env *Environment) *context.Context {
|
||||
// Build authorized info from environment
|
||||
authorized := &types.AuthorizedInfo{
|
||||
Subject: env.UserID,
|
||||
UserID: env.UserID,
|
||||
TenantID: env.TeamID,
|
||||
}
|
||||
authorized := buildAuthorizedInfo(env)
|
||||
|
||||
// Create context with standard initialization
|
||||
ctx := context.New(stdContext.Background(), authorized, chatID)
|
||||
|
|
@ -34,6 +30,13 @@ func NewTestContext(chatID, assistantID string, env *Environment) *context.Conte
|
|||
ctx.IDGenerator = message.NewIDGenerator()
|
||||
ctx.Metadata = make(map[string]interface{})
|
||||
|
||||
// Apply metadata from context config if available
|
||||
if env.ContextConfig != nil && env.ContextConfig.Metadata != nil {
|
||||
for k, v := range env.ContextConfig.Metadata {
|
||||
ctx.Metadata[k] = v
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize interrupt controller
|
||||
ctx.Interrupt = context.NewInterruptController()
|
||||
|
||||
|
|
@ -47,6 +50,56 @@ func NewTestContext(chatID, assistantID string, env *Environment) *context.Conte
|
|||
return ctx
|
||||
}
|
||||
|
||||
// buildAuthorizedInfo builds AuthorizedInfo from Environment
|
||||
func buildAuthorizedInfo(env *Environment) *types.AuthorizedInfo {
|
||||
authorized := &types.AuthorizedInfo{
|
||||
Subject: env.UserID,
|
||||
UserID: env.UserID,
|
||||
TenantID: env.TeamID,
|
||||
}
|
||||
|
||||
// Apply custom authorized config if available
|
||||
if env.ContextConfig != nil && env.ContextConfig.Authorized != nil {
|
||||
authCfg := env.ContextConfig.Authorized
|
||||
|
||||
if authCfg.Sub != "" {
|
||||
authorized.Subject = authCfg.Sub
|
||||
}
|
||||
if authCfg.ClientID != "" {
|
||||
authorized.ClientID = authCfg.ClientID
|
||||
}
|
||||
if authCfg.Scope != "" {
|
||||
authorized.Scope = authCfg.Scope
|
||||
}
|
||||
if authCfg.SessionID != "" {
|
||||
authorized.SessionID = authCfg.SessionID
|
||||
}
|
||||
if authCfg.UserID != "" {
|
||||
authorized.UserID = authCfg.UserID
|
||||
}
|
||||
if authCfg.TeamID != "" {
|
||||
authorized.TeamID = authCfg.TeamID
|
||||
}
|
||||
if authCfg.TenantID != "" {
|
||||
authorized.TenantID = authCfg.TenantID
|
||||
}
|
||||
authorized.RememberMe = authCfg.RememberMe
|
||||
|
||||
// Apply constraints
|
||||
if authCfg.Constraints != nil {
|
||||
authorized.Constraints = types.DataConstraints{
|
||||
OwnerOnly: authCfg.Constraints.OwnerOnly,
|
||||
CreatorOnly: authCfg.Constraints.CreatorOnly,
|
||||
EditorOnly: authCfg.Constraints.EditorOnly,
|
||||
TeamOnly: authCfg.Constraints.TeamOnly,
|
||||
Extra: authCfg.Constraints.Extra,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return authorized
|
||||
}
|
||||
|
||||
// NewTestContextFromOptions creates a test context from test options and test case
|
||||
func NewTestContextFromOptions(chatID, assistantID string, opts *Options, tc *Case) *context.Context {
|
||||
// Get environment from test case (with options override)
|
||||
|
|
|
|||
|
|
@ -257,6 +257,52 @@ func (w *OutputWriter) DirectOutput(output interface{}) {
|
|||
}
|
||||
}
|
||||
|
||||
// ScriptTestSummary prints the script test summary
|
||||
func (w *OutputWriter) ScriptTestSummary(summary *ScriptTestSummary, duration time.Duration) {
|
||||
w.SubHeader("Summary")
|
||||
|
||||
// Results
|
||||
color.New(color.FgWhite).Printf(" Total: ")
|
||||
fmt.Printf("%d\n", summary.Total)
|
||||
|
||||
color.New(color.FgWhite).Printf(" Passed: ")
|
||||
if summary.Passed > 0 {
|
||||
color.New(color.FgGreen).Printf("%d\n", summary.Passed)
|
||||
} else {
|
||||
fmt.Printf("%d\n", summary.Passed)
|
||||
}
|
||||
|
||||
color.New(color.FgWhite).Printf(" Failed: ")
|
||||
if summary.Failed > 0 {
|
||||
color.New(color.FgRed).Printf("%d\n", summary.Failed)
|
||||
} else {
|
||||
fmt.Printf("%d\n", summary.Failed)
|
||||
}
|
||||
|
||||
if summary.Skipped > 0 {
|
||||
color.New(color.FgWhite).Printf(" Skipped: ")
|
||||
color.New(color.FgYellow).Printf("%d\n", summary.Skipped)
|
||||
}
|
||||
|
||||
// Pass rate
|
||||
passRate := float64(0)
|
||||
if summary.Total > 0 {
|
||||
passRate = float64(summary.Passed) / float64(summary.Total) * 100
|
||||
}
|
||||
color.New(color.FgWhite).Printf(" Pass Rate: ")
|
||||
if passRate == 100 {
|
||||
color.New(color.FgGreen, color.Bold).Printf("%.1f%%\n", passRate)
|
||||
} else if passRate >= 80 {
|
||||
color.New(color.FgYellow).Printf("%.1f%%\n", passRate)
|
||||
} else {
|
||||
color.New(color.FgRed).Printf("%.1f%%\n", passRate)
|
||||
}
|
||||
|
||||
// Duration
|
||||
color.New(color.FgWhite).Printf(" Duration: ")
|
||||
fmt.Printf("%s\n", formatDuration(duration))
|
||||
}
|
||||
|
||||
// StabilityResult prints stability analysis result for a test case
|
||||
func (w *OutputWriter) StabilityResult(sr *StabilityResult) {
|
||||
color.New(color.FgWhite).Printf(" [%s] ", sr.ID)
|
||||
|
|
|
|||
|
|
@ -213,8 +213,16 @@ func DefaultOptions() *Options {
|
|||
}
|
||||
|
||||
// DetectInputMode detects the input mode from the input string
|
||||
// Returns InputModeFile if input looks like a file path, InputModeMessage otherwise
|
||||
// Returns:
|
||||
// - InputModeScript: if input starts with "scripts."
|
||||
// - InputModeFile: if input ends with ".jsonl" or is an existing file
|
||||
// - InputModeMessage: otherwise (direct message mode)
|
||||
func DetectInputMode(input string) InputMode {
|
||||
// Check for script test prefix
|
||||
if strings.HasPrefix(input, "scripts.") {
|
||||
return InputModeScript
|
||||
}
|
||||
|
||||
// If input ends with .jsonl or .json, treat as file
|
||||
if strings.HasSuffix(input, ".jsonl") || strings.HasSuffix(input, ".json") {
|
||||
return InputModeFile
|
||||
|
|
@ -269,6 +277,12 @@ func MergeOptions(opts *Options, defaults *Options) *Options {
|
|||
if opts.ReporterID != "" {
|
||||
result.ReporterID = opts.ReporterID
|
||||
}
|
||||
if opts.ContextFile != "" {
|
||||
result.ContextFile = opts.ContextFile
|
||||
}
|
||||
if opts.Run != "" {
|
||||
result.Run = opts.Run
|
||||
}
|
||||
if opts.Verbose {
|
||||
result.Verbose = opts.Verbose
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,6 +33,11 @@ func NewRunner(opts *Options) *Executor {
|
|||
|
||||
// Run executes all test cases and returns a report
|
||||
func (r *Executor) Run() (*Report, error) {
|
||||
// For script test mode, use script runner
|
||||
if r.opts.InputMode == InputModeScript {
|
||||
return r.RunScriptTests()
|
||||
}
|
||||
|
||||
// For direct message mode, use simplified output (development mode)
|
||||
if r.opts.InputMode == InputModeMessage {
|
||||
return r.RunDirect()
|
||||
|
|
@ -41,6 +46,33 @@ func (r *Executor) Run() (*Report, error) {
|
|||
return r.RunTests()
|
||||
}
|
||||
|
||||
// RunScriptTests executes script tests and returns a report
|
||||
func (r *Executor) RunScriptTests() (*Report, error) {
|
||||
scriptRunner := NewScriptRunner(r.opts)
|
||||
scriptReport, err := scriptRunner.Run()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Convert to standard report for unified output handling
|
||||
report := scriptReport.ToReport()
|
||||
|
||||
// Write output if specified
|
||||
if r.opts.OutputFile != "" {
|
||||
err = r.writeOutput(report)
|
||||
if err != nil {
|
||||
r.output.Error("Failed to write output: %s", err.Error())
|
||||
} else {
|
||||
r.output.OutputFile(r.opts.OutputFile)
|
||||
}
|
||||
}
|
||||
|
||||
// Print final result
|
||||
r.output.FinalResult(!report.HasFailures())
|
||||
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// RunDirect executes a single direct message and outputs the result directly
|
||||
// This is optimized for development/debugging scenarios
|
||||
func (r *Executor) RunDirect() (*Report, error) {
|
||||
|
|
|
|||
482
agent/test/script.go
Normal file
482
agent/test/script.go
Normal file
|
|
@ -0,0 +1,482 @@
|
|||
package test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/yaoapp/gou/application"
|
||||
v8 "github.com/yaoapp/gou/runtime/v8"
|
||||
"github.com/yaoapp/gou/runtime/v8/bridge"
|
||||
"github.com/yaoapp/yao/agent/context"
|
||||
"rogchap.com/v8go"
|
||||
)
|
||||
|
||||
// ScriptRunner executes script tests
|
||||
type ScriptRunner struct {
|
||||
opts *Options
|
||||
output *OutputWriter
|
||||
}
|
||||
|
||||
// NewScriptRunner creates a new script test runner
|
||||
func NewScriptRunner(opts *Options) *ScriptRunner {
|
||||
return &ScriptRunner{
|
||||
opts: opts,
|
||||
output: NewOutputWriter(opts.Verbose),
|
||||
}
|
||||
}
|
||||
|
||||
// ResolveScript resolves the script path from scripts.xxx.yyy format
|
||||
func ResolveScript(input string) (*ScriptInfo, error) {
|
||||
// Remove "scripts." prefix
|
||||
path := strings.TrimPrefix(input, "scripts.")
|
||||
|
||||
// Split into parts: "expense.setup" -> ["expense", "setup"]
|
||||
parts := strings.Split(path, ".")
|
||||
if len(parts) < 2 {
|
||||
return nil, fmt.Errorf("invalid script path: %s (expected format: scripts.assistant.module)", input)
|
||||
}
|
||||
|
||||
// Build paths
|
||||
// assistantDir: expense
|
||||
// moduleName: setup
|
||||
// scriptPath: expense/src/setup.ts (or assistants/expense/src/setup.ts)
|
||||
// testPath: expense/src/setup_test.ts
|
||||
assistantDir := parts[0]
|
||||
moduleName := parts[1]
|
||||
|
||||
// Try different path patterns
|
||||
basePaths := []string{
|
||||
filepath.Join("assistants", assistantDir, "src"),
|
||||
filepath.Join(assistantDir, "src"),
|
||||
}
|
||||
|
||||
var scriptPath, testPath string
|
||||
for _, basePath := range basePaths {
|
||||
// Check for TypeScript files first, then JavaScript
|
||||
for _, ext := range []string{".ts", ".js"} {
|
||||
candidateScript := filepath.Join(basePath, moduleName+ext)
|
||||
candidateTest := filepath.Join(basePath, moduleName+"_test"+ext)
|
||||
|
||||
// Check if test file exists
|
||||
exists, err := application.App.Exists(candidateTest)
|
||||
if err == nil && exists {
|
||||
scriptPath = candidateScript
|
||||
testPath = candidateTest
|
||||
break
|
||||
}
|
||||
}
|
||||
if testPath != "" {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if testPath == "" {
|
||||
return nil, fmt.Errorf("test file not found for %s (tried: %s)", input, strings.Join(basePaths, ", "))
|
||||
}
|
||||
|
||||
return &ScriptInfo{
|
||||
ID: input,
|
||||
Assistant: assistantDir,
|
||||
Module: moduleName,
|
||||
ScriptPath: scriptPath,
|
||||
TestPath: testPath,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// DiscoverTests finds all Test* functions in the script
|
||||
func DiscoverTests(scriptPath string) ([]*ScriptTestCase, error) {
|
||||
// Read the script file
|
||||
content, err := application.App.Read(scriptPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read script: %w", err)
|
||||
}
|
||||
|
||||
// Parse the script to find Test* functions
|
||||
// We use a simple regex-like approach to find function declarations
|
||||
tests := make([]*ScriptTestCase, 0)
|
||||
lines := strings.Split(string(content), "\n")
|
||||
|
||||
for _, line := range lines {
|
||||
line = strings.TrimSpace(line)
|
||||
|
||||
// Match function declarations: function TestXxx( or export function TestXxx(
|
||||
if strings.Contains(line, "function Test") {
|
||||
// Extract function name
|
||||
name := extractFunctionName(line)
|
||||
if name != "" && strings.HasPrefix(name, "Test") {
|
||||
tests = append(tests, &ScriptTestCase{
|
||||
Name: name,
|
||||
Function: name,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return tests, nil
|
||||
}
|
||||
|
||||
// extractFunctionName extracts the function name from a line
|
||||
func extractFunctionName(line string) string {
|
||||
// Remove "export" prefix if present
|
||||
line = strings.TrimPrefix(line, "export ")
|
||||
line = strings.TrimSpace(line)
|
||||
|
||||
// Match "function Name("
|
||||
if !strings.HasPrefix(line, "function ") {
|
||||
return ""
|
||||
}
|
||||
|
||||
line = strings.TrimPrefix(line, "function ")
|
||||
|
||||
// Find the opening parenthesis
|
||||
idx := strings.Index(line, "(")
|
||||
if idx == -1 {
|
||||
return ""
|
||||
}
|
||||
|
||||
return strings.TrimSpace(line[:idx])
|
||||
}
|
||||
|
||||
// filterTests filters test cases by a regex pattern (similar to go test -run)
|
||||
func (r *ScriptRunner) filterTests(tests []*ScriptTestCase, pattern string) ([]*ScriptTestCase, error) {
|
||||
re, err := regexp.Compile(pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
filtered := make([]*ScriptTestCase, 0)
|
||||
for _, tc := range tests {
|
||||
if re.MatchString(tc.Name) {
|
||||
filtered = append(filtered, tc)
|
||||
}
|
||||
}
|
||||
|
||||
return filtered, nil
|
||||
}
|
||||
|
||||
// Run executes all script tests and returns a report
|
||||
func (r *ScriptRunner) Run() (*ScriptTestReport, error) {
|
||||
startTime := time.Now()
|
||||
|
||||
// Resolve script
|
||||
scriptInfo, err := ResolveScript(r.opts.Input)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Print header
|
||||
r.output.Header("Script Test")
|
||||
r.output.Info("Script: %s", scriptInfo.TestPath)
|
||||
|
||||
// Discover tests
|
||||
tests, err := DiscoverTests(scriptInfo.TestPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Filter tests by -run pattern if specified
|
||||
if r.opts.Run != "" {
|
||||
tests, err = r.filterTests(tests, r.opts.Run)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid -run pattern: %w", err)
|
||||
}
|
||||
r.output.Info("Tests: %d functions (filtered by: %s)", len(tests), r.opts.Run)
|
||||
} else {
|
||||
r.output.Info("Tests: %d functions", len(tests))
|
||||
}
|
||||
|
||||
if len(tests) == 0 {
|
||||
r.output.Warning("No tests to run")
|
||||
}
|
||||
|
||||
// Load context config if specified
|
||||
var ctxConfig *ContextConfig
|
||||
if r.opts.ContextFile != "" {
|
||||
var err error
|
||||
ctxConfig, err = LoadContextConfig(r.opts.ContextFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to load context file: %w", err)
|
||||
}
|
||||
r.output.Info("Context: %s", r.opts.ContextFile)
|
||||
}
|
||||
|
||||
// Create environment with optional context config
|
||||
var env *Environment
|
||||
if ctxConfig != nil {
|
||||
env = NewEnvironmentWithContext(r.opts.UserID, r.opts.TeamID, ctxConfig)
|
||||
} else {
|
||||
env = NewEnvironment(r.opts.UserID, r.opts.TeamID)
|
||||
}
|
||||
r.output.Info("User: %s", env.UserID)
|
||||
r.output.Info("Team: %s", env.TeamID)
|
||||
|
||||
// Load all scripts from src directory (including the test file)
|
||||
// This ensures imports can be resolved properly
|
||||
srcDir := filepath.Dir(scriptInfo.TestPath)
|
||||
loadedCount, err := r.loadAllScripts(srcDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to load scripts: %w", err)
|
||||
}
|
||||
r.output.Info("Loaded: %d scripts", loadedCount)
|
||||
|
||||
// Create report
|
||||
report := &ScriptTestReport{
|
||||
Type: "script_test",
|
||||
Script: scriptInfo.ID,
|
||||
ScriptPath: scriptInfo.TestPath,
|
||||
Summary: &ScriptTestSummary{Total: len(tests)},
|
||||
Environment: env,
|
||||
Results: make([]*ScriptTestResult, 0, len(tests)),
|
||||
Metadata: &ScriptTestMetadata{
|
||||
StartedAt: startTime,
|
||||
},
|
||||
}
|
||||
|
||||
// Run tests
|
||||
r.output.SubHeader("Running Tests")
|
||||
|
||||
for _, tc := range tests {
|
||||
result := r.runScriptTest(tc, scriptInfo, env)
|
||||
report.Results = append(report.Results, result)
|
||||
|
||||
// Update summary
|
||||
switch result.Status {
|
||||
case StatusPassed:
|
||||
report.Summary.Passed++
|
||||
case StatusFailed:
|
||||
report.Summary.Failed++
|
||||
case StatusSkipped:
|
||||
report.Summary.Skipped++
|
||||
}
|
||||
|
||||
// Check fail-fast
|
||||
if r.opts.FailFast && result.Status == StatusFailed {
|
||||
r.output.Warning("Stopping due to --fail-fast")
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Complete report
|
||||
report.Summary.DurationMs = time.Since(startTime).Milliseconds()
|
||||
report.Metadata.CompletedAt = time.Now()
|
||||
|
||||
// Print summary
|
||||
r.output.ScriptTestSummary(report.Summary, time.Since(startTime))
|
||||
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// runScriptTest runs a single script test function
|
||||
func (r *ScriptRunner) runScriptTest(tc *ScriptTestCase, scriptInfo *ScriptInfo, env *Environment) *ScriptTestResult {
|
||||
r.output.TestStart(tc.Name, "", 1)
|
||||
startTime := time.Now()
|
||||
|
||||
result := &ScriptTestResult{
|
||||
Name: tc.Name,
|
||||
Status: StatusPassed,
|
||||
}
|
||||
|
||||
// Create testing.T object
|
||||
testingT := NewTestingT(tc.Name)
|
||||
|
||||
// Create agent context
|
||||
chatID := fmt.Sprintf("script-test-%s", tc.Name)
|
||||
agentCtx := NewTestContext(chatID, scriptInfo.Assistant, env)
|
||||
defer agentCtx.Release()
|
||||
|
||||
// Execute the test function
|
||||
err := r.executeTestFunction(tc, scriptInfo, testingT, agentCtx)
|
||||
|
||||
duration := time.Since(startTime)
|
||||
result.DurationMs = duration.Milliseconds()
|
||||
result.Logs = testingT.Logs()
|
||||
|
||||
if err != nil {
|
||||
result.Status = StatusError
|
||||
result.Error = err.Error()
|
||||
r.output.TestResult(result.Status, duration)
|
||||
r.output.TestError(result.Error)
|
||||
return result
|
||||
}
|
||||
|
||||
if testingT.Skipped() {
|
||||
result.Status = StatusSkipped
|
||||
r.output.TestResult(result.Status, duration)
|
||||
return result
|
||||
}
|
||||
|
||||
if testingT.Failed() {
|
||||
result.Status = StatusFailed
|
||||
errors := testingT.Errors()
|
||||
if len(errors) > 0 {
|
||||
result.Error = errors[0]
|
||||
}
|
||||
result.Assertion = testingT.AssertionInfo()
|
||||
r.output.TestResult(result.Status, duration)
|
||||
r.output.TestError(result.Error)
|
||||
return result
|
||||
}
|
||||
|
||||
r.output.TestResult(result.Status, duration)
|
||||
return result
|
||||
}
|
||||
|
||||
// loadAllScripts loads all scripts from the src directory
|
||||
// This ensures that imports can be resolved properly
|
||||
func (r *ScriptRunner) loadAllScripts(srcDir string) (int, error) {
|
||||
count := 0
|
||||
|
||||
// Check if src directory exists
|
||||
exists, err := application.App.Exists(srcDir)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if !exists {
|
||||
return 0, fmt.Errorf("src directory not found: %s", srcDir)
|
||||
}
|
||||
|
||||
// Walk through src directory to find all script files
|
||||
exts := []string{"*.ts", "*.js"}
|
||||
|
||||
err = application.App.Walk(srcDir, func(root, file string, isdir bool) error {
|
||||
if isdir {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get relative path
|
||||
relPath := strings.TrimPrefix(file, root+"/")
|
||||
|
||||
// Generate script ID from file path
|
||||
scriptID := generateTestScriptID(file, root)
|
||||
|
||||
// Load the script
|
||||
_, err := v8.Load(file, scriptID)
|
||||
if err != nil {
|
||||
// Log warning but continue loading other scripts
|
||||
if r.opts.Verbose {
|
||||
r.output.Warning("Failed to load %s: %v", relPath, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
count++
|
||||
if r.opts.Verbose {
|
||||
r.output.Verbose("Loaded: %s", relPath)
|
||||
}
|
||||
|
||||
return nil
|
||||
}, exts...)
|
||||
|
||||
if err != nil {
|
||||
return count, fmt.Errorf("failed to walk src directory: %w", err)
|
||||
}
|
||||
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// generateTestScriptID generates a script ID from file path for testing
|
||||
func generateTestScriptID(filePath string, srcDir string) string {
|
||||
// Normalize path separators
|
||||
filePath = filepath.ToSlash(filePath)
|
||||
srcDir = filepath.ToSlash(srcDir)
|
||||
|
||||
// Remove src directory prefix
|
||||
relPath := strings.TrimPrefix(filePath, srcDir+"/")
|
||||
relPath = strings.TrimPrefix(relPath, "/")
|
||||
|
||||
// Remove file extension
|
||||
relPath = strings.TrimSuffix(relPath, filepath.Ext(relPath))
|
||||
|
||||
// Replace path separators with dots and add test prefix
|
||||
scriptID := "test." + strings.ReplaceAll(relPath, "/", ".")
|
||||
|
||||
return scriptID
|
||||
}
|
||||
|
||||
// executeTestFunction executes a single test function using V8
|
||||
func (r *ScriptRunner) executeTestFunction(tc *ScriptTestCase, scriptInfo *ScriptInfo, testingT *TestingT, agentCtx *context.Context) error {
|
||||
// Get the test script (already loaded by loadAllScripts)
|
||||
testScriptID := generateTestScriptID(scriptInfo.TestPath, filepath.Dir(scriptInfo.TestPath))
|
||||
script, ok := v8.Scripts[testScriptID]
|
||||
if !ok {
|
||||
return fmt.Errorf("test script not found: %s (id: %s)", scriptInfo.TestPath, testScriptID)
|
||||
}
|
||||
|
||||
// Create a new script context
|
||||
scriptCtx, err := script.NewContext("", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create script context: %w", err)
|
||||
}
|
||||
defer scriptCtx.Close()
|
||||
|
||||
// Get the V8 context
|
||||
v8ctx := scriptCtx.Context
|
||||
|
||||
// Create testing.T JavaScript object
|
||||
testingTObj, err := NewTestingTObject(v8ctx, testingT)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create testing.T object: %w", err)
|
||||
}
|
||||
|
||||
// Create agent context JavaScript object
|
||||
agentCtxObj, err := agentCtx.JsValue(v8ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create agent context object: %w", err)
|
||||
}
|
||||
|
||||
// Get the test function
|
||||
global := v8ctx.Global()
|
||||
fnValue, err := global.Get(tc.Function)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get test function %s: %w", tc.Function, err)
|
||||
}
|
||||
|
||||
if !fnValue.IsFunction() {
|
||||
return fmt.Errorf("test function %s is not a function", tc.Function)
|
||||
}
|
||||
|
||||
fn, err := fnValue.AsFunction()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to convert to function: %w", err)
|
||||
}
|
||||
|
||||
// Call the test function with (t, ctx)
|
||||
_, err = fn.Call(global, testingTObj, agentCtxObj)
|
||||
if err != nil {
|
||||
// Check if this is an assertion failure or a real error
|
||||
if testingT.Failed() {
|
||||
// Assertion failure - already recorded
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("test function error: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// RegisterTestingGlobals registers testing-related global functions for V8
|
||||
// This is called once during initialization
|
||||
func RegisterTestingGlobals() {
|
||||
v8.RegisterFunction("__testing_log", testingLogEmbed)
|
||||
}
|
||||
|
||||
// testingLogEmbed provides a console.log-like function for tests
|
||||
func testingLogEmbed(iso *v8go.Isolate) *v8go.FunctionTemplate {
|
||||
return v8go.NewFunctionTemplate(iso, func(info *v8go.FunctionCallbackInfo) *v8go.Value {
|
||||
args := info.Args()
|
||||
parts := make([]string, len(args))
|
||||
for i, arg := range args {
|
||||
goVal, err := bridge.GoValue(arg, info.Context())
|
||||
if err != nil {
|
||||
parts[i] = arg.String()
|
||||
} else {
|
||||
parts[i] = fmt.Sprintf("%v", goVal)
|
||||
}
|
||||
}
|
||||
fmt.Println(strings.Join(parts, " "))
|
||||
return v8go.Undefined(iso)
|
||||
})
|
||||
}
|
||||
1000
agent/test/script_assert.go
Normal file
1000
agent/test/script_assert.go
Normal file
File diff suppressed because it is too large
Load diff
170
agent/test/script_types.go
Normal file
170
agent/test/script_types.go
Normal file
|
|
@ -0,0 +1,170 @@
|
|||
package test
|
||||
|
||||
import "time"
|
||||
|
||||
// ScriptInfo contains information about the script being tested
|
||||
type ScriptInfo struct {
|
||||
// ID is the script identifier (e.g., "scripts.expense.setup")
|
||||
ID string `json:"id"`
|
||||
|
||||
// Assistant is the assistant directory name (e.g., "expense")
|
||||
Assistant string `json:"assistant"`
|
||||
|
||||
// Module is the module name (e.g., "setup")
|
||||
Module string `json:"module"`
|
||||
|
||||
// ScriptPath is the path to the main script file (e.g., "expense/src/setup.ts")
|
||||
ScriptPath string `json:"script_path"`
|
||||
|
||||
// TestPath is the path to the test script file (e.g., "expense/src/setup_test.ts")
|
||||
TestPath string `json:"test_path"`
|
||||
}
|
||||
|
||||
// ScriptTestCase represents a single script test function
|
||||
type ScriptTestCase struct {
|
||||
// Name is the test function name (e.g., "TestSystemReady")
|
||||
Name string `json:"name"`
|
||||
|
||||
// Function is the full function reference
|
||||
Function string `json:"function"`
|
||||
}
|
||||
|
||||
// ScriptTestResult represents the result of running a script test function
|
||||
type ScriptTestResult struct {
|
||||
// Name is the test function name
|
||||
Name string `json:"name"`
|
||||
|
||||
// Status is the test execution status
|
||||
Status Status `json:"status"`
|
||||
|
||||
// DurationMs is the execution duration in milliseconds
|
||||
DurationMs int64 `json:"duration_ms"`
|
||||
|
||||
// Error contains the error message if the test failed
|
||||
Error string `json:"error,omitempty"`
|
||||
|
||||
// Assertion contains assertion failure details
|
||||
Assertion *ScriptAssertionInfo `json:"assertion,omitempty"`
|
||||
|
||||
// Logs contains log messages from the test
|
||||
Logs []string `json:"logs,omitempty"`
|
||||
}
|
||||
|
||||
// ScriptAssertionInfo contains details about an assertion failure
|
||||
type ScriptAssertionInfo struct {
|
||||
// Type is the assertion type (e.g., "Equal", "True")
|
||||
Type string `json:"type"`
|
||||
|
||||
// Expected is the expected value
|
||||
Expected interface{} `json:"expected,omitempty"`
|
||||
|
||||
// Actual is the actual value
|
||||
Actual interface{} `json:"actual,omitempty"`
|
||||
|
||||
// Message is the custom failure message
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
|
||||
// ScriptTestSummary contains aggregated statistics for script tests
|
||||
type ScriptTestSummary struct {
|
||||
// Total number of test functions
|
||||
Total int `json:"total"`
|
||||
|
||||
// Passed number of test functions that passed
|
||||
Passed int `json:"passed"`
|
||||
|
||||
// Failed number of test functions that failed
|
||||
Failed int `json:"failed"`
|
||||
|
||||
// Skipped number of test functions that were skipped
|
||||
Skipped int `json:"skipped"`
|
||||
|
||||
// DurationMs is the total execution duration in milliseconds
|
||||
DurationMs int64 `json:"duration_ms"`
|
||||
}
|
||||
|
||||
// ScriptTestReport represents the complete script test report
|
||||
type ScriptTestReport struct {
|
||||
// Type indicates this is a script test report
|
||||
Type string `json:"type"` // "script_test"
|
||||
|
||||
// Script is the script identifier (e.g., "scripts.expense.setup")
|
||||
Script string `json:"script"`
|
||||
|
||||
// ScriptPath is the path to the test script file
|
||||
ScriptPath string `json:"script_path"`
|
||||
|
||||
// Summary contains aggregated statistics
|
||||
Summary *ScriptTestSummary `json:"summary"`
|
||||
|
||||
// Environment contains the test environment configuration
|
||||
Environment *Environment `json:"environment"`
|
||||
|
||||
// Results contains individual test results
|
||||
Results []*ScriptTestResult `json:"results"`
|
||||
|
||||
// Metadata contains additional report metadata
|
||||
Metadata *ScriptTestMetadata `json:"metadata"`
|
||||
}
|
||||
|
||||
// ScriptTestMetadata contains metadata about the script test report
|
||||
type ScriptTestMetadata struct {
|
||||
// StartedAt is when the test run started
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
|
||||
// CompletedAt is when the test run completed
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
|
||||
// Version is the Yao version
|
||||
Version string `json:"version"`
|
||||
}
|
||||
|
||||
// HasFailures returns true if there are any failed tests
|
||||
func (r *ScriptTestReport) HasFailures() bool {
|
||||
return r.Summary.Failed > 0
|
||||
}
|
||||
|
||||
// PassRate returns the pass rate as a percentage (0-100)
|
||||
func (r *ScriptTestReport) PassRate() float64 {
|
||||
if r.Summary.Total == 0 {
|
||||
return 0
|
||||
}
|
||||
return float64(r.Summary.Passed) / float64(r.Summary.Total) * 100
|
||||
}
|
||||
|
||||
// ToReport converts ScriptTestReport to a standard Report for unified reporting
|
||||
func (r *ScriptTestReport) ToReport() *Report {
|
||||
return &Report{
|
||||
Summary: &Summary{
|
||||
Total: r.Summary.Total,
|
||||
Passed: r.Summary.Passed,
|
||||
Failed: r.Summary.Failed,
|
||||
Skipped: r.Summary.Skipped,
|
||||
DurationMs: r.Summary.DurationMs,
|
||||
AgentID: r.Script,
|
||||
AgentPath: r.ScriptPath,
|
||||
},
|
||||
Environment: r.Environment,
|
||||
Results: r.toResults(),
|
||||
Metadata: &ReportMetadata{
|
||||
StartedAt: r.Metadata.StartedAt,
|
||||
CompletedAt: r.Metadata.CompletedAt,
|
||||
Version: r.Metadata.Version,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// toResults converts script test results to standard results
|
||||
func (r *ScriptTestReport) toResults() []*Result {
|
||||
results := make([]*Result, len(r.Results))
|
||||
for i, sr := range r.Results {
|
||||
results[i] = &Result{
|
||||
ID: sr.Name,
|
||||
Status: sr.Status,
|
||||
Input: sr.Name,
|
||||
DurationMs: sr.DurationMs,
|
||||
Error: sr.Error,
|
||||
}
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
|
@ -1,7 +1,10 @@
|
|||
package test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/yaoapp/yao/agent/context"
|
||||
|
|
@ -57,6 +60,8 @@ const (
|
|||
InputModeFile InputMode = "file"
|
||||
// InputModeMessage indicates input from a direct message string
|
||||
InputModeMessage InputMode = "message"
|
||||
// InputModeScript indicates script test mode (testing agent handler scripts)
|
||||
InputModeScript InputMode = "script"
|
||||
)
|
||||
|
||||
// Options represents the configuration options for running tests
|
||||
|
|
@ -96,6 +101,14 @@ type Options struct {
|
|||
// Locale is the locale for the test context (default: "en-us")
|
||||
Locale string `json:"locale,omitempty"`
|
||||
|
||||
// ContextFile is the path to a JSON file containing custom context data (-ctx flag)
|
||||
// This allows full customization of authorized info, metadata, etc.
|
||||
ContextFile string `json:"context_file,omitempty"`
|
||||
|
||||
// ContextData is the parsed context data from ContextFile
|
||||
// This is populated internally after loading the file
|
||||
ContextData *ContextConfig `json:"-"`
|
||||
|
||||
// Execution
|
||||
// ===============================
|
||||
|
||||
|
|
@ -126,6 +139,92 @@ type Options struct {
|
|||
|
||||
// FailFast stops execution on first failure
|
||||
FailFast bool `json:"fail_fast,omitempty"`
|
||||
|
||||
// Run is a regex pattern to filter which tests to run (similar to go test -run)
|
||||
// Only tests matching the pattern will be executed
|
||||
// Example: "TestSystem" matches TestSystemReady, TestSystemError, etc.
|
||||
Run string `json:"run,omitempty"`
|
||||
}
|
||||
|
||||
// ContextConfig represents custom context configuration from JSON file
|
||||
// This allows full customization of the test context including authorized info
|
||||
type ContextConfig struct {
|
||||
// Authorized contains custom authorization data
|
||||
Authorized *AuthorizedConfig `json:"authorized,omitempty"`
|
||||
|
||||
// Metadata contains custom metadata to pass to the context
|
||||
Metadata map[string]interface{} `json:"metadata,omitempty"`
|
||||
|
||||
// Client contains custom client information
|
||||
Client *ClientConfig `json:"client,omitempty"`
|
||||
|
||||
// Locale overrides the locale setting
|
||||
Locale string `json:"locale,omitempty"`
|
||||
|
||||
// Referer overrides the referer setting
|
||||
Referer string `json:"referer,omitempty"`
|
||||
}
|
||||
|
||||
// AuthorizedConfig represents custom authorization configuration
|
||||
// Matches the structure of types.AuthorizedInfo from openapi/oauth/types
|
||||
type AuthorizedConfig struct {
|
||||
// Sub is the subject identifier (JWT sub claim)
|
||||
Sub string `json:"sub,omitempty"`
|
||||
|
||||
// ClientID is the OAuth client ID
|
||||
ClientID string `json:"client_id,omitempty"`
|
||||
|
||||
// Scope is the access scope
|
||||
Scope string `json:"scope,omitempty"`
|
||||
|
||||
// SessionID is the session identifier
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
|
||||
// UserID is the user identifier
|
||||
UserID string `json:"user_id,omitempty"`
|
||||
|
||||
// TeamID is the team identifier
|
||||
TeamID string `json:"team_id,omitempty"`
|
||||
|
||||
// TenantID is the tenant identifier
|
||||
TenantID string `json:"tenant_id,omitempty"`
|
||||
|
||||
// RememberMe is the remember me flag
|
||||
RememberMe bool `json:"remember_me,omitempty"`
|
||||
|
||||
// Constraints contains data access constraints (set by ACL enforcement)
|
||||
Constraints *DataConstraintsConfig `json:"constraints,omitempty"`
|
||||
}
|
||||
|
||||
// DataConstraintsConfig represents data access constraints
|
||||
// Matches the structure of types.DataConstraints from openapi/oauth/types
|
||||
type DataConstraintsConfig struct {
|
||||
// OwnerOnly - only access owner's data
|
||||
OwnerOnly bool `json:"owner_only,omitempty"`
|
||||
|
||||
// CreatorOnly - only access creator's data
|
||||
CreatorOnly bool `json:"creator_only,omitempty"`
|
||||
|
||||
// EditorOnly - only access editor's data
|
||||
EditorOnly bool `json:"editor_only,omitempty"`
|
||||
|
||||
// TeamOnly - only access team's data (filter by team_id)
|
||||
TeamOnly bool `json:"team_only,omitempty"`
|
||||
|
||||
// Extra contains user-defined constraints (department, region, etc.)
|
||||
Extra map[string]interface{} `json:"extra,omitempty"`
|
||||
}
|
||||
|
||||
// ClientConfig represents custom client configuration
|
||||
type ClientConfig struct {
|
||||
// Type is the client type (e.g., "web", "mobile", "test")
|
||||
Type string `json:"type,omitempty"`
|
||||
|
||||
// UserAgent is the client user agent string
|
||||
UserAgent string `json:"user_agent,omitempty"`
|
||||
|
||||
// IP is the client IP address
|
||||
IP string `json:"ip,omitempty"`
|
||||
}
|
||||
|
||||
// Environment configures the test execution context
|
||||
|
|
@ -150,6 +249,9 @@ type Environment struct {
|
|||
|
||||
// Accept is the accept format (default: "standard")
|
||||
Accept string `json:"accept"`
|
||||
|
||||
// ContextConfig contains custom context configuration (from -ctx flag)
|
||||
ContextConfig *ContextConfig `json:"-"`
|
||||
}
|
||||
|
||||
// NewEnvironment creates a new test environment with defaults
|
||||
|
|
@ -175,6 +277,61 @@ func NewEnvironment(userID, teamID string) *Environment {
|
|||
return env
|
||||
}
|
||||
|
||||
// NewEnvironmentWithContext creates a new test environment with custom context config
|
||||
func NewEnvironmentWithContext(userID, teamID string, ctxConfig *ContextConfig) *Environment {
|
||||
env := NewEnvironment(userID, teamID)
|
||||
|
||||
if ctxConfig == nil {
|
||||
return env
|
||||
}
|
||||
|
||||
env.ContextConfig = ctxConfig
|
||||
|
||||
// Override with context config values
|
||||
if ctxConfig.Locale != "" {
|
||||
env.Locale = ctxConfig.Locale
|
||||
}
|
||||
if ctxConfig.Referer != "" {
|
||||
env.Referer = ctxConfig.Referer
|
||||
}
|
||||
if ctxConfig.Client != nil {
|
||||
if ctxConfig.Client.Type != "" {
|
||||
env.ClientType = ctxConfig.Client.Type
|
||||
}
|
||||
if ctxConfig.Client.IP != "" {
|
||||
env.ClientIP = ctxConfig.Client.IP
|
||||
}
|
||||
}
|
||||
if ctxConfig.Authorized != nil {
|
||||
if ctxConfig.Authorized.UserID != "" {
|
||||
env.UserID = ctxConfig.Authorized.UserID
|
||||
}
|
||||
// TeamID takes precedence over TenantID for team override
|
||||
if ctxConfig.Authorized.TeamID != "" {
|
||||
env.TeamID = ctxConfig.Authorized.TeamID
|
||||
} else if ctxConfig.Authorized.TenantID != "" {
|
||||
env.TeamID = ctxConfig.Authorized.TenantID
|
||||
}
|
||||
}
|
||||
|
||||
return env
|
||||
}
|
||||
|
||||
// LoadContextConfig loads context configuration from a JSON file
|
||||
func LoadContextConfig(filePath string) (*ContextConfig, error) {
|
||||
data, err := os.ReadFile(filePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read context file: %w", err)
|
||||
}
|
||||
|
||||
var config ContextConfig
|
||||
if err := json.Unmarshal(data, &config); err != nil {
|
||||
return nil, fmt.Errorf("failed to parse context file: %w", err)
|
||||
}
|
||||
|
||||
return &config, nil
|
||||
}
|
||||
|
||||
// Case represents a single test case loaded from JSONL
|
||||
type Case struct {
|
||||
// ID is the unique identifier for this test case (e.g., "T001")
|
||||
|
|
|
|||
177
cmd/README.md
177
cmd/README.md
|
|
@ -14,17 +14,17 @@ go install github.com/yaoapp/yao@latest
|
|||
|
||||
## Global Flags
|
||||
|
||||
| Flag | Short | Description |
|
||||
|------|-------|-------------|
|
||||
| `--app` | `-a` | Application directory path |
|
||||
| `--file` | `-f` | Application package file (.yaz) |
|
||||
| `--key` | `-k` | Application license key |
|
||||
| Flag | Short | Description |
|
||||
| -------- | ----- | ------------------------------- |
|
||||
| `--app` | `-a` | Application directory path |
|
||||
| `--file` | `-f` | Application package file (.yaz) |
|
||||
| `--key` | `-k` | Application license key |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `YAO_ROOT` | Application root directory |
|
||||
| Variable | Description |
|
||||
| ---------- | -------------------------------------------- |
|
||||
| `YAO_ROOT` | Application root directory |
|
||||
| `YAO_LANG` | Language setting (e.g., `zh-CN` for Chinese) |
|
||||
|
||||
## Commands
|
||||
|
|
@ -46,10 +46,10 @@ yao start --debug
|
|||
|
||||
**Flags:**
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--debug` | Enable development/debug mode |
|
||||
| `--disable-watching` | Disable file watching |
|
||||
| Flag | Description |
|
||||
| -------------------- | ----------------------------- |
|
||||
| `--debug` | Enable development/debug mode |
|
||||
| `--disable-watching` | Disable file watching |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -70,9 +70,9 @@ yao run -s models.user.Find 1
|
|||
|
||||
**Flags:**
|
||||
|
||||
| Flag | Short | Description |
|
||||
|------|-------|-------------|
|
||||
| `--silent` | `-s` | Silent mode - output result as JSON only |
|
||||
| Flag | Short | Description |
|
||||
| ---------- | ----- | ---------------------------------------- |
|
||||
| `--silent` | `-s` | Silent mode - output result as JSON only |
|
||||
|
||||
**Argument Syntax:**
|
||||
|
||||
|
|
@ -102,11 +102,11 @@ yao migrate --reset
|
|||
|
||||
**Flags:**
|
||||
|
||||
| Flag | Short | Description |
|
||||
|------|-------|-------------|
|
||||
| `--name` | `-n` | Specific model name to migrate |
|
||||
| `--force` | | Force migrate in production mode |
|
||||
| `--reset` | | Drop tables before migration |
|
||||
| Flag | Short | Description |
|
||||
| --------- | ----- | -------------------------------- |
|
||||
| `--name` | `-n` | Specific model name to migrate |
|
||||
| `--force` | | Force migrate in production mode |
|
||||
| `--reset` | | Drop tables before migration |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -134,8 +134,8 @@ yao version --all
|
|||
|
||||
**Flags:**
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| Flag | Description |
|
||||
| ------- | -------------------------------------------------------------------- |
|
||||
| `--all` | Print all version information (Go version, commit, build time, etc.) |
|
||||
|
||||
---
|
||||
|
|
@ -146,7 +146,7 @@ Commands for testing and managing AI agents.
|
|||
|
||||
### `yao agent test`
|
||||
|
||||
Test an agent with input cases from a JSONL file or direct message.
|
||||
Test an agent with input cases from a JSONL file, direct message, or script tests.
|
||||
|
||||
```bash
|
||||
# Test with direct message (development mode)
|
||||
|
|
@ -169,45 +169,104 @@ yao agent test -i tests/inputs.jsonl --parallel 4
|
|||
|
||||
# Verbose output
|
||||
yao agent test -i tests/inputs.jsonl -v
|
||||
|
||||
# Script tests (test agent handler scripts)
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Script tests with test filtering
|
||||
yao agent test -i scripts.expense.setup --run "TestSystemReady"
|
||||
|
||||
# Script tests with custom context
|
||||
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
|
||||
```
|
||||
|
||||
**Flags:**
|
||||
|
||||
| Flag | Short | Description |
|
||||
|------|-------|-------------|
|
||||
| `--input` | `-i` | Input: JSONL file path or direct message (required) |
|
||||
| `--output` | `-o` | Output file path (default: `output-{timestamp}.jsonl`) |
|
||||
| `--name` | `-n` | Agent ID (default: auto-detect from path) |
|
||||
| `--connector` | `-c` | Override default connector |
|
||||
| `--user` | `-u` | Test user ID (default: `test-user`) |
|
||||
| `--team` | `-t` | Test team ID (default: `test-team`) |
|
||||
| `--reporter` | `-r` | Reporter agent ID for custom report generation |
|
||||
| `--runs` | | Number of runs per test case for stability analysis (default: 1) |
|
||||
| `--timeout` | | Timeout per test case (default: `5m`) |
|
||||
| `--parallel` | | Number of parallel test cases (default: 1) |
|
||||
| `--verbose` | `-v` | Enable verbose output |
|
||||
| `--fail-fast` | | Stop on first failure |
|
||||
| `--app` | `-a` | Application directory |
|
||||
| `--env` | `-e` | Environment file |
|
||||
| Flag | Short | Description |
|
||||
| ------------- | ----- | ---------------------------------------------------------------- |
|
||||
| `--input` | `-i` | Input: JSONL file path, message, or script ID (required) |
|
||||
| `--output` | `-o` | Output file path (default: `output-{timestamp}.jsonl`) |
|
||||
| `--name` | `-n` | Agent ID (default: auto-detect from path) |
|
||||
| `--connector` | `-c` | Override default connector |
|
||||
| `--user` | `-u` | Test user ID (default: `test-user`) |
|
||||
| `--team` | `-t` | Test team ID (default: `test-team`) |
|
||||
| `--ctx` | | Path to context JSON file for custom authorization |
|
||||
| `--reporter` | `-r` | Reporter agent ID for custom report generation |
|
||||
| `--runs` | | Number of runs per test case for stability analysis (default: 1) |
|
||||
| `--run` | | Regex pattern to filter which tests to run |
|
||||
| `--timeout` | | Timeout per test case (default: `5m`) |
|
||||
| `--parallel` | | Number of parallel test cases (default: 1) |
|
||||
| `--verbose` | `-v` | Enable verbose output |
|
||||
| `--fail-fast` | | Stop on first failure |
|
||||
| `--app` | `-a` | Application directory |
|
||||
| `--env` | `-e` | Environment file |
|
||||
|
||||
**Input Modes:**
|
||||
|
||||
1. **Direct Message Mode**: For quick development/debugging
|
||||
|
||||
```bash
|
||||
yao agent test -i "Hello world" -n my.agent
|
||||
```
|
||||
|
||||
- Outputs result directly to stdout
|
||||
- No report file generated
|
||||
- Ideal for iterative development
|
||||
|
||||
2. **File Mode**: For comprehensive testing
|
||||
|
||||
```bash
|
||||
yao agent test -i tests/inputs.jsonl
|
||||
```
|
||||
|
||||
- Reads test cases from JSONL file
|
||||
- Generates detailed report
|
||||
- Supports stability analysis
|
||||
|
||||
3. **Script Test Mode**: For testing agent handler scripts
|
||||
```bash
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
```
|
||||
- Tests TypeScript/JavaScript handler scripts (hooks, tools, setup functions)
|
||||
- Input format: `scripts.<assistant>.<module>` (e.g., `scripts.expense.setup`)
|
||||
- Automatically discovers and runs all `Test*` functions
|
||||
- Uses Go-like testing interface with assertions
|
||||
|
||||
**Script Test Function Signature:**
|
||||
|
||||
```typescript
|
||||
// assistants/expense/src/setup_test.ts
|
||||
import { SystemReady } from "./setup";
|
||||
|
||||
export function TestSystemReady(t: testing.T, ctx: agent.Context) {
|
||||
const result = SystemReady(ctx);
|
||||
t.assert.True(result.success, "SystemReady should succeed");
|
||||
t.assert.Equal(result.status, "ready", "Status should be ready");
|
||||
}
|
||||
```
|
||||
|
||||
**Context JSON Format (for `--ctx` flag):**
|
||||
|
||||
```json
|
||||
{
|
||||
"authorized": {
|
||||
"sub": "user-12345",
|
||||
"client_id": "my-app",
|
||||
"user_id": "admin",
|
||||
"team_id": "team-001",
|
||||
"tenant_id": "acme-corp",
|
||||
"constraints": {
|
||||
"owner_only": true,
|
||||
"team_only": false,
|
||||
"extra": { "department": "engineering" }
|
||||
}
|
||||
},
|
||||
"metadata": { "request_id": "req-123" },
|
||||
"client": { "type": "web", "ip": "192.168.1.100" },
|
||||
"locale": "zh-cn"
|
||||
}
|
||||
```
|
||||
|
||||
**JSONL Input Format:**
|
||||
|
||||
```jsonl
|
||||
|
|
@ -221,12 +280,12 @@ yao agent test -i tests/inputs.jsonl -v
|
|||
|
||||
**Output Formats:**
|
||||
|
||||
| Extension | Format | Description |
|
||||
|-----------|--------|-------------|
|
||||
| `.jsonl` | JSONL | Streaming format (default) |
|
||||
| `.json` | JSON | Complete structured report |
|
||||
| `.md` | Markdown | Human-readable with tables |
|
||||
| `.html` | HTML | Interactive web report |
|
||||
| Extension | Format | Description |
|
||||
| --------- | -------- | -------------------------- |
|
||||
| `.jsonl` | JSONL | Streaming format (default) |
|
||||
| `.json` | JSON | Complete structured report |
|
||||
| `.md` | Markdown | Human-readable with tables |
|
||||
| `.html` | HTML | Interactive web report |
|
||||
|
||||
**Agent Resolution:**
|
||||
|
||||
|
|
@ -281,11 +340,11 @@ yao sui trans default index -l "en-US,zh-CN,ja-JP"
|
|||
|
||||
**SUI Flags:**
|
||||
|
||||
| Flag | Short | Description |
|
||||
|------|-------|-------------|
|
||||
| `--data` | `-d` | Session data as JSON (prefix with `::`) |
|
||||
| `--debug` | `-D` | Enable debug mode |
|
||||
| `--locales` | `-l` | Locales for translation (comma-separated) |
|
||||
| Flag | Short | Description |
|
||||
| ----------- | ----- | ----------------------------------------- |
|
||||
| `--data` | `-d` | Session data as JSON (prefix with `::`) |
|
||||
| `--debug` | `-D` | Enable debug mode |
|
||||
| `--locales` | `-l` | Locales for translation (comma-separated) |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -313,6 +372,15 @@ yao sui watch default home
|
|||
# Run comprehensive agent tests
|
||||
yao agent test -i tests/inputs.jsonl -o report.html -v
|
||||
|
||||
# Run script tests for agent handlers
|
||||
yao agent test -i scripts.expense.setup -v
|
||||
|
||||
# Run specific script tests with filtering
|
||||
yao agent test -i scripts.expense.setup --run "TestSystem.*" -v
|
||||
|
||||
# Run script tests with custom context
|
||||
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
|
||||
|
||||
# Stability analysis (run each test 10 times)
|
||||
yao agent test -i tests/inputs.jsonl --runs 10 -o stability-report.json
|
||||
|
||||
|
|
@ -337,10 +405,10 @@ yao migrate -n user --reset --force
|
|||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Description |
|
||||
|------|-------------|
|
||||
| 0 | Success |
|
||||
| 1 | Error or test failure |
|
||||
| Code | Description |
|
||||
| ---- | --------------------- |
|
||||
| 0 | Success |
|
||||
| 1 | Error or test failure |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -370,4 +438,3 @@ myapp/
|
|||
- [Yao Documentation](https://yaoapps.com/docs)
|
||||
- [Agent Test Design](../agent/test/DESIGN.md)
|
||||
- [SUI Documentation](https://yaoapps.com/docs/sui)
|
||||
|
||||
|
|
|
|||
|
|
@ -22,8 +22,10 @@ var langs = map[string]string{
|
|||
"Override connector": "覆盖连接器",
|
||||
"Test user ID (default: test-user)": "测试用户 ID (默认: test-user)",
|
||||
"Test team ID (default: test-team)": "测试团队 ID (默认: test-team)",
|
||||
"Path to context JSON file for custom authorization": "自定义认证信息的 JSON 文件路径",
|
||||
"Reporter agent ID for custom report": "自定义报告生成器智能体 ID",
|
||||
"Number of runs for stability analysis": "稳定性分析的运行次数",
|
||||
"Regex pattern to filter which tests to run": "用于过滤测试的正则表达式",
|
||||
"Default timeout per test case": "每个测试用例的默认超时时间",
|
||||
"Number of parallel test cases": "并行测试用例数",
|
||||
"Verbose output": "详细输出",
|
||||
|
|
|
|||
|
|
@ -25,8 +25,10 @@ var (
|
|||
testConnector string
|
||||
testUser string
|
||||
testTeam string
|
||||
testContext string // --ctx flag for custom context JSON file
|
||||
testReporter string
|
||||
testRuns int
|
||||
testRun string // --run flag for test filtering (regex pattern)
|
||||
testTimeout string
|
||||
testParallel int
|
||||
testVerbose bool
|
||||
|
|
@ -140,19 +142,21 @@ var TestCmd = &cobra.Command{
|
|||
|
||||
// Build test options
|
||||
opts := &test.Options{
|
||||
Input: testInput,
|
||||
InputMode: inputMode,
|
||||
OutputFile: testOutput,
|
||||
AgentID: testAgent,
|
||||
Connector: testConnector,
|
||||
UserID: testUser,
|
||||
TeamID: testTeam,
|
||||
ReporterID: testReporter,
|
||||
Runs: testRuns,
|
||||
Timeout: timeout,
|
||||
Parallel: testParallel,
|
||||
Verbose: testVerbose,
|
||||
FailFast: testFailFast,
|
||||
Input: testInput,
|
||||
InputMode: inputMode,
|
||||
OutputFile: testOutput,
|
||||
AgentID: testAgent,
|
||||
Connector: testConnector,
|
||||
UserID: testUser,
|
||||
TeamID: testTeam,
|
||||
ContextFile: testContext,
|
||||
ReporterID: testReporter,
|
||||
Runs: testRuns,
|
||||
Run: testRun,
|
||||
Timeout: timeout,
|
||||
Parallel: testParallel,
|
||||
Verbose: testVerbose,
|
||||
FailFast: testFailFast,
|
||||
}
|
||||
|
||||
// Merge with defaults
|
||||
|
|
@ -232,8 +236,10 @@ func init() {
|
|||
TestCmd.Flags().StringVarP(&testConnector, "connector", "c", "", L("Override connector"))
|
||||
TestCmd.Flags().StringVarP(&testUser, "user", "u", "", L("Test user ID (default: test-user)"))
|
||||
TestCmd.Flags().StringVarP(&testTeam, "team", "t", "", L("Test team ID (default: test-team)"))
|
||||
TestCmd.Flags().StringVar(&testContext, "ctx", "", L("Path to context JSON file for custom authorization"))
|
||||
TestCmd.Flags().StringVarP(&testReporter, "reporter", "r", "", L("Reporter agent ID for custom report"))
|
||||
TestCmd.Flags().IntVar(&testRuns, "runs", 1, L("Number of runs for stability analysis"))
|
||||
TestCmd.Flags().StringVar(&testRun, "run", "", L("Regex pattern to filter which tests to run"))
|
||||
TestCmd.Flags().StringVar(&testTimeout, "timeout", "5m", L("Default timeout per test case"))
|
||||
TestCmd.Flags().IntVar(&testParallel, "parallel", 1, L("Number of parallel test cases"))
|
||||
TestCmd.Flags().BoolVarP(&testVerbose, "verbose", "v", false, L("Verbose output"))
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue