yao/sandbox/v2/TEST.md

Sandbox V2 — Test Specification

Design: DESIGN.md | Implementation: IMPL.md

Principles

• Black-box testing: all *_test.go files use package sandbox_test — tests only access exported API
• Real containers: tests create real Docker containers via tai SDK, no mocking
• Skip when unavailable: skipIfNoDocker(t) / skipIfNoTai(t) — CI has Docker and Tai; local dev may not
• Tests follow implementation: *_test.go lives next to the code it tests
• Coverage > 80%: per file and overall

Prerequisites

source $YAO_SOURCE_ROOT/env.local.sh

Docker (required for all container tests)

Docker daemon must be running. Tests connect via default socket.

Tai (required for remote-mode tests only)

docker run -d --name tai \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -p 2375:2375 -p 9100:9100 -p 8080:8080 -p 6080:6080 \
  yaoapp/tai:latest

Environment Variables

Variable	Purpose	Default
YAO_TEST_APPLICATION	Path to yao-dev-app	— (required)
YAO_DB_DRIVER / YAO_DB_PRIMARY	Database connection	— (required)
YAO_JWT_SECRET / YAO_DB_AESKEY	Crypto keys (for OAuth token creation)	— (required)
SANDBOX_TEST_IMAGE	Container image for tests	yaoapp/sandbox-v2-test:latest
SANDBOX_TEST_REMOTE_ADDR	Tai remote address, e.g. tai://127.0.0.1	— (skip remote tests if empty)
TAI_TEST_HOST	Tai HTTP proxy host	127.0.0.1

Directory Structure

sandbox/v2/
├── sandbox.go
├── sandbox_test.go             # Init/M singleton tests
├── manager.go
├── manager_test.go             # Create/Get/GetOrCreate/List/Remove, pool management
├── manager_lifecycle_test.go   # Start recovery, Cleanup, idle tracking, heartbeat
├── box.go
├── box_test.go                 # Exec/Stream/Workspace/Proxy/VNC, lifecycle
├── box_attach_test.go          # Attach WS/SSE (needs service in container)
├── config.go
├── types.go
├── errors.go
├── grpc.go
├── grpc_test.go                # OAuth token creation/revocation, env var building
├── testutils_test.go           # shared test helpers (unexported, package sandbox_test)
└── DESIGN.md

testutils (internal to sandbox_test)

Shared helpers in testutils_test.go — not a separate package, lives inside package sandbox_test.

// testutils_test.go
package sandbox_test

// skipIfNoDocker skips the test if Docker is not available.
func skipIfNoDocker(t *testing.T)

// skipIfNoTai skips the test if SANDBOX_TEST_REMOTE_ADDR is empty.
func skipIfNoTai(t *testing.T)

// testImage returns SANDBOX_TEST_IMAGE or "yaoapp/sandbox-v2-test:latest".
func testImage() string

// setupManager initializes sandbox with a local pool, returns cleanup func.
// Calls sandbox.Init + sandbox.M().Start.
func setupManager(t *testing.T) func()

// setupManagerWithRemote initializes sandbox with local + remote pools.
func setupManagerWithRemote(t *testing.T) func()

// createTestBox creates a box with defaults and returns it. Registers t.Cleanup for removal.
func createTestBox(t *testing.T, opts ...sandbox.CreateOption) *sandbox.Box

How to Write a Test

Standard pattern

// manager_test.go
package sandbox_test

import (
    "context"
    "testing"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"

    sandbox "github.com/yaoapp/yao/sandbox/v2"
)

func TestCreate(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()

    box, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image: testImage(),
        Owner: "test-user",
    })
    require.NoError(t, err)
    defer box.Remove(context.Background())

    assert.NotEmpty(t, box.ID())
    assert.Equal(t, "test-user", box.Owner())
}

func TestCreate_NoImage(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()

    _, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{})
    assert.Error(t, err) // Image is required
}

Container execution tests

// box_test.go
package sandbox_test

func TestExec(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    result, err := box.Exec(context.Background(), []string{"echo", "hello"})
    require.NoError(t, err)
    assert.Equal(t, 0, result.ExitCode)
    assert.Equal(t, "hello\n", result.Stdout)
    assert.Empty(t, result.Stderr)
}

func TestExec_NonZeroExit(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    result, err := box.Exec(context.Background(), []string{"sh", "-c", "exit 42"})
    require.NoError(t, err)
    assert.Equal(t, 42, result.ExitCode)
}

Streaming tests

// box_test.go
package sandbox_test

func TestStream(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    s, err := box.Stream(context.Background(), []string{"sh", "-c", "echo a; sleep 0.1; echo b"})
    require.NoError(t, err)

    out, _ := io.ReadAll(s.Stdout)
    code, err := s.Wait()
    assert.NoError(t, err)
    assert.Equal(t, 0, code)
    assert.Contains(t, string(out), "a\n")
    assert.Contains(t, string(out), "b\n")
}

func TestStream_Cancel(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    s, err := box.Stream(context.Background(), []string{"sleep", "60"})
    require.NoError(t, err)

    s.Cancel()
    code, _ := s.Wait()
    assert.NotEqual(t, 0, code) // killed
}

func TestStream_Stdin(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    s, err := box.Stream(context.Background(), []string{"cat"})
    require.NoError(t, err)

    s.Stdin.Write([]byte("hello\n"))
    s.Stdin.Close()

    out, _ := io.ReadAll(s.Stdout)
    code, _ := s.Wait()
    assert.Equal(t, 0, code)
    assert.Equal(t, "hello\n", string(out))
}

Workspace tests

// box_test.go
package sandbox_test

func TestWorkspace_ReadWrite(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    ws := box.Workspace()
    err := ws.WriteFile("test.txt", []byte("hello"), 0644)
    require.NoError(t, err)

    data, err := fs.ReadFile(ws, "test.txt")
    require.NoError(t, err)
    assert.Equal(t, "hello", string(data))
}

func TestWorkspace_MkdirAll(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    ws := box.Workspace()
    err := ws.MkdirAll("a/b/c", 0755)
    require.NoError(t, err)

    info, err := fs.Stat(ws, "a/b/c")
    require.NoError(t, err)
    assert.True(t, info.IsDir())
}

func TestWorkspace_WalkDir(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    ws := box.Workspace()
    ws.MkdirAll("src", 0755)
    ws.WriteFile("src/main.go", []byte("package main"), 0644)
    ws.WriteFile("src/util.go", []byte("package main"), 0644)

    var files []string
    fs.WalkDir(ws, "src", func(path string, d fs.DirEntry, err error) error {
        if !d.IsDir() { files = append(files, path) }
        return nil
    })
    assert.Len(t, files, 2)
}

Lifecycle tests

// manager_lifecycle_test.go
package sandbox_test

func TestIdleCleanup_Session(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()

    box, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image:       testImage(),
        Policy:      sandbox.Session,
        IdleTimeout: 2 * time.Second,
    })
    require.NoError(t, err)

    // Box exists
    _, err = sandbox.M().Get(context.Background(), box.ID())
    assert.NoError(t, err)

    // Wait for idle + cleanup cycle
    time.Sleep(4 * time.Second)
    sandbox.M().Cleanup(context.Background())

    // Box should be gone
    _, err = sandbox.M().Get(context.Background(), box.ID())
    assert.ErrorIs(t, err, sandbox.ErrNotFound)
}

func TestStartRecovery(t *testing.T) {
    skipIfNoDocker(t)

    // Phase 1: create a box, then shut down Manager
    cleanup1 := setupManager(t)
    box, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image: testImage(),
        Owner: "recovery-test",
    })
    require.NoError(t, err)
    boxID := box.ID()
    cleanup1() // closes Manager, but container stays

    // Phase 2: new Manager, Start should discover the container
    cleanup2 := setupManager(t)
    defer cleanup2()

    recovered, err := sandbox.M().Get(context.Background(), boxID)
    require.NoError(t, err)
    assert.Equal(t, boxID, recovered.ID())
    assert.Equal(t, "recovery-test", recovered.Owner())

    // Clean up
    recovered.Remove(context.Background())
}

func TestHeartbeat(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()
    box := createTestBox(t)

    // Simulate heartbeat
    err := sandbox.M().Heartbeat(box.ID(), true, 3)
    assert.NoError(t, err)

    info, _ := box.Info(context.Background())
    assert.Equal(t, 3, info.ProcessCount)
}

func TestHeartbeat_NotFound(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()

    err := sandbox.M().Heartbeat("nonexistent", true, 1)
    assert.ErrorIs(t, err, sandbox.ErrNotFound)
}

Pool management tests

// manager_test.go
package sandbox_test

func TestPoolLimits_MaxTotal(t *testing.T) {
    skipIfNoDocker(t)

    // Init with MaxTotal=1
    err := sandbox.Init(sandbox.Config{
        Pool: []sandbox.Pool{{
            Name:     "limited",
            Addr:     "local",
            MaxTotal: 1,
        }},
    })
    require.NoError(t, err)
    sandbox.M().Start(context.Background())
    defer sandbox.M().Close()

    box1, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image: testImage(),
    })
    require.NoError(t, err)
    defer box1.Remove(context.Background())

    _, err = sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image: testImage(),
    })
    assert.ErrorIs(t, err, sandbox.ErrLimitExceeded)
}

func TestAddPool(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()

    err := sandbox.M().AddPool(context.Background(), sandbox.Pool{
        Name: "new-pool",
        Addr: "local",
    })
    assert.NoError(t, err)

    pools := sandbox.M().Pools()
    names := make([]string, len(pools))
    for i, p := range pools { names[i] = p.Name }
    assert.Contains(t, names, "new-pool")
}

func TestRemovePool_InUse(t *testing.T) {
    skipIfNoDocker(t)
    cleanup := setupManager(t)
    defer cleanup()

    box := createTestBox(t)
    _ = box

    err := sandbox.M().RemovePool(context.Background(), "local", false)
    assert.ErrorIs(t, err, sandbox.ErrPoolInUse)
}

Multi-pool tests

// manager_test.go
package sandbox_test

func TestMultiPool(t *testing.T) {
    skipIfNoDocker(t)
    skipIfNoTai(t)
    cleanup := setupManagerWithRemote(t)
    defer cleanup()

    // Create on local
    local, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image: testImage(),
        Pool:  "local",
    })
    require.NoError(t, err)
    defer local.Remove(context.Background())

    // Create on remote
    remote, err := sandbox.M().Create(context.Background(), sandbox.CreateOptions{
        Image: testImage(),
        Pool:  "remote",
    })
    require.NoError(t, err)
    defer remote.Remove(context.Background())

    // Both should exec
    r1, _ := local.Exec(context.Background(), []string{"echo", "local"})
    r2, _ := remote.Exec(context.Background(), []string{"echo", "remote"})
    assert.Equal(t, "local\n", r1.Stdout)
    assert.Equal(t, "remote\n", r2.Stdout)
}

OAuth / gRPC env injection tests

// grpc_test.go
package sandbox_test

func TestBuildGRPCEnv_Local(t *testing.T) {
    env := sandbox.BuildGRPCEnv(&sandbox.Pool{Addr: "local"}, "sb-001", "tok", "ref")
    assert.Equal(t, "sb-001", env["YAO_SANDBOX_ID"])
    assert.Equal(t, "tok", env["YAO_TOKEN"])
    assert.Equal(t, "ref", env["YAO_REFRESH_TOKEN"])
    assert.NotEmpty(t, env["YAO_GRPC_ADDR"])
    assert.Empty(t, env["YAO_GRPC_TAI"])
}

func TestBuildGRPCEnv_Remote(t *testing.T) {
    env := sandbox.BuildGRPCEnv(&sandbox.Pool{Addr: "tai://gpu.internal"}, "sb-002", "tok", "ref")
    assert.Equal(t, "enable", env["YAO_GRPC_TAI"])
    assert.NotEmpty(t, env["YAO_GRPC_UPSTREAM"])
}

func TestCreateContainerTokens(t *testing.T) {
    // Requires Yao runtime for OAuth
    cleanup := setupManager(t)
    defer cleanup()

    access, refresh, err := sandbox.CreateContainerTokens("sb-test", "user-1")
    require.NoError(t, err)
    assert.NotEmpty(t, access)
    assert.NotEmpty(t, refresh)
}

Required Test Cases

File	Required Cases
sandbox_test.go	Init succeeds / M() panics before Init / double Init is safe
manager_test.go	Create / Create with explicit ID / Create no image (error) / Get / Get not found / GetOrCreate / List / List with owner filter / Remove / pool limits MaxTotal / pool limits MaxPerUser / AddPool / RemovePool / RemovePool in use / Pools
manager_lifecycle_test.go	Start recovery from labels / Cleanup Session idle / Cleanup LongRunning stop then remove / Persistent never cleaned / Heartbeat updates / Heartbeat not found / OneShot removed after Exec
box_test.go	Exec success / Exec non-zero exit / Exec with WorkDir / Exec with Env / Exec with Timeout / Stream read / Stream cancel / Stream stdin / Workspace ReadFile+WriteFile / Workspace MkdirAll / Workspace Remove / Workspace Rename / Workspace WalkDir / VNC (skip if no VNC image) / Proxy URL / Start+Stop+Start / Info
box_attach_test.go	Attach WS (skip if no WS server image) / Attach SSE (skip if no SSE server image)
grpc_test.go	BuildGRPCEnv local / BuildGRPCEnv remote / CreateContainerTokens / RevokeContainerTokens

Makefile

Add to Makefile:

TESTFOLDER_SANDBOX_V2 := $(shell $(GO) list ./sandbox/v2/...)

.PHONY: unit-test-sandbox-v2
unit-test-sandbox-v2:
	echo "mode: count" > coverage.out
	for d in $(TESTFOLDER_SANDBOX_V2); do \
		$(GO) test -tags $(TESTTAGS) -v -timeout=10m \
			-covermode=count -coverprofile=profile.out \
			-coverpkg=$$d \
			$$d > tmp.out; \
		cat tmp.out; \
		if grep -q "^--- FAIL" tmp.out; then \
			rm tmp.out; \
			exit 1; \
		elif grep -q "build failed" tmp.out; then \
			rm tmp.out; \
			exit 1; \
		fi; \
		if [ -f profile.out ]; then \
			cat profile.out | grep -v "mode:" >> coverage.out; \
			rm profile.out; \
		fi; \
	done

CI Integration

Add sandbox-v2-test job to unit-test.yml and pr-test.yml:

sandbox-v2-test:
  runs-on: ubuntu-latest
  services:
    mongodb:
      image: mongo:6.0
      ports:
        - 27017:27017
      env:
        MONGO_INITDB_ROOT_USERNAME: root
        MONGO_INITDB_ROOT_PASSWORD: "123456"
        MONGO_INITDB_DATABASE: test
  strategy:
    matrix:
      go: ["1.25"]
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-go@v5
      with:
        go-version: ${{ matrix.go }}

    - name: Start Tai container
      run: |
        docker run -d --name tai \
          -v /var/run/docker.sock:/var/run/docker.sock \
          -p 2375:2375 -p 9100:9100 -p 8080:8080 -p 6080:6080 \
          yaoapp/tai:latest
        sleep 3

    - name: Build V2 test image
      run: |
        cd sandbox/docker
        bash build.sh v2

    - name: Setup ENV
      run: |
        echo "YAO_DB_DRIVER=sqlite3" >> $GITHUB_ENV
        echo "YAO_DB_PRIMARY=${{ github.WORKSPACE }}/../app/db/yao.db" >> $GITHUB_ENV
        echo "SANDBOX_TEST_IMAGE=yaoapp/sandbox-v2-test:latest" >> $GITHUB_ENV
        echo "SANDBOX_TEST_REMOTE_ADDR=tai://127.0.0.1" >> $GITHUB_ENV
        echo "TAI_TEST_HOST=127.0.0.1" >> $GITHUB_ENV
        mkdir -p ${{ github.WORKSPACE }}/../app/db

    - name: Run Sandbox V2 Tests
      run: make unit-test-sandbox-v2

    - name: Codecov Report
      uses: codecov/codecov-action@v4
      with:
        token: ${{ secrets.CODECOV_TOKEN }}

Key decisions:

• SQLite only — sandbox is infrastructure, not data-model dependent
• Tai container provides remote mode — exercises the full proxy path
• sandbox-v2-test as default test image — includes yao-grpc (heartbeat), openai-proxy, Nginx, WS echo + SSE test services
• CI builds test image from source (Step 4.5) — ensures binary compatibility with latest tai SDK + yao-grpc changes
• Attach tests (WS/SSE) use sandbox-v2-test image's built-in test services

Coverage

• Target: >80% per file, >80% overall
• sandbox.go (singleton) covered via sandbox_test.go
• manager.go is the heaviest file — must have dedicated manager_test.go + manager_lifecycle_test.go
• box.go exercises all tai SDK integration points
• grpc.go tested with pure unit tests (token generation, env building)

Running Tests

# All sandbox v2 tests (local Docker only)
make unit-test-sandbox-v2

# With remote mode (start Tai first)
SANDBOX_TEST_REMOTE_ADDR=tai://127.0.0.1 make unit-test-sandbox-v2

# Single file
go test -v ./sandbox/v2/ -run TestCreate

# Single test
go test -v ./sandbox/v2/ -run TestExec_NonZeroExit

# With race detector
go test -race -v ./sandbox/v2/