@detiq/agents — Generation Agents & E2E Evaluation Harness

AI agents that produce test artifacts grounded in the ZeTa Core, coordinated by orchestrators:

• DualContextOrchestrator / SwarmOrchestrator: Multi-agent pipeline coordinating CAG (Component-Augmented Generation) and KAG (Knowledge-Augmented Generation).
• test-case-agent.ts: 22-pass test case generation grounded in real crawled elements with semantic dedup.
• data-agent.ts: Realistic synthetic test data and slot generation matching field constraints.
• script-agent.ts: Deterministic Playwright test script generation and linting.
• alignment-agent.ts: Verification of script actions against test case steps.
• llm.ts: Choke-point wrapper around LLM providers with token tracking, caching, and error sanitization.
• llm-mock.ts: Mock / record / replay engine for zero-cost, deterministic CI runs.

E2E Evaluation Harness (scripts/agent-e2e.ts)

The E2E harness tests the entire agent pipeline from screen fixture to verified Playwright script without requiring live browser crawling or UI clicks.

Commands

# 1. Run all fixtures in mock mode (CI default, zero token cost)
npm run e2e -- --fixture all --mock --fail-on-assert

# 2. Run a specific fixture
npm run e2e -- --fixture login --mock

# 3. Run against an existing project in the database
npm run e2e -- --project <projectId> --screen <screenId>

# 4. Limit pipeline stages
npm run e2e -- --fixture login --mock --stages cases,data

# 5. Output JSON report for CI/monitoring
npm run e2e -- --fixture all --mock --json report.json

Mock & Record Modes

The mock engine intercepts calls in askTracked and askTrackedWithImages.

| Environment Variable | Value | Description | |---|---|---| | ZETA_LLM_MOCK | 1 or replay | Replays saved fixtures from eval/fixtures/llm/. Falls back to schema-valid deterministic mock if fixture is missing. | | ZETA_LLM_MOCK | record | Hits the live LLM provider and automatically records prompt/response pairs to eval/fixtures/llm/ for future replay. | | ZETA_LLM_MOCK_FAIL | 429 | 503 | malformed-json | empty | Injects synthetic provider failures to test error handling and retry resilience. |

Safety: ZETA_LLM_MOCK immediately throws an exception if NODE_ENV=production.

Golden Set Evaluation (eval/)

eval/
  fixtures/screens/<name>.json     # Screen + elements as captured by crawler
  fixtures/llm/<key>.json          # Recorded LLM responses for deterministic replay
  golden/<name>.golden.json        # Golden assertion criteria (minCases, kinds, title pattern)
  baseline.json                    # Historical score baseline
  run-eval.ts                      # Evaluator runner comparing outputs against baseline

Golden Evaluation Commands

# Run eval across all golden screens
npm run eval

# Accept current scores and update baseline.json
npm run eval:accept

# Run eval for specific screens only
npm run eval -- login checkout

Golden Spec Format (eval/golden/<name>.golden.json)

{
  "screen": "login",
  "minCases": 12,
  "mockMinCases": 1,
  "mustHaveKinds": ["FUNCTIONAL", "SECURITY"],
  "kindMinimums": { "SECURITY": 2 },
  "maxDuplicateRatio": 0.10,
  "titlePattern": "^[^—]+ — [^—]+ — .+$",
  "requiredCoverageTerms": ["invalid password", "empty email", "sql injection"],
  "script": { "minAlignment": 75, "maxLintViolations": 0 }
}

CI Integration

Mock evaluation is automated via GitHub Actions in .github/workflows/agents-eval.yml:

• Triggers on PRs modifying packages/agents/**, packages/core/**, packages/app-brain/**, or agent prompt seeds.
• Runs unit tests and npm run e2e -- --fixture all --mock --fail-on-assert.
• Uploads report.json as a build artifact.