Agent Check

Provider-neutral testing plan runner CLI and TypeScript library.

Agent Check accepts a human-readable YAML testing plan, validates it, executes it through runtime provider plugins, collects evidence, classifies failures, and writes run artifacts.

The durable input is a testing plan, not a creator/checker contract. Authoring can be manual, agentic, or external.

Install

npm install -g @codebolt/agent-check

CLI

agent-check validate examples/web-local.plan.yaml
agent-check providers --config agent-check.config.yaml
agent-check run examples/web-local.plan.yaml --config agent-check.config.yaml
agent-check refine <run-id>

Built-In Providers

• mock: deterministic provider for runner tests and examples.
• web-playwright: Playwright-backed website provider with headed/headless mode.
• tui-process: spawned process provider for terminal/TUI plans.
• electron-playwright: Electron provider using Playwright renderer controls.
• windows-desktop: Windows UI Automation provider for basic desktop controls.

Mobile candidate types are present in the schema, but mobile execution requires a future provider.

LLM Inference

The LLM adapter uses the Vercel AI SDK. The included config uses the Codex CLI community provider:

llm:
  enabled: true
  model: codex-cli/gpt-5.5

AI_LLM_TIMEOUT_MS bounds each LLM call. The default is 60000.

Zhipu AI / Z.AI is also available through the Vercel AI SDK community provider:

$env:ZHIPU_API_KEY = "<your-api-key>"
$env:AI_MODEL = "zhipu/glm-4.7"

Use zai/glm-4.7 for Z.AI's https://api.z.ai/api/paas/v4 endpoint.

Testing Plan Schema

See docs/TEST_PLAN_SCHEMA.md for the detailed YAML schema and examples.