@twai/cli

Command-line interface for the TWAI (Test With AI) testing framework.

Installation

pnpm add -g @twai/cli

Usage

# Initialize a new project (creates aitest.config.ts)
aitest init

# List providers and whether their env vars are set
aitest providers

# Plan specs (writes .aitest/plans/<id>.json)
aitest plan

# Run specs (writes .aitest/reports/<id>.json/.md)
aitest run

# Show help
aitest --help

Commands

aitest init

Initialize a new TWAI project with a config file.

aitest providers

List available LLM providers (based on installed adapter packages) and whether required env vars are present.

aitest plan

Plan specs with the configured LLM and write plans to .aitest/plans/<id>.json.

aitest run

Run specs: ensure a plan exists, then generate/run static tests or run agentic runner; write reports to .aitest/reports.

Options:

• -c, --config <path> - Path to config file

Shared LLM options (apply to plan/run):

• --provider <provider> - openai|anthropic|mock
• --model <model> - Model name (warns if empty for non-mock)
• --api-key <key> - API key (discouraged; prefer env)
• --api-key-env <envVar> - Env var name containing the API key
• --base-url <url> - Base URL override (also used as spec baseUrl)
• --temperature <n>
• --max-tokens <n>

Configuration

Create an aitest.config.ts file:

import type { AITestConfig } from "@twai/cli";

export default {
  // Optional: override where specs are loaded from
  testsGlob: "tests/**/*.md",

  // Optional: Base URL applied to loaded specs (useful for relative URLs)
  baseUrl: "http://localhost:3000",

  // LLM configuration
  llm: {
    provider: "openai", // openai | anthropic | mock
    model: "gpt-4.1-mini",

    // Prefer env vars (examples below); apiKey in config works but is discouraged.
    apiKey: process.env.OPENAI_API_KEY,

    // Optional overrides
    // baseUrl: "https://api.openai.com/v1",
    // temperature: 0.2,
    // maxTokens: 512,
  },

  // Optional Next.js automation (used by `aitest run`)
  next: {
    devServer: {
      enabled: false,
      // cwd: ".",
      // port: 3000,
      // host: "127.0.0.1",
      // reuseIfRunning: true,
    },
  },
} satisfies AITestConfig;

Environment variables

• AITEST_PROVIDER: openai|anthropic|mock
• AITEST_MODEL: model name (used when llm.model not provided)
• AITEST_API_KEY: generic API key override
• OPENAI_API_KEY: OpenAI key (used when provider=openai and no AITEST_API_KEY)
• ANTHROPIC_API_KEY: Anthropic key (used when provider=anthropic and no AITEST_API_KEY)
• AITEST_BASE_URL: base URL override (also applied to specs)
• AITEST_TEMPERATURE: number
• AITEST_MAX_TOKENS: integer

CLI override examples

# Override provider/model via CLI (highest precedence)
aitest plan --provider openai --model "gpt-4.1-mini"

# Pass api key via env (recommended)
OPENAI_API_KEY="..." aitest run

# Reference a custom env var name
MY_OPENAI_KEY="..." aitest run --provider openai --api-key-env MY_OPENAI_KEY --model "gpt-4.1-mini"