Schöneck

AI-validated visual tests for web applications.

schoeneck discovers test specifications from .schoeneck.md spec files, captures screenshots using Playwright, and validates them against natural language conditions using AI agents (Claude, Codex, or Gemini).

Installation

# Using npm (recommended)
npm install -g schoeneck

# Run without installing
npx schoeneck

Quick Start

1. Initialize configuration:

   schoeneck init

   This creates schoeneck.config.json with detected settings. Only values that differ from defaults are written.

2. Create a test spec (specs/homepage.schoeneck.md):

   ---
   route: /
   viewport: { width: 1280, height: 720 }
   ---
   The page should display a heading with the text "Example Domain"

3. Run tests:

   schoeneck run

CLI Commands

| Command | Description | | ---------------------------------- | -------------------------------------- | | schoeneck init | Initialize configuration interactively | | schoeneck run | Execute all discovered tests | | schoeneck run --filter <pattern> | Run tests matching pattern | | schoeneck list | List all discovered test specs | | schoeneck doctor | Check environment and dependencies | | schoeneck clean | Remove cached artifacts |

Configuration

Create schoeneck.config.json in your project root. You only need to specify the fields you want to override — all other fields use defaults and are deep-merged automatically:

{
  "schemaVersion": 1,
  "server": {
    "command": "npm start",
    "baseUrl": "http://localhost:3000"
  },
  "agent": {
    "provider": "claude"
  }
}

Common Configuration Options

| Option | Type | Default | Description | | --------------------- | -------- | -------------------------------------------- | ---------------------------------------------------- | | schemaVersion | number | — | Required. Must be 1 | | agent.provider | string | codex | AI provider: claude, codex, or gemini | | agent.timeoutMs | number | 90000 | AI agent timeout in milliseconds | | browser.headless | boolean | true | Run browser in headless mode | | browser.viewport | object | { width: 1280, height: 720, deviceScaleFactor: 1 } | Default viewport for tests | | browser.locale | string | en-US | Browser locale | | server.command | string | npm run dev | Command to start dev server | | server.baseUrl | string | http://127.0.0.1:3000 | Base URL of the dev server | | server.ready.strategy | string | http | Ready check: http, tcp, or stdout | | run.specGlobs | string[] | ["**/*.schoeneck.md"] | Glob patterns for .schoeneck.md spec files | | run.concurrency | number | 2 | Parallel test execution limit | | run.retries | number | 0 | Retry count for failed tests |

Run schoeneck init to generate a minimal config. Defaults are deep-merged automatically at runtime.

Test Spec Format

Tests are defined in .schoeneck.md files with YAML frontmatter and markdown condition content. Each file is one test spec.

---
route: /dashboard
id: dashboard-test
viewport: { width: 1280, height: 720 }
tags: [smoke, dashboard]
---

# Dashboard Page

The dashboard should display user info and be responsive.

## Layout
1. **Header** with navigation
2. **Sidebar** with menu items

The YAML frontmatter between --- delimiters defines test configuration. The markdown body after the closing --- is the condition text that the AI agent evaluates against the screenshot.

Specs without an explicit id get one auto-generated from the relative file path (e.g., specs/dashboard.schoeneck.md).

The run.specGlobs array controls which files are discovered (defaults to ["**/*.schoeneck.md"]). A file fingerprint cache (.schoeneck/scan-cache.json) avoids re-parsing unchanged files on subsequent runs. Pass --no-cache to bypass it.

Frontmatter Options

| Option | Type | Required | Description | | ------------ | -------- | -------- | ------------------------------------------------------------------------ | | route | string | Yes | Route/path to navigate to | | id | string | No | Unique test identifier (auto-generated if omitted) | | tags | string[] | No | Tags for filtering tests | | headers | object | No | HTTP headers as { key: value } pairs | | method | string | No | HTTP method: GET, POST, PUT, DELETE, or PATCH | | body | string | No | Request body (used with method) | | viewport | object | No | { width, height, deviceScaleFactor? } for viewport | | waitFor | object | No | { strategy, value? } — strategy: networkIdle, load, domContentLoaded, selector, timeout. The selector and timeout strategies require value. | | timeoutMs | number | No | Per-test timeout in milliseconds | | mask | string[] | No | CSS selectors for elements to mask before screenshot | | screenshot | object | No | { fullPage?, selector?, omitBackground? } screenshot options | | agent | object | No | { provider?, model?, maxTokens? } per-test agent override |

Condition Syntax

The condition is the markdown content after frontmatter. It contains natural language assertions that the AI agent evaluates against the screenshot. Write clear, specific assertions:

Good:

• "The navigation menu should have exactly 5 items"
• "The error message should display 'Invalid email address'"

Avoid:

• "The page looks correct" (too vague)
• "Everything is working" (not testable)

Output

Exit Codes

| Code | Meaning | | ---- | ------------------------------ | | 0 | All tests passed | | 1 | One or more tests failed | | 2 | Configuration or runtime error |

Artifacts

Test artifacts are saved to .schoeneck/artifacts/:

.schoeneck/artifacts/
  summary.json              # Overall run summary
  server.log                # Server output (if server was spawned)
  <test-id>/
    meta.json               # Test metadata and result
    timings.json            # Timing breakdown
    prompt.txt              # Prompt sent to AI agent

Environment Variables

schoeneck spawns AI provider CLI binaries (claude, codex, gemini) rather than calling APIs directly. The following environment variables are passed through to the CLI tools:

| Variable | Description | | ------------------- | ------------------------------------ | | ANTHROPIC_API_KEY | Passed to the claude CLI | | OPENAI_API_KEY | Passed to the codex CLI | | GOOGLE_API_KEY | Passed to the gemini CLI |

Requirements

• Node.js >= 20
• Playwright browsers (npx playwright install)
• CLI binary for your chosen AI provider (claude, codex, or gemini)

License

EUPL-1.2