Evalora

Measure what frontend coding agents build.

Version 3 installs as Evalora Free. The public npm package excludes Pro implementations. See the 3.0 migration notes when upgrading.

Give models the same task, run their applications, and compare the results in a local workspace. Evalora checks behavior, accessibility, source, rendered HTML, and screenshots. Every score links to the checks and evidence behind it.

Start

Install Evalora in your project with Node.js 22.14 or later:

npm install -D evalora
npx evalora start

Choose Demo replay to try a recorded example without an account or model usage. For live work, connect OpenAI Codex, GitHub Copilot, or Claude Code through its existing CLI sign-in, then choose a model.

The setup wizard creates an Angular, React, or Plain HTML starter. Choose Set up with assistant for an existing application or custom setup. Review the assistant's files and commands before Prepare workspace. Angular and React setup downloads framework dependencies. Angular 22 needs Node 22.22.3+, 24.15.0+, or 26+ within its supported major versions.

Keep the terminal running. Open the same workspace later with npx evalora start. The first-evaluation guide covers the complete flow.

Prepare a case, run it, inspect the result

1. In Test cases, describe what the agent should build and add the checks that define success. Add expected source, HTML, or visual screens when needed.
2. Save case, then open Runs. Choose the cases, models, and repeats. Free uses your applied skill. Pro also lets you select skill versions. Preview run shows the exact number of attempts.
3. Confirm and run, then follow Run activity. Evalora generates each application in a separate workspace and saves its results.
4. Open Summary or Test results to inspect failures, screenshots, source differences, and logs. Use Dashboard for history. Run comparison, skill comparison, and dashboard run inspection require Pro.

A test case contains the build instructions and their success checks. General accessibility or page-health checks alone do not establish task correctness. An expected example adds a similarity requirement, with default minimums of 100% for HTML and 80% for source. Source-only application examples skip browser checks until App must build before checking is enabled. Visual screens always require a build and browser. See how scoring works.

HTML and source references stay private during generation. Visual references also give the agent design inputs. Add a PNG, a Figma frame exported as PNG, or a reference page URL with Add visual screen. The Visual result tab shows the expected screen, actual screen, and difference image.

Skills and editions

Free uses one applied skill. Open Skills to apply or replace it. Pro adds multiple skills and versions, AI improvement, Compare runs, Compare skills, Inspect a run, and Figma integration. These destinations show a Pro explanation in Free. Basic test results and the dashboard overview stay available.

The following version-management workflow requires Pro. See editions for local development and package behavior.

Compare skills and get help

Open Skills > Add skill version to import a local skill folder or bundle. Name the version, review it, and choose Run this version. Later imports keep earlier versions available. Skill versions let you compare changes against the same cases and models.

Evalora installs evalora-setup, evalora-author, and evalora-support into .agents/skills/ on start, including when npm lifecycle scripts are disabled. Existing user edits are preserved. Install them separately with:

npx evalora skill
# For an agent with a different discovery directory:
npx evalora skill --to .claude/skills

Ask AI answers workspace questions and supports follow-ups through a connected model. The setup assistant helps prepare the workspace. Need help writing checks? helps draft a case. Failed operations offer Diagnose and propose a fix with the recorded error and logs. Review AI suggestions before applying them. Your provider's authentication, model access, and usage limits apply.

Commands

npx evalora doctor              # Check local prerequisites
npx evalora configure           # Change agents and default models
npx evalora run --dry-run       # Preview the configured evaluation matrix
npx evalora run                 # Run that complete matrix
npx evalora history             # List saved runs

Saving cases and previewing runs do not call a model. Live generation and AI assistance use the selected provider. The default workspace stores cases, the application template, and imported skills under evalora/; evidence lives under .evalora/history/ as run metadata and a compact evidence archive. Temporary application workspaces are cleaned up after the evidence is saved.

Documentation

• First evaluation
• Custom workspace and skill versions
• Harnesses and authentication
• Troubleshooting
• Case and configuration reference
• Scoring and history and CI
• Upgrade to version 2
• Architecture and release process

Development

npm ci
npm exec -- playwright install chromium
npm test
npm run check

npm start opens setup in this checkout. npm run ui:demo opens the bundled Angular example. CI verifies an installed package against known replay fixtures; see release acceptance.

License: MIT.