evals-lab

Test what a model, an HTTP endpoint or an API answers, side by side, and score it with evals you define in advance. Runs locally in a friendly UI, and doesn't send your data anywhere you haven't pointed it.

An experiment takes items from a Source and sends each one to one or more Variants, then scores the Responses.

• Variants can be models, local (Ollama, llama.cpp) or hosted (Anthropic, OpenAI, any compatible API), or any HTTP endpoint: your own service, an internal API, a gateway in front of a model. The lab sends each request as you write it, with the key held in a connection rather than in the request, and grades whatever comes back.
• Sources can be files you upload, a prompt, Power Automate runs, or live data from an endpoint: an HTTP GET, or an API call with its own method, headers and body. Grade a saved snapshot of the response, or fetch a fresh one at run time.

Evals include parsing checks, item counts, comparison with production responses, grading by a model, and evals (evals) shared between experiments. Run history is kept and can be exported.

The same evals run in CI: evals-lab run fails the build when an eval fails, like a failing unit test.

Currently in early release.

Example uses:

• Checking that an HTTP endpoint or internal API still answers correctly after a change, in the lab or on every commit in CI
• Keywording images with a model's vision, and running evals on the keywords it returns
• Comparing two models, or two prompts, on the same items and seeing which scores higher
• Grading what an API returns against an eval of expected answers
• Checking which model can meet the evals in a Power Automate workflow
• Sending images to a model with vision and comparing its answers for accuracy

Install and run

npm install -g evals-lab
evals-lab

Open http://localhost:8080. A new lab starts with six demos, each an eval and its experiment, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without installing: npx evals-lab.

| Needs | Windows | Linux | |---|---|---| | Node.js 22.18+ | comes with npm | same | | Python 3.10+ | winget install Python.Python.3.13 | your distribution's python3 | | ImageMagick, for runs that send images | winget install ImageMagick.ImageMagick | your distribution's imagemagick |

evals-lab names anything it cannot find. Open a new terminal after installing one. macOS should work and is untried.

User data

User data is kept in a single folder. Upgrades intend to keep user data. The folders are:

| | | |---|---| | Windows | %LOCALAPPDATA%\evals-lab | | Linux | ~/.local/share/evals-lab (or $XDG_DATA_HOME/evals-lab) | | macOS | ~/Library/Application Support/evals-lab |

Delete this folder to start again fresh.

Options

evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]

| Option | Environment | Default | |---|---|---| | --port | PORT | 8080 | | --host | LAB_HOST | 127.0.0.1 | | --data-dir | DATA_DIR | the directory above | | | LAB_PASSWORD | none | | | OLLAMA_URL | http://127.0.0.1:11434 | | | PYTHON | py -3, python or python3, whichever is found |

--host 0.0.0.0 makes the lab reachable from your network and needs LAB_PASSWORD to be set (otherwise it refuses connections).

In CI

An experiment can run with no lab, from a repository. In the lab, the experiment's menu › Export for CI… downloads it as a bundle: unzip it into the repository, then:

EVALSLAB_API_KEY_ANTHROPIC=... evals-lab run evals/

evals-lab run <bundle-dir | pipeline.yaml> [--items <dir>] [--json <file|->]
              [--junit <file>] [--summary <file>] [--min-pass <0..1>] [--progress]

| Option | | |---|---| | --items | the items the experiment's Source reads; default the bundle's items/ | | --json | the JSON report; - for stdout | | --junit | JUnit XML, for a CI test panel | | --summary | the results as Markdown, added to the file: for GitHub, "$GITHUB_STEP_SUMMARY" | | --min-pass | an eval passes when this share of its items pass | | --progress | a line per item as it finishes |

The bundle holds no key. Each AI Connection's key comes from EVALSLAB_API_KEY_ and its slug in Setup, in capitals with hyphens as underscores: a connection slugged anthropic reads EVALSLAB_API_KEY_ANTHROPIC, and gpt-4o-mini reads EVALSLAB_API_KEY_GPT_4O_MINI. A table of results goes to stderr.

| Exit code | | |---|---| | 0 | every eval passed | | 1 | an eval failed | | 2 | incomplete: an item is missing, or did not run | | 3 | broken or refused |

An item an eval case names that is not in the items is reported as Missing. Images need ImageMagick, as in the lab.

GitHub Actions

name: Evals
on: pull_request

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      # Only when the items are images:
      # - run: sudo apt-get update && sudo apt-get install -y imagemagick
      - run: >
          npx --yes [email protected] run evals/
          --junit evals.xml --json evals.json --summary "$GITHUB_STEP_SUMMARY"
        env:
          EVALSLAB_API_KEY_ANTHROPIC: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evals
          path: evals.*

A failing eval fails the job, and the results show on the run's page. Other CI systems run the same command and read --junit in their test panel.

Against a running lab

CI can run a lab's own experiment there instead, so the run shows in that lab's History and its keys never leave it:

LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: images pipeline" --wait

evals-lab run --lab <url> --pipeline <name> [--wait] [--json <file|->]
              [--junit <file>] [--summary <file>] [--min-pass <0..1>] [--progress]

| Option | | |---|---| | --lab | the lab's address; LAB_PASSWORD is sent when it is set | | --pipeline | the experiment, by its name or its id | | --wait | wait for the run, and exit with the lab's verdict on it |

Without --wait the run's id is printed once it is queued, and the exit code is 0; --json, --junit, --summary, --min-pass and --progress need --wait. The exit codes are the ones above, and a cancelled run is 2.

Presets

An app that sends decoding settings of its own can compare them with the presets its pack gives the lab, as the request fields each sends:

evals-lab presets compare app-presets.json --pack my-app.labpack [--fail-on-difference]
evals-lab presets compare app-presets.json --lab https://lab.example

It reports each preset as the same, different or on one side only, and exits 0; --fail-on-difference exits 1 on a difference. The app's file names each preset's request fields:

{ "format": "evals-lab/presets", "version": 1,
  "presets": [{ "id": "repeat-1.2", "request": { "repeat_penalty": 1.2 } }] }

evals-lab presets --help has the rest.

Getting started

First set up a Setup profile: Setup › Add profile, then pick its type — Ollama (local or cloud), OpenAI-compatible, Anthropic, llama.cpp or an HTTP endpoint — and give its address, model and key. NB: An Ollama profile without an address uses OLLAMA_URL.

Set up your data source(s) in Library.

Then create an Experiment to Run. Changes are auto-saved. Use the "Restore" menu to go back and undo any changes you don't want to keep.

Power Automate

The lab can read a Power Automate cloud flow and its run history from your org's Microsoft 365 tenant then grade the flow's HTTP step. It needs a work or school account. Personal accounts (outlook.com, hotmail.com) have no cloud flows. The sign-in data is kept in your browser to avoid it being stored elsewhere, the downside is you need to re-sign in if you close/reopen your web browser. The evals can keep running however, the sign-in is just to get the workflow data, which does persist in the evals-lab user data locally.

Register an app

A tenant administrator does this once, at https://entra.microsoft.com:

1. App registrations › New registration.
   • Supported account types: Accounts in this organizational directory only.
   • Redirect URI: Single-page application, http://localhost/. This covers any --port. A lab served elsewhere needs its own address here too, over https.
2. API permissions › Add a permission › APIs my organization uses › Microsoft Flow Service, delegated Flows.Read.All. If it isn't listed, open https://make.powerautomate.com once and search again.
3. Grant admin consent. No client secret is needed.

Connect the lab

1. Setup › Connections, the Microsoft 365 row's menu › Configure…:
   • Client ID: the app's Application (client) ID.
   • Tenant: the Directory (tenant) ID or one of its domains. Left empty, a single-tenant app's sign-in fails with AADSTS50194.
2. The same menu › Sign in, with the lab open at localhost. A sign-in from 127.0.0.1 reloads the page there and loses any open dialog.
3. Library › Sources › Add source › Power Automate workflow.

Or set M365_CLIENT_ID and M365_TENANT when starting the lab, for the whole machine; Setup then shows them read-only.

Upgrading

Read CHANGELOG.md first, in the package or on https://www.npmjs.com/package/evals-lab?activeTab=code. When its Upgrade notes say a release rewrites your data, backup the data directory before upgrading.

npm install -g evals-lab@latest   # upgrade
npm uninstall -g evals-lab        # remove; your data directory stays

Contact

[email protected] for questions, problems and feedback. More at https://www.grainbox.com/products/evals-lab/.

Licence

Copyright 2026 Grainbox Limited, under the PolyForm Internal Use License 1.0.0 (LICENSE.md): you and your company may use and change it for your own internal business, but not give it, or a changed copy, to anyone else, sell it, or offer it as a service. The demo files are public domain; each is credited in lab/demo/CREDITS.md.