neens-eval — pre-prod evaluation runner for TypeScript/Node (PULL + PUSH)

The thin CI glue that turns a Neens pre-prod evaluation into a one-liner — the TypeScript parity port of the Python neens-eval SDK, for the half of the agent ecosystem that ships in TypeScript. Before a change ships, replay a golden dataset's frozen prompts against your candidate agent, capture the resulting OpenTelemetry traces, let Neens score them with your existing judges, and gate the build on regressions vs a baseline.

Two models are supported. In the PULL model (below) your harness runs your agent per prompt and emits traces Neens correlates. In the PUSH model (neens-eval push) you register your agent as an agent_http connection and Neens calls it per prompt itself — no harness, no OTel wiring.

This runner does not try to be a universal agent-caller. In the pull model your harness runs your agent; the runner just:

1. fetches the run's frozen prompts,
2. runs your command once per prompt with the Neens correlation attributes injected into the environment,
3. starts the run, polls until scoring finishes, fetches the gate, and
4. returns an exit code (0 = gate passed) so CI can gate on it.

Requirement (be honest about it): your agent must already be OpenTelemetry-instrumented and exporting traces to the Neens OTLP receiver. Neens is your observability tool — it correlates the traces your agent emits; it does not instrument your agent for you. See the framework quickstarts for copy-paste OTel config (LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, Claude Agent SDK, Vercel AI SDK).

Zero runtime dependencies — the runner uses the platform fetch (Node 18+) and node:child_process only, so npm i neens-eval adds nothing to a CI image beyond the package.

Documentation: Pre-prod evaluations guide · Framework quickstarts · neens.ai

Install

npm install --save-dev neens-eval    # or: pnpm add -D neens-eval / yarn add -D neens-eval

This installs the neens-eval bin. Requires Node 18+ (for global fetch). Check it with npx neens-eval --version; pin the version in CI so a new release never changes a gate under you.

The CI one-liner

npx neens-eval run \
  --base-url "$NEENS_BASE_URL" \
  --api-key  "$NEENS_API_KEY" \
  --run-id   "$RUN_ID" \
  -- node dist/my-agent.js --answer   # <-- YOUR agent command, after `--`

Everything after -- is your command. It is invoked once per frozen prompt. The exit code is the gate verdict, so a GitHub Actions / GitLab step fails automatically when the gate fails:

- name: Pre-prod eval gate
  env:
    NEENS_BASE_URL: https://neens.example.com
    NEENS_API_KEY: ${{ secrets.NEENS_PROJECT_KEY }}   # a nk_live_… project key
    # Point YOUR agent's OTel exporter at the Neens receiver:
    OTEL_EXPORTER_OTLP_ENDPOINT: https://neens.example.com
    OTEL_EXPORTER_OTLP_HEADERS: Authorization=Bearer ${{ secrets.NEENS_PROJECT_KEY }}
  run: npx neens-eval run --run-id "$RUN_ID" -- node dist/my-agent.js --answer

Create the run in the same step

If you don't pre-create the run, --create makes one against a dataset's golden version:

npx neens-eval run --create \
  --dataset "$DATASET_ID" --version-label "$GIT_SHA" \
  --max-regressions 0 --min-pass-rate 0.9 \
  -- node dist/my-agent.js --answer

Which baseline? — --baseline

--max-regressions counts items the baseline passed and the candidate failed, so the baseline decides whether it means anything:

| --baseline | Compares each golden item with | | --- | --- | | branch:main | the newest completed run on that branch over the same frozen dataset version (resolved when the run is created, then pinned) | | run:<preprod_run_id> | one specific prior run | | prod / prod:30d | production scores over a window, matched by prompt text |

Default in CI (0.6.0+): with no --baseline, a run created in GitHub Actions or GitLab CI asks for branch:<the repository's default branch> (GitLab CI_DEFAULT_BRANCH; GitHub repository.default_branch from the event payload). Outside CI, or when the default branch can't be detected, the server default applies (the 7-day production window). An explicit --baseline always wins, and a gates-only run (no --dataset) gets no default. Run the job on pushes to the default branch too, so pull requests have a run to compare with. The CLI prints the baseline it got:

Created run ppr_51c0…
Baseline: run ppr_9b3e… — branch:main: newest completed run on branch 'main' for this dataset version

A baseline that matched none of the run's items compared nothing: the summary prints regressions: not compared (no baseline) plus a WARNING, and a max_regressions policy rule is treated as a missing baseline (on_missing_baseline), never as "0 regressions".

Enforce eval gates in CI — --gates active

An eval gate is a failure mode Neens already caught in production, frozen into a golden set and a judge. --gates active makes the build replay every gate that is active (enforced in CI) in the project, with YOUR agent, and fail when a gate's pass rate drops below its threshold — so a failure you fixed cannot quietly come back. Paused gates are not enforced.

npx neens-eval run --gates active -- node dist/my-agent.js --answer

• --dataset becomes optional: a gates-only run replays just the gates' golden sets. Pass --dataset too to gate the release on your own golden set in the same run.
• --gates gate-abc,gate-def enforces exactly those gates (a paused one is reported, not enforced). NEENS_GATES is the environment default for --gates.
• Commit, branch and pull request are detected from GitHub Actions (GITHUB_SHA, the PR head commit from the event payload, GITHUB_HEAD_REF/GITHUB_REF_NAME, refs/pull/<n>/merge) and GitLab CI (CI_COMMIT_SHA, CI_MERGE_REQUEST_IID, …); --git-sha, --branch, --pr-number and --pr-url override them. Without --version-label the candidate is labelled with the short commit. They are recorded on each gate's CI history in Neens.
• The summary prints one row per gate:

  eval gates:
    GATE                      ENFORCED  PASS RATE  THRESHOLD  SCORED  VERDICT
    FM: Wrong refund amount   yes       33%        70%        3/3     FAIL
    FM: Leaks internal notes  yes       100%       70%        4/4     PASS
  eval gates:  1/2 passed, 1 enforced gate(s) failing the build

A gate whose items produced no score is NO DATA and fails the build — measuring nothing is never a pass. A project with no active gates (and no --dataset) is a run error (exit 2), not a green build.

Declare the gates in the repo — the policy's gates block

Instead of --gates, the --gate-policy file can declare which eval gates this repo enforces, so the list is reviewed in pull requests like any other code:

{
  "gates": {
    "mode": "exact",
    "severity": "block",
    "enforce": [
      { "id": "gate-f3c1f69c137e", "min_pass_rate": 0.7 },
      { "id": "gate-f631a3197d43", "name": "Refund policy" },
      "gate-0a1b2c3d4e5f"
    ]
  }
}

• mode: "exact" (default) enforces exactly the listed gates; "include" enforces the listed gates plus every active gate ("enforce": [] in include mode is the same as --gates active).
• min_pass_rate overrides that gate's threshold for this repo; name is a comment for readers.
• guardrails (optional, per entry; neens-eval 0.8.0+) names business KPIs the candidate must not be predicted to worsen: [{"kpi": "<KPI id>", "tolerance": 0.01, "severity": "block"}] (tolerance in the KPI's own units — 0.01 is one point of a rate KPI; severity block (default) or warn, at most 10 per gate). Neens predicts each KPI's change from the gate's golden pass rate and the failure mode's measured cost, and grades it: a prediction beyond the tolerance, or one it cannot make (unknown), fails the gate under block; a straddling one is a warning. The summary prints a guardrail KPIs (prediction, correlational) block, a gate with a guardrail warning is never counted as passed, and server-reported guardrail warnings make the reported build verdict warned. A repo guardrail retunes or adds to the gate's own in Neens; it never drops one.
• Neens compares the declaration with the project and reports drift: a listed gate that is missing or paused in Neens, or (exact mode) an active gate the repo does not list (unlisted_active). severity: "block" (default) fails the build on drift (exit 1); "warn" reports it and passes. Drift appears in the policy rules as gate_drift:<gate id>, in the build verdict reported to Neens, and in --json as drift (next to gatesPolicy).
• A gates block and --gates / NEENS_GATES together are a usage error (exit 2) — one source of truth. A Neens server too old to understand the block is a run error (exit 2), never a green build.
• The repository is recorded on the run (GITHUB_REPOSITORY, CI_PROJECT_PATH, BITBUCKET_REPO_FULL_NAME, BUILD_REPOSITORY_NAME, or CircleCI's user/repo; override with --repo or NEENS_REPO), so the Eval Gates page shows which repo enforces each gate.

The summary adds a gates: repo vs Neens table:

  gates: repo vs Neens (exact mode, drift blocks):
    GATE                    IN REPO        IN NEENS  THRESHOLD     VERDICT  DRIFT
    Wrong refund (gate-a)   yes (min 80%)  active    80% (repo)    PASS     —
    Refund policy (gate-b)  yes            paused    70% (server)  FAIL     paused
  gate drift:  1 gate(s) differ between the repo and Neens — blocking the build
    ! gate-b is listed in the repo's gate policy but is paused in Neens

Local dev loop — neens-eval watch

watch is run --create on every file save. It watches your source tree, re-runs the gate against your local agent after each save, and prints what changed since the previous iteration: items that newly fail (▼), newly pass (▲) or still fail, the gate verdict, and the pass-rate / avg-score / failure deltas.

npx neens-eval watch --dataset "$DATASET_ID" --version-label feat-refunds \
  --include "*.ts" --include "prompts/*" \
  -- node dist/my-agent.js --answer

• Every iteration is a real pre-prod run and scores every golden item with your judges, on your LLM connection. That costs judge tokens on every save. Cap a session with --max-iterations N.
• Iteration N is labelled <label>-watch.<session>.<N> (UTC start time as the session).
• Dependency-free polling watcher (no recursive fs.watch): --path (repeatable, default .), --include / --exclude globs, --debounce (default 1s). .git, node_modules, __pycache__, .venv/venv, dist, build and tool caches are always skipped.
• Runs never overlap. Saves made while an iteration runs produce exactly one follow-up.
• --once runs a single iteration and exits with the gate code (0/1/2). Otherwise Ctrl-C kills the agent processes, cancels the in-flight run, and exits 130.
• A failing iteration is printed, its run is cancelled, and the watcher keeps going. A run that cannot be created at all (bad key, unknown dataset, no golden version) exits 2.
• --json writes one JSON object per iteration to stdout.

Same flags and behaviour as the Python neens eval watch.

How each prompt reaches your command

For each frozen prompt the runner spawns your command with:

• stdin — the prompt input, so a cat-style agent works out of the box.
• $NEENS_ITEM_INPUT — the same prompt, for agents that read env instead of stdin.
• OTEL_RESOURCE_ATTRIBUTES — the three Neens correlation attributes (neens.eval_run_id, neens.dataset_item_id, neens.version_label) merged into any value you already set, so every span your agent exports is correlated to this run + item + version.
• NEENS_EVAL_RUN_ID / NEENS_DATASET_ITEM_ID / NEENS_VERSION_LABEL — the same values as plain env vars.

Your process environment is never mutated — a fresh env object is built per item.

Under sweep-run every invocation also gets the arm it belongs to:

• $NEENS_EVAL_MODEL — the arm's model. Route your LLM calls to it, or every arm runs the same model.
• $NEENS_SWEEP_ID / $NEENS_SWEEP_ARM / $NEENS_SWEEP_REP — the sweep, the arm label and the repetition (1..k); also stamped as the neens.sweep_id / neens.sweep_arm / neens.sweep_rep resource attributes.

sweepContext() reads these back as {sweepId, arm, rep, model} (or null outside a sweep).

PUSH model — Neens calls your agent endpoint (no harness)

Register your agent as an agent_http connection in Neens (Settings → Connections), then let Neens call it once per frozen prompt. No local command, no OTel wiring:

npx neens-eval push \
  --base-url "$NEENS_BASE_URL" --api-key "$NEENS_API_KEY" \
  --dataset "$DATASET_ID" --version-label "$GIT_SHA" \
  --agent-connection "$AGENT_CONN_ID" \
  --baseline prod:7d --min-pass-rate 0.9

Same exit-code contract (0 = gate passed). Neens captures + links each response as a session, scores it with your judges, and gates the result.

Gate-as-code (client-side policy)

The server gate owns two authoritative knobs (--max-regressions, --min-pass-rate). For richer budgets — cost/latency/step ceilings, per-metric floors, absolute failure caps — commit a neens-gate.json policy file and pass --gate-policy:

npx neens-eval run --run-id "$RUN_ID" --gate-policy neens-gate.json -- node dist/my-agent.js

See neens-gate.example.json (shipped in this package, at node_modules/neens-eval/neens-gate.example.json) for the full shape. A block-severity rule that fails blocks the build (exit 1); warn-severity rules are reported but never block. Rules whose signal is unavailable degrade to skipped (baseline-relative rules honor on_missing_baseline) — the policy never turns a data gap into a crash. A cost rule is a special case: when a side of the comparison ran entirely on models with no rate in the price table, avg_cost is null and the rule is reported skipped because it is unpriced (the unpriced models are named in the reason) — it is never silently passed, and never reported as a missing baseline. Set the rate in Settings → Model pricing.

JSON only. Unlike the Python SDK (which optionally reads YAML when PyYAML is installed), this zero-dependency package reads JSON policy files; a .yml/.yaml path is rejected with a clear error. Convenience flags --max-cost-delta-pct, --max-latency-delta-pct, --min-avg-score, and --max-abs-failures inject/override block rules without a file.

The server gate speaks cost too

The policy's top-level gate block — the one handed to createRun at run-create time — accepts the same cost family, spelled identically to the rules[].cost dimension, so a budget can live server-side where every consumer of GET /preprod-evals/{id}/gate sees it (the UI, another CI job, a model sweep), not just this CLI process:

{
  "gate": { "max_regressions": 0, "min_pass_rate": 0.9,
            "cost": { "max_avg_usd": 0.004, "max_delta_pct": -50 } },
  "rules": [ { "cost": { "max_avg_usd": 0.05 }, "severity": "warn" } ]
}

Two evaluation sites, one vocabulary. rules[].cost is evaluated client-side by this runner against the /comparison payload — it carries a severity and honours on_missing_baseline. gate.cost is evaluated server-side and is stored with the run; it has no severity, and a breach fails the gate. The same three keys are accepted in both places (an unknown one inside gate.cost is a GatePolicyError at parse time), and the SDK omits cost from the posted gate entirely when the policy declares none — it never sends "cost": null, which would fingerprint differently from an absent key server-side and void in-flight model sweeps.

The gate response then carries a costRules array ({rule, status, observed, threshold, reason}), absent when the gate declares no cost family. status is pass|fail|skipped, and only fail blocks: a run whose sessions all ran on models with no price reports skipped, naming the unpriced models — never a pass.

The build verdict is reported back to Neens

After neens-eval run / neens-eval push decides its exit code, it reports the build verdict to Neens (POST /preprod-evals/{id}/ci-result): passed, warned (a warn rule fired, or --no-gate let a breach through), blocked (exit 1) or errored (exit 2), with the exit code and every policy rule's rule/observed/threshold/severity/outcome/message, the policy file's name and SHA-256, and the SDK name + version. The run page and the Eval Gates strip then match the CI check — e.g. "blocked by policy: regressions 4 > 0 (all enforced gates passed)" — instead of showing only the server gate.

• Best-effort. A failed report never changes the exit code; the CLI prints one line to stderr: neens: warning: could not report the build verdict to Neens (…); the exit code is unaffected.
• No policy? The verdict still reflects the server gate and the exit status (rules: []).
• --json carries buildVerdict: {"verdict": "blocked", "reported": true}.
• neens-eval watch (a local loop) never reports. Library callers can opt out with runEval(..., { reportCiResult: false }).
• A CI run created by an older CLI (< 0.6.0) shows build verdict unknown in Neens — never passed.

Multi-model sweeps — sweep-preview / sweep-start / sweep-get / sweep-decide / sweep-run

A sweep replays ONE frozen golden dataset version against N candidate model arms, k times each, under identical conditions (same items, same judges, same k, same gate) and reports pass^k per arm. It answers which model should we ship, so the CI story is estimate, then decide:

# 1. What would this cost? Writes nothing, spends nothing.
neens-eval sweep-preview \
  --dataset-id ds_checkout \
  --arm "sonnet=conn_sonnet" --arm "haiku=conn_haiku" --pass-k 3

# 2. Launch it, and wait for the per-arm verdict.
neens-eval sweep-start \
  --name "nightly model bake-off" --version-label "$GIT_SHA" \
  --dataset-id ds_checkout \
  --arm "sonnet=conn_sonnet" --arm "haiku=conn_haiku" \
  --pass-k 3 --budget-usd 10 --wait --json

# 3. Or poll it later.
neens-eval sweep-get msw-4f2c91ab0d3e

# 4. Get the ANSWER: which arm should we ship, and does it differ per agent?
neens-eval sweep-decide --sweep-id msw-4f2c91ab0d3e --bar 0.9

Each --arm is LABEL=CONNECTION_ID, where the connection is a registered agent_http LLM connection whose endpoint runs that model. Two arms on the same connection is rejected — that is the same model twice, not a comparison. Give one of --dataset-id (its frozen golden version), --dataset-version-id, or --scenario-suite-id. Identical flags and exit codes to the Python SDK's neens sweep … — the two stay in lockstep.

sweep-run — runner mode (no agent endpoint)

sweep-start needs one publicly reachable agent_http endpoint per model, because Neens calls it. sweep-run is the pull-mode twin of neens-eval run: the CLI runs your own agent command once per arm x repetition x golden item, so it works on a laptop or in CI with no public URL and no tunnel.

npx neens-eval sweep-run --create --name "model choice" --dataset ds_checkout --version-label "$GIT_SHA" \
  --arm haiku=claude-haiku-4-5 --arm sonnet=claude-sonnet-4-6 --arm oss=openai/gpt-oss-20b \
  --k 3 --concurrency 2 --budget-usd 5 \
  -- node ./eval-item.js

Here --arm is LABEL=MODEL, not LABEL=CONNECTION_ID. The model id is handed to your command as $NEENS_EVAL_MODEL; your agent must route its LLM calls to it. If every arm's traces show the same observed model, the verdict carries a prominent WARNING [arms_same_observed_model] — the agent is probably ignoring $NEENS_EVAL_MODEL.

What it does:

1. Creates a runner sweep (--create) — or attaches to one created elsewhere (MCP, API) with --sweep-id ID. A push sweep is refused. The sweep id is printed first, so a CI log can link it.
2. Runs your command for every arm x rep x item with the run environment plus $NEENS_EVAL_MODEL / $NEENS_SWEEP_ID / $NEENS_SWEEP_ARM / $NEENS_SWEEP_REP. Work is interleaved — rep 1 of every arm before rep 2 of any, arms round-robin within a rep — under one global --concurrency, so provider load and rate-limit drift never land on one arm. Each arm x rep's run is started as soon as all its items have executed; a transient error on that start (network, 5xx, 429) is retried up to 3 times with backoff.
3. Polls until the sweep is scored, then prints the same per-arm table and verdict as sweep-decide (pass rate with n and CI, pass^k, cost per case, regressions) plus any warnings.

Flags: --create --name --version-label --dataset-id/--dataset --dataset-version-id --scenario-suite-id --arm LABEL=MODEL --pass-k/--k --judge-deployment-id/--judge --budget-usd (create), --sweep-id (attach), --concurrency N (default 1), --timeout S (per invocation), --bar, --baseline-arm, --poll-interval (default 10, must be > 0), --poll-timeout (default 3600), --json (adds an invocations list, and failedStarts when a run could not be started), plus the connection flags. Identical to the Python SDK's neens sweep run.

| Exit code | sweep-run situation | | --- | --- | | 0 | An arm cleared the bar — the decision is made. | | 1 | The sweep was scored and no arm clears the bar. | | 2 | Run error: void / failed / cancelled sweep, poll timeout, API error (including a budget refusal), a push sweep, or every agent invocation failed (the sweep is then cancelled). Also a child run that still could not be started after the retries: the other runs are driven to the end, the sweep is not cancelled, and the log prints the neens-eval sweep-run --sweep-id ID -- <command> that resumes it. | | 130 | Interrupted (Ctrl-C or SIGTERM): the agent processes are killed and the sweep is cancelled server-side, so it does not sit waiting for traces. |

A single failing invocation behaves exactly as in run: it is logged, its run is still started, and the judges score what arrived.

In GitHub Actions — no tunnel, no inbound endpoint:

- name: Model sweep (pass^k per model, no public endpoint)
  env:
    NEENS_BASE_URL: https://neens.example.com
    NEENS_API_KEY: ${{ secrets.NEENS_PROJECT_KEY }}   # a nk_live_… project key
    # Point YOUR agent's OTel exporter at the Neens receiver:
    OTEL_EXPORTER_OTLP_ENDPOINT: https://neens.example.com
    OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: https://neens.example.com/v1/traces
  run: |
    npx --yes neens-eval sweep-run --create --name "model choice ${{ github.sha }}" \
      --dataset "${{ vars.NEENS_GOLDEN_DATASET }}" --version-label "${{ github.sha }}" \
      --arm haiku=claude-haiku-4-5 --arm sonnet=claude-sonnet-4-6 \
      --k 3 --concurrency 2 --budget-usd 5 \
      -- node ./eval-item.js

The step fails (exit 1) when no model clears the bar and errors (exit 2) on a void or failed sweep; cancelling the job (SIGTERM) cancels the sweep too.

sweep-decide — the verdict

sweep-decide reads GET /model-sweeps/{id}/comparison and prints the server's verdict: the winning arm and why, a per-arm table (pass rate with its sample size and confidence interval, pass^k, cost per case, regressions), and one verdict per agent — because "Haiku is good enough" is routinely true for one agent and false for another.

Sweep msw-4f2c91ab0d3e — status=completed
  VERDICT: haiku clears the 90% bar at $0.000600 per case — 81% cheaper than sonnet.
  bar 90.0% (source: gate) | baseline: sonnet (incumbent) | 40 items x k=3
  ARM     MODEL              PASS RATE                    PASS^K    COST/CASE  REGRESSIONS
  sonnet  claude-sonnet-4-6  95.0% (n=40, CI 83.0-99.0%)  3/3 PASS  $0.003100  0
  haiku   claude-haiku-4-5   92.5% (n=40, CI 80.0-98.0%)  3/3 PASS  $0.000600  3
  local   gpt-oss:20b        no data                      0/3 —     unpriced   —

Flags: --bar 0.9 (the pass rate an arm must clear — omit it and the sweep's pinned gate.min_pass_rate is used, else the deployment default; the source is always reported), --baseline-arm ID (the incumbent the savings are measured against — defaults to the arm at position 0, never the best performer, because a baseline chosen after seeing the results is not a baseline), --require-winner, --wait, --json.

Two properties this command will not violate:

• The ranking is the server's. sweep-decide renders verdict/agents[].verdict verbatim — it never computes a second ranking, because two rankings that can disagree is worse than one.
• An unpriced arm never wins "cheapest." No rate means unpriced, never $0.00: it is still shown, still judged against the bar, and excluded from the cost ranking with a named reason.

Exit codes

| Code | Meaning | | --- | --- | | 0 | Gate passed (or --no-gate and the run did not error). | | 1 | Gate failed — a server-gate breach, an enforced eval gate that failed or scored nothing, or a block-severity policy rule. | | 2 | Run error — bad flags, no items, a run that finished failed, or a transport/timeout error. |

--no-gate reports the gate and exits 0 on a failed gate; a run error still exits 2. --json prints a machine-readable result to stdout (human logs stay on stderr). watch --once follows this table; a continuous watch exits 130 on Ctrl-C, 2 when no run can be created, or the last iteration's code at --max-iterations.

For a sweep the contract differs, on purpose:

| Code | Situation | | --- | --- | | 0 | The sweep completed — including when an arm missed pass^k. A sweep is a model-selection decision, not a gate. Also: launched without --wait. | | 1 | --require-all-arms was passed and some arm did not hit pass^k (or there is no verdict yet). | | 2 | The sweep is void — the arms did not run under identical conditions, so the comparison is refused and voidReason is printed. A result we cannot trust must never look like a pass. Also: the sweep failed, or a transport/API error (a budget refusal is a 422). |

For sweep-decide the same table holds, with --require-winner in place of --require-all-arms: a comparison that names no clearing arm still exits 0 (naming a winner is a decision, not a gate), and a void / comparable: false comparison is exit 2 before any verdict is read — a void sweep still has per-arm numbers, and reading them first is exactly how an untrustworthy comparison gets to look like a pass.

--require-all-arms (and --require-winner) implies --wait. An estimate including a model with no price in your tenant's table is printed as partial, naming the unpriced models — the total covers the priced arms only and is a floor, never the cost.

Programmatic use

The same building blocks are exported for use inside your own Node tooling:

import { NeensEvalClient, runEval, GatePolicy } from "neens-eval";

const client = new NeensEvalClient(process.env.NEENS_BASE_URL!, {
  apiKey: process.env.NEENS_API_KEY,
});

const outcome = await runEval("ppr_123", ["node", "dist/my-agent.js"], {
  client,
  policy: GatePolicy.fromFile("neens-gate.json"),
  concurrency: 4,
  log: () => {},          // suppress the default stderr logging
});

process.exit(outcome.exitCode);

evaluate(policy, { comparison, metrics, run }) is a pure function over the backend payloads, so you can unit-test your gate policy with zero network access. NeensEvalClient is a thin typed client over the /preprod-evals routes; inject a fetchImpl to test without a live server.

License

Apache-2.0 — see the LICENSE file shipped with this package.