@evalgate/sdk

EvalGate helps you answer one release question: did this AI change preserve the behavior that is safe to ship?

The package combines five surfaces around that goal:

• a local evaluation and regression gate that can run without an EvalGate account;
• a TypeScript library for assertions, evals, traces, judges, and integrations;
• a CLI for setup, baselines, release decisions, and evidence workflows;
• optional hosted collaboration, repository intelligence, and shared evidence;
• CI output and exit codes designed for release automation.

Install it as a development dependency:

npm install --save-dev @evalgate/sdk

Node >=18 is supported. The OpenAI and Anthropic SDKs are optional peer dependencies; install only the provider libraries your application uses.

Start in a cloud-agent sandbox — no account or browser required

Availability: the credential-free init --local lifecycle below ships in @evalgate/[email protected]. Check the public feature-status page before depending on it in a critical release path.

Use this path when a coding agent receives an isolated checkout without a browser, saved login, GitHub remote, or interactive TTY. Here, local names the execution and evidence boundary: EvalGate runs against files in the current checkout without establishing hosted identity. The same boundary also works on a developer laptop, in a container, in an air-gapped repository, or in CI.

# Machine-readable preview. No writes and no network access.
npx /sdk init --local --format json

# Apply only after the agent/user has reviewed the plan.
npx /sdk init --local --apply --format json

# Replace the generated placeholder with real deterministic cases, then commit
# the durable config/spec files before accepting the first reviewed baseline.
npx /sdk baseline update

# Run the same gate after each change.
npx /sdk gate

The daily local loop is deliberately short:

change -> evalgate gate -> PASS / WARN / BLOCK + evidence

init --local requires neither login nor a GitHub remote. It performs no hosted API call and never fabricates hosted identity. JSON and non-TTY execution remain preview-only unless --apply is explicit. On a human TTY, bare init can preview and ask before writing; explicit --preview is always non-mutating.

The deterministic gate is offline by default. gate --allow-network is an explicit opt-in for provider-backed judges or other intended network evaluators. On Linux, EvalGate also attempts an unprivileged network namespace for project commands. Windows and macOS receive EVALGATE_OFFLINE=1; use Linux CI when hard operating-system network isolation is required.

What the local workflow writes

| Artifact | Lifecycle | | --- | --- | | evalgate.config.json | Durable, reviewable local gate configuration | | evalgate.project.json | Durable local project and baseline lineage; never a credential | | evals/**/*.eval.* | Durable evaluation specifications | | evals/baseline.json | Durable reviewed baseline; pending until baseline update succeeds | | .github/workflows/evalgate-gate.yml | Optional reviewable CI gate | | .evalgate/last-report.json | Ephemeral canonical report for the latest gate | | .evalgate/runs/ | Ephemeral local run evidence | | evals/regression-report.json | Regenerable detailed regression report |

Init merges managed ignore rules for EvalGate-owned ephemeral artifacts without ignoring the durable configuration, specifications, or accepted baseline. It does not overwrite existing project files silently.

Add hosted collaboration

Use the hosted path when a team wants shared history, browser review, managed traces, repository intelligence, organization policy, or cloud-backed evidence:

npx /sdk login
npx /sdk link
npx /sdk status
npx /sdk open

login creates one expiring, remotely revocable human CLI session in the operating-system user configuration directory. link associates the current organization, GitHub repository, and selected root while keeping credentials outside the repository. It never mints or replaces an automation key. link writes only non-secret version 2 repository context to .evalgate/project.json — durable organization/repository/root identity plus the last exact activation snapshot — never a credential.

Repository linkage is durable; every cloud intake and evidence artifact is pinned separately to an exact commit. Branch movement does not unlink a project. status reports the durable link, current checkout, cloud snapshot, local-gate readiness, and cloud-target readiness independently.

End the human session with:

npx /sdk logout

Logout revokes the saved CLI session remotely before removing it locally. If EVALGATE_API_KEY is set, it remains a separate credential in the current environment.

Local and hosted boundaries

| Command or action | Account | GitHub remote | Network | Writes repository files | Safe unattended default | | --- | --- | --- | --- | --- | --- | | init --local --preview | No | No | No | No | Yes | | init --local --apply | No | No | No | Yes | Only with explicit --apply | | baseline update | No | No | No by default | Yes, reviewed baseline/evidence | Run only on an intended clean source state | | gate | No | No | No by default | Ephemeral reports | Yes | | status --json | No for local status | No for local status | Hosted verification when a session/link exists | No | Yes | | login | Creates session | No | Yes | No | Human approval required | | link | Yes | Yes | Yes | Non-secret link context | Preview unless interactive confirmation or --apply | | open | Yes | Linked repository for repository views | Yes | No | No; opens a browser |

An EVALGATE_API_KEY is for authorized automation against an already established hosted context. It is not a bootstrap shortcut for login or link, cannot create a fake tenant for local mode, and must be supplied by the CI secret store rather than committed in evalgate.config.json or .evalgate/.

Check readiness without guessing

npx /sdk status
npx /sdk status --json

Status separates at least these decisions:

• whether the local gate can run;
• whether hosted identity and repository linkage are available;
• whether the current checkout is exact, behind, ahead, unpushed, or diverged;
• whether cloud evidence matches the current checkout;
• whether the current commit can be targeted by cloud evidence.

A repository can be locally gate-ready while hosted collaboration is unconfigured. A checkout behind its remote branch can remain both linked and locally ready. Status exit 0 means a status document was produced; it does not mean every readiness dimension is true. Use the structured readiness and nextActions fields for automation.

Use the TypeScript library

Create deterministic local assertions and evaluation cases:

import { createResult, defineEval } from "@evalgate/sdk";

defineEval("refund policy", async () => {
  const output = await supportAgent("How long do refunds take?");
  const passed = output.toLowerCase().includes("refund");

  return createResult({
    pass: passed,
    score: passed ? 100 : 0,
    output,
  });
});

Use @evalgate/sdk/testing for repeated trials and deterministic assertions, or the exported clients when an application intentionally talks to the hosted control plane:

import { AIEvalClient } from "@evalgate/sdk";

const client = AIEvalClient.init();
const evaluations = await client.evaluations.list();

// Purpose-built governed Hub clients share the web app's exact operations.
const prompts = await client.prompts.list();
const datasets = await client.datasets.list();

const trace = await client.traces.create({
  name: "support-run",
  traceId: `support-${Date.now()}`,
  metadata: {
    input: "Cancel my subscription",
    output: "I've canceled your plan effective today.",
  },
});

Public entry points include ., ./openapi, ./assertions, ./evaluators, ./trajectory-match, ./test-evidence, ./testing, ./evaluation-process, ./replay-decision, ./promote, ./regression, ./otel, and the OpenAI/Anthropic/Vercel AI integrations.

CI and machine output

Use JSON for agents and CI:

npx /sdk status --json
npx /sdk gate --format json
npx /sdk capabilities --format json

Machine mode emits one parseable document on stdout. The process exit status is the shell/CI result; JSON reports also carry the applicable exitCode or effectiveExitCode, reason code, and release-readiness fields. Key values are:

• 0 (PASS) — all required evidence passed; also confirm releaseReady: true for a release-bearing report;
• 1 and 2 — measured score/regression outcomes, not argument errors;
• 3 — policy violation;
• 4 — hosted API/network error;
• 5 (BAD_ARGS) — unknown command, removed command, or invalid arguments;
• 6–15 — named evidence, warning, baseline, provider, and integrity states;
• 99 — unexpected internal error;
• 130 — cancellation.

The installed, version-bound authority is always:

npx /sdk capabilities --format json

Missing, invalid, stale, or mismatched evidence never becomes a behavioral success merely because command execution completed.

Red Team release evidence

Red Team evidence is a first-class release input, not an undocumented API-only feature:

npx /sdk red-team evidence --run <campaign-run-id> --out .evalgate/red-team/evidence.json

The command fetches one exact hosted campaign, verifies its organization, target, snapshot, canonical hash, signature, lifecycle, and policy binding, and writes the artifact atomically only after admission succeeds. Syntax errors use BAD_ARGS; missing or mismatched policy/integrity evidence remains blocking; authentication/API failures remain distinct. A malformed or cross-tenant artifact is never admitted as weaker evidence.

Migrating from 3.8.x

Existing evaluation specs, configuration, and accepted baselines remain usable. Upgrading does not require reinitializing the repository or replacing a valid baseline. The command surface changed so identity, repository linkage, and automation credentials are no longer conflated:

| 3.8.x command | Current path | | --- | --- | | evalgate auth status | evalgate whoami | | evalgate auth configure --api-key ... | Human: evalgate login; automation: an authorized EVALGATE_API_KEY with an existing hosted context; local-only: evalgate init --local | | evalgate setup | evalgate login, then evalgate link | | evalgate setup status | evalgate status | | evalgate start | No direct alias: use evalgate init/init --local for one-time setup, evalgate gate for a release decision, or evalgate watch for continuous local reruns | | evalgate verify | evalgate status for readiness, then evalgate gate for the release decision | | --plan | --preview | | --yes | --apply in explicit automation |

Removed commands return targeted migration guidance. They do not restore repository-scoped secret storage or treat an automation key as a human actor.

Version 3.7.9 briefly wrote a credential to .evalgate/config.json. Current versions do not read that file. Delete it and rotate the credential if it may have entered source control.

Optional: understand the product (Experimental)

understand is an optional, Experimental product-context synthesis command. It is not required to initialize, accept a baseline, run evaluations, or determine whether a change is safe to release.

npx /sdk understand --format json
npx /sdk understand --apply

It reads a bounded set of local product files and produces a provisional, evidence-linked theory with confidence, unknowns, and risk hypotheses. Preview writes nothing. The command makes no model or network call and does not execute the application, accept a baseline, or promote evidence.

Product workflows

The purpose-built client and CLI cover the same governed workflows as the web app, including pack installation, exact-commit repository intelligence, Playground case authoring, Prompt Hub, and Dataset Hub.

npx @evalgate/sdk repo repositories
npx @evalgate/sdk repo scan --repository 42 --head-sha <sha>
npx @evalgate/sdk repo ask --repository 42 --question "What AI exists?"
npx @evalgate/sdk packs list --domain coding
npx @evalgate/sdk packs install coding-agent-release-safety
npx @evalgate/sdk playground from-trace --evaluation 7 --trace 123
npx @evalgate/sdk prompts --help
npx @evalgate/sdk datasets --help

Explore the rest of EvalGate

The capability contract is a better navigation surface than a flat command catalog:

npx /sdk capabilities
npx /sdk capabilities --format json

Additional workflows include traces and OpenTelemetry/OpenInference export, reviewed failure promotion, prompt and dataset sync, repository intelligence, evaluation packs, judge orchestration and calibration, application-health probes, and bounded improvement experiments. Hosted and Experimental maturity varies by surface; check feature status before making a production dependency.

Documentation:

• Changelog
• Quickstart
• CLI reference
• TypeScript SDK
• Feature availability
• EvalGate

Repository parity is tracked in docs/sdk-parity.md in the EvalGate repository.