@bryel/evals

@bryel/evals is the zero-dependency TypeScript SDK for running eval work outside Bryel and attaching artifacts or final rubric judgments to Bryel runs. It supports a mixed benchmark case: one run can contain LLM-adjudicated and deterministically adjudicated rubrics at the same time.

Source: github.com/bryel-ai/platform/tree/main/packages/evals

Install

Install the published SDK in a Node 18+ server or trusted local harness:

npm install @bryel/evals

To build and verify the package from source:

git clone https://github.com/bryel-ai/platform.git
cd platform
bun install --frozen-lockfile
bun run --filter '@bryel/evals' test
bun run --filter '@bryel/evals' typecheck
bun run --filter '@bryel/evals' build

Credentials

The authoring, judgment-submission, and evaluation-job APIs require a secret project API key with the evals scope.

export BRYEL_API_KEY='bk_your_secret_project_key'
export BRYEL_API_ENDPOINT='https://app.bryel.ai/api/benchmarks' # optional

Keep BRYEL_API_KEY in a server-side secret store or local environment. Never embed it in browser code, commit it, print it, or send it to an end user. The SDK rejects publishable bkp_ keys before making these requests. A bkp_ key is only appropriate for browser-safe artifact ingestion where its origin is locked.

BRYEL_API_ENDPOINT is optional. It is useful for a self-hosted deployment or local API and maps to the SDK's endpoint option. The default is exported as DEFAULT_EVAL_BENCHMARKS_ENDPOINT.

The Mixed-Adjudication Model

Use one suite, one case, and one agent run. Put each requirement in a separate versioned rubric and declare how it is evaluated:

• deterministic: a named evaluator such as [email protected] returns a binary result locally or through a registered platform adapter.
• llm: a local named LLM evaluator can submit a binary result, or a platform job can run the platform's configured LLM judge.
• external: evaluation ran outside Bryel.
• platform: evaluation is requested or performed by Bryel.

The run is complete only when every active rubric version has an accepted result. This keeps a single comparable run and avoids duplicating the same case into deterministic and LLM suites.

1. Configure the Suite

Create the suite and configure its approvers in the Bryel Platform UI. Case imports intentionally cannot create or modify suite governance.

The executable example defines one deterministic rubric, one locally judged LLM rubric, and one platform-judged LLM rubric:

examples/mixed-adjudication.ts

2. Author Collision-Safe Rubrics

Local rubric definitions use a stable external identity:

{
  key: "no_visible_overlap",
  title: "No visible overlap",
  rubricText: "Visible elements do not overlap.",
  weight: 2,
  requiredEvidenceRoles: ["final_surface"],
  externalRef: {
    namespace: "acme.design",
    id: "no-visible-overlap",
    version: "1"
  },
  adjudicationKind: "deterministic",
  evaluatorKey: "dom.no-overlap",
  evaluatorVersion: "1.0.0",
  allowedExecutionOrigins: ["external"]
}

externalRef.namespace, externalRef.id, and externalRef.version identify the local definition. Reimporting the same reference and content is idempotent. Reusing it for different content returns 409 Conflict; publish a new external version instead.

Canonical Bryel IDs such as erub_... and erv_... are server-owned. Never send id, rubricId, rubricVersionId, or their snake-case equivalents in a local rubric definition. Bryel allocates them after import and approval.

Submit case definitions to an existing suite:

import { importEvalCases } from "@bryel/evals";

await importEvalCases({
  apiKey: process.env.BRYEL_API_KEY!,
  endpoint: process.env.BRYEL_API_ENDPOINT,
  suiteIdOrSlug: "my-suite",
  dryRun: false,
  cases: [caseDefinition]
});

An import may create change requests rather than activate changes immediately. Approve them in Bryel before fetching a manifest or running the case. Use dryRun: true to inspect the plan without writing.

3. Create the Agent Run and Fetch Canonical IDs

Run your agent harness for the approved case. Keep the returned runId; all judgments attach to that existing run. startEval() can create suite/model runs for compatible harnesses, while browser harnesses may create runs through their own Bryel integration.

Fetch the approved case manifest before evaluating. It is the authority for the exact case version, evaluator contracts, and canonical rubricVersionIds:

GET /suites/{suiteSlug}/cases/{caseKey}
Authorization: Bearer <BRYEL_API_KEY>

With the default base, the full URL is:

https://app.bryel.ai/api/benchmarks/suites/{suiteSlug}/cases/{caseKey}

Resolve local rubrics by externalRef, then use the returned rubricVersionId. Do not cache an ID across rubric-version changes.

4. Submit a Local Deterministic Result

Execute the deterministic evaluator locally against the run artifacts, then submit only its binary outcome and optional evidence metadata:

import { submitEvalJudgments } from "@bryel/evals";

const result = await submitEvalJudgments({
  apiKey: process.env.BRYEL_API_KEY!,
  endpoint: process.env.BRYEL_API_ENDPOINT,
  runId: "erun_...",
  submissionId: "erun_...:dom-no-overlap:v1",
  evaluator: {
    kind: "deterministic",
    key: "dom.no-overlap",
    version: "1.0.0"
  },
  executionOrigin: "external",
  judgments: [{
    rubricVersionId: "erv_from_manifest",
    met: true,
    reasoning: "The local geometry check found zero visible overlaps.",
    details: { overlapCount: 0 }
  }]
});

console.log(result.coverage);

Each judgment accepts met: true | false. Fractional scores are not allowed. Do not send score, weight, or awardedPoints; the SDK rejects them and the server never trusts them.

Bryel loads the versioned rubric weight and calculates:

score = met ? 1 : 0
awardedPoints = met ? weight : 0

This also handles penalties: when a rubric has weight -3 and met is true, the awarded points are -3.

5. Submit a Locally Computed LLM Result

Local LLM judging uses the same immutable submission API. Give a named local judge the exact key/version configured on the rubric:

await submitEvalJudgments({
  apiKey: process.env.BRYEL_API_KEY!,
  runId: "erun_...",
  submissionId: "erun_...:local-visual-judge:2026-08",
  evaluator: {
    kind: "llm",
    key: "local.visual-judge",
    version: "2026-08"
  },
  executionOrigin: "external",
  judgments: [{
    rubricVersionId: "erv_from_manifest",
    met: false,
    reasoning: "The primary action is visually ambiguous."
  }]
});

The server verifies that the run, case version, rubric, evaluator identity, execution origin, and optional job all match before accepting the submission.

6. Request or Cancel Platform Evaluation

Request a job only for rubric versions with the same evaluator contract:

import { createEvaluationJob, cancelEvaluationJob } from "@bryel/evals";

const job = await createEvaluationJob({
  apiKey: process.env.BRYEL_API_KEY!,
  runId: "erun_...",
  rubricVersionIds: ["erv_from_manifest"],
  evaluator: { kind: "llm", key: null, version: null },
  executionTarget: "platform"
});

await cancelEvaluationJob({
  apiKey: process.env.BRYEL_API_KEY!,
  jobId: job.id
});

Platform LLM jobs can be processed by Bryel. A deterministic platform job stays queued until a matching deterministic adapter is registered; Bryel does not execute arbitrary customer code implicitly. External workers can link their submission by passing the job's ID as evaluationJobId.

Cancellation is idempotent for canceled jobs. A succeeded job cannot be canceled.

Idempotency and Corrections

submissionId is required and scoped to a run. Choose a stable value from the run, evaluator, and evaluator version.

• Retrying the same run, submission ID, and payload returns HTTP 200 with idempotent: true.
• The first accepted submission returns HTTP 201.
• Reusing the ID with different content returns 409 Conflict.
• To correct a result, submit the corrected immutable payload with a new submissionId. Never overwrite accepted history.

Use the same object for a network retry so the immutable payload is byte-for- byte equivalent after SDK normalization:

const deterministicRequest = {
  apiKey: process.env.BRYEL_API_KEY!,
  runId: "erun_...",
  submissionId: "erun_...:dom-no-overlap:v1",
  evaluator: {
    kind: "deterministic" as const,
    key: "dom.no-overlap",
    version: "1.0.0"
  },
  executionOrigin: "external" as const,
  judgments: [{
    rubricVersionId: "erv_from_manifest",
    met: true,
    reasoning: "The local geometry check found zero visible overlaps.",
    details: { overlapCount: 0 }
  }]
};

const first = await submitEvalJudgments(deterministicRequest);
const retry = await submitEvalJudgments(deterministicRequest);

console.assert(first.idempotent === false);
console.assert(retry.idempotent === true);
console.assert(first.submission.id === retry.submission.id);

A correction is a new immutable submission. Keep the evaluator identity and rubric version, but use a new submission ID and the corrected boolean:

const correction = await submitEvalJudgments({
  apiKey: process.env.BRYEL_API_KEY!,
  runId: "erun_...",
  submissionId: "erun_...:dom-no-overlap:v1:correction-1",
  evaluator: {
    kind: "deterministic",
    key: "dom.no-overlap",
    version: "1.0.0"
  },
  executionOrigin: "external",
  judgments: [{
    rubricVersionId: "erv_from_manifest",
    met: false,
    reasoning: "Correction: the evaluator found one visible overlap.",
    details: { overlapCount: 1 }
  }]
});

For each active rubric, Bryel composes the newest accepted compatible submission. Where no submission exists, historical successful LLM judge-run results remain readable as a legacy fallback. Failed or rejected submissions never replace an accepted result.

The submission response reports:

{
  total: number;
  judged: number;
  pendingLlm: number;
  pendingDeterministic: number;
  complete: boolean;
}

Partial coverage is visible, but only complete runs participate in fully judged leaderboard statistics.

API Errors

The SDK throws an Error containing the HTTP status and response body for non-success responses:

| Status | Meaning | | --- | --- | | 400 | Malformed body, evaluator mismatch, duplicate rubric, invalid origin, or wrong job subset. | | 401 | Missing or invalid API key. | | 403 | Publishable key, missing evals scope, or wrong project permission. | | 404 | Run, job, case version, or rubric does not exist in the key's tenant/project. | | 409 | Reused external identity or submission ID has different immutable content, or the requested state transition conflicts. | | 500 | Unexpected platform failure; the public response does not expose internal error details. |

The SDK performs strict local validation before fetch: IDs must be nonempty, rubric IDs must be unique, evaluator contracts must be exact, results must be boolean, details must be a plain JSON object, and unknown request fields are rejected.

Run the Complete Example

First verify every documented request locally without credentials or network:

git clone https://github.com/bryel-ai/platform.git
cd platform
bun install --frozen-lockfile
cd packages/evals
BRYEL_EXAMPLE_STUB=1 bun run examples/mixed-adjudication.ts

The package test suite imports this same example and asserts every request URL, authorization header, payload, idempotent retry, correction, job creation, and cancellation. It is not a separate documentation-only mock:

bun run --filter '@bryel/evals' test

For a live project, set the required values and submit the example case:

export BRYEL_API_KEY='bk_your_secret_project_key'
export BRYEL_SUITE='your-existing-suite-slug'
export BRYEL_CASE_KEY='mixed-adjudication'
export BRYEL_API_ENDPOINT='https://app.bryel.ai/api/benchmarks' # optional

BRYEL_IMPORT_CASES=1 bun run examples/mixed-adjudication.ts

Approve the proposed case and rubrics in Bryel, then create the agent run for that approved case version. Export its run ID and execute without the import flag:

export BRYEL_RUN_ID='erun_for_the_approved_case'
bun run examples/mixed-adjudication.ts

The live command submits one deterministic result, verifies its idempotent retry, submits one local LLM result, and creates one platform LLM job. To also exercise immutable correction and cancellation explicitly, opt in:

BRYEL_DEMONSTRATE_CORRECTION=1 \
BRYEL_CANCEL_PLATFORM_JOB=1 \
bun run examples/mixed-adjudication.ts

Do not set BRYEL_CANCEL_PLATFORM_JOB=1 when the platform judge should actually complete the job. The example reads secrets only from the environment and never prints them.

Artifact Ingestion

The earlier artifact API remains available for agent harnesses. It attaches screenshots or output text to an existing run; it does not create the run:

import { startEvalSession } from "@bryel/evals";

const session = startEvalSession(sessionId, { apiKey: "bkp_origin_locked_key" });
await session.export({
  images: [base64Png],
  outputTexts: [finalAnswer]
});

Artifact ingestion may use an origin-locked publishable bkp_ key in a browser. Case authoring, final judgment submission, and evaluation jobs may not.