jev-kit

A typed client and benchmark harness for Jev, TypeSafe AI's "System One" decision model, reached through the Vercel AI Gateway as typesafe-ai/jev. Jev does not write text: you send a state plus typed questions — a yes/no noul, a choice from a fixed list, or a score on an ordered scale — and it answers every question in one parallel call, with a calibrated confidence attached.

• Why
• Install
• Quick start
• The bench
• Recipes
• When Jev is the wrong tool
• What we measured
• For AI coding agents
• Contributing

Why

The real question when adopting a new decision model is never "does the client compile" — it is "is Jev accurate enough on my data, and where do I set the threshold". That question needs a harness, not a client: labeled examples, a way to run your real questions against them, and a report you can read a number off before a threshold ships into a live path. jev-kit is a small typed client (so a wrong option label is a compile error, not a runtime surprise) built around that harness.

Install

pnpm add @friqdev/jev-kit ai

Set AI_GATEWAY_API_KEY in the environment. It is read by the AI SDK at call time and is server-side only.

Quick start

import { createJev, noul, choice, score } from "@friqdev/jev-kit";

const jev = createJev();

const { answers, costUsd, latencyMs } = await jev.evaluate(
	{ subject: "Invoice #4471 overdue 45 days", body: "..." },
	{
		is_urgent: noul("Does this need a response within 24 hours?", {
			true: "explicit deadline, threat of service interruption, or an angry escalation",
			false: "routine, no time pressure stated or implied",
		}),
		category: choice("Which category best describes this ticket?", {
			billing: "payment, invoice, refund, or subscription questions",
			technical: "something in the product is broken or behaving unexpectedly",
			other: null,
		}),
		severity: score("How severe is the customer's situation, on our support ladder?", [
			{ what: "minor", examples: ["cosmetic issue, no impact on use"] },
			{ what: "major", examples: ["a core workflow is blocked, no workaround"] },
			{ what: "critical", examples: ["data loss or an outage affecting many users"] },
		]),
	},
);

answers.is_urgent.probability; // 0-1
answers.category.choice; // "billing" | "technical" | "other"
answers.severity.level; // nearest level, 0-based

Every field on the result is documented in src/types.ts, including which ones this kit normalizes from a raw Gateway response (confidence, the Score legend, and a Choice's runnerUp) rather than receiving as-is.

The bench

Run your real questions against a labeled JSONL file — one {"state": ..., "expected": {...}} record per line — and get back accuracy, latency, cost, calibration, and a threshold-coverage table you read a number off before wiring a threshold into a live path:

npx @friqdev/jev-kit bench ./fixtures.jsonl ./my-questions.ts

The questions module exports a QuestionMap, either named questions or as its default. A .ts module needs node --experimental-strip-types.

This is the headline feature of the kit: a client alone answers "can I call Jev", the bench answers "should I trust it here, and at what confidence cutoff". See src/bench.ts for the programmatic API.

What a run looks like

Real output, recipes/pr-finding-triage.ts against its 15 fixtures, from a Hetzner EU server on 2026-09-20 against typesafe-ai/jev:

Accuracy
  is_real (boolean): n=15 correct=13 accuracy=86.7%
  severity (score): n=15 correct=14 accuracy=93.3%, within-1 100.0%
  category (choice): n=15 correct=15 accuracy=100.0%
  needs_human (boolean): n=15 correct=7 accuracy=46.7%
  overall: n=60 correct=49 accuracy=81.7%

Latency (measured from wherever this bench ran, not TypeSafe's US-West figures)
  p50=809ms p95=1172ms max=1172ms

Cost
  total=$0.000601 mean/call=$0.0000401
  mean input tokens/call=954.6

Threshold / coverage
  >= 0.5: coverage=75.0% (n=45) accuracy=93.3% | below: accuracy=46.7% (n=15)
  >= 0.7: coverage=60.0% (n=36) accuracy=100.0% | below: accuracy=54.2% (n=24)
  >= 0.9: coverage=30.0% (n=18) accuracy=100.0% | below: accuracy=73.8% (n=42)

Two things to take from that, and they are why the bench exists rather than a client alone.

The threshold table is a decision you can ship: at a 0.7 cutoff this question set auto-handles 60% of answers at 100% accuracy and routes the rest, and that sentence is not one anybody could have written from the vendor's benchmark page.

And needs_human scoring 46.7% — worse than a coin flip — is the harness earning its keep. That question asks "would a senior reviewer want a human to look at this", which is an indirect, compound judgment, exactly the shape TypeSafe documents Jev as bad at. Three of the four questions are fine. Without measuring you would have shipped all four.

Recipes

recipes/ holds real question sets pulled from real projects, each with the source-project context left in a comment at the top of the file:

• pr-finding-triage.ts — from a self-hosted PR reviewer: re-scores a code-review finding's realness and severity from the finding text alone, because the reviewing LLM's own self-reported confidence is not calibrated and cannot be safely thresholded.
• source-triage.ts — from a knowledge-base ingestion pipeline: routes an incoming source before it enters the pipeline.
• capture-classify.ts — from a German family inbox app: classifies a captured note or message into the app's fixed set of item types.

Read them as documentation of question style, not as an API this package guarantees to keep matching your project's exact schema.

When Jev is the wrong tool

Jev cannot write text, summarize, explain its reasoning, generate code, do arithmetic, count reliably, or compare dates. It needs the full answer space enumerated before the call — it cannot answer an open-ended question. And it is not a security boundary: it reads adversarial or prompt-injected content as literally as anything else. If your task needs generated prose, an answer space you cannot enumerate yet, or a computed (not semantic) result, use something else.

What we measured

Checked on this machine on 2026-09-20; treat anything not listed here as unverified.

• Latency: three consecutive calls through the Gateway from a Hetzner EU server measured 504ms, 315ms, 272ms. TypeSafe quotes 70-500ms and "~100ms typical", but measures from the US West Coast. If you are calling from the EU, measure your own latency before promising a number to a UI — do not assume TypeSafe's figures transfer.
• Cost: a real 3-question support-ticket call used 464 input tokens, which the Gateway billed (as marketCost) at $0.000019488. Output tokens are free.
• Context window: vercel ai-gateway models endpoints typesafe-ai/jev reports a 32,000 token context on the Gateway path, at $0.000000042 per input token. TypeSafe's direct API advertises 64k — the Gateway path is the smaller of the two, and this kit talks to the Gateway path only.
• Versions verified on npm: [email protected] (experimental_evaluate needs >= 7.0.105), @ai-sdk/[email protected], @typesafe-ai/[email protected].
• Accuracy is not peak-of-market: TypeSafe's own "193.6x faster, 444.6x cheaper" figures come from vendor-authored workflow evals whose reference answers were the average of two frontier models. On TypeSafe's own eval, Jev scores 67.8% versus GPT-5.6 Terra's 67.9%, behind GPT-5.6 Sol (74.1%) and Claude Opus 5 (73.1%). Jev buys speed and price, not peak accuracy — validate against your own data with the bench before trusting it on a judgment that matters.

For AI coding agents

This repository ships an Agent Skill covering when to use Jev, how to write good questions, and threshold pitfalls. Install it with:

npx skills add FlorianRiquelme/jev-kit

A machine-readable index of the docs and source lives at llms.txt. Agents working on this repository itself should read AGENTS.md.

Contributing

Issues and pull requests are welcome. pnpm install, then pnpm check runs lint, typecheck, test and build. See AGENTS.md for repository conventions.