finest

The Finest data-plane shim. It runs inside your process, watches what your app asks AI to do, and — once a cheaper configuration has been proven on your own workload — switches that job over safely.

This README opens with what the SDK will and will not do to your traffic, because that is the only part you have to trust before you install it. Every claim below points at the line of code and the test that enforces it. The brief's promise is that you can read every line that touches your traffic. It is Apache-2.0 and the runtime implementation is in src/.

What this does to your traffic

Finest has two doors. This section describes Door 2 — wrap() on your own keys, the default install. Door 1 (door1() — the Finest key) is different by explicit, disclosed choice: traffic you address to the Finest gateway is served by Finest, with a receipt on every request and a fail-open shim that replays straight to your provider on your own key whenever the gateway fails before serving a call. → src/door1.ts, test/door1.test.ts

On Door 2, your provider destination stays unchanged until an eligible signed policy is active.

1. Requests go straight from your process to your provider, on your key. Nothing on this door is proxied through Finest. Our control plane is never synchronously required. → src/transport.ts

2. The SDK may never throw into your request path. Every stage of the hot path is individually wrapped. Any internal failure — a bad policy, a corrupt cache, an unreachable control plane, a full disk, a bug of ours — logs once per rate-limit window and passes the call through to your original client, unchanged. → src/guard.ts, src/pipeline.ts, proven for all ten stages by name in test/passthrough.test.ts

3. Your provider's own errors reach you untouched. The one exception the guards must not absorb is your provider's. A rejection from your client is tagged so it is re-thrown as the original object, never swallowed and never turned into a second dispatch. A challenger transport rejection is treated as post-dispatch/billing-ambiguous: it is journalled as provider_error, its worst-case reservation remains accounted, and the incumbent is not retried. → ProviderCallError in src/guard.ts

4. An unannotated call is observed and never routed. Traffic is promoted out of observe mode only by an explicit annotation you write at the call site. There is no inference and no heuristic. → src/routes.ts

5. FINEST_DISABLE=1 returns your client object itself. Not a pass-through proxy — the same reference, Object.is-identical, with no code of ours between you and your provider. → test/disable.test.ts

6. A refusal is never a fallback trigger. A model declining a request is a legitimate outcome of that request. The SDK reports it and releases it; it never re-runs a refused prompt on another model looking for compliance. → runValidators in src/pipeline.ts, test/pipeline.test.ts, test/stream.test.ts

7. Ambiguous dispatch is never retried, even pre-first-byte. A network error before a response byte does not prove that the provider never accepted or billed the request. Stream requests stay on the incumbent: routed streaming remains disabled until a crash-durable two-phase journal can commit an active dispatch before any byte is released. → src/stream.ts

8. No healthy durable journal ⇒ no routing. If the SDK cannot durably record what it did, it will not do anything. It falls back to observer/passthrough mechanically, and Finest.state() says so. → durableSink in src/pipeline.ts, src/journal.ts

9. Nothing is dispatched before its worst case is priced. max_tokens × the price epoch is checked against this SDK process's rolling spend-at-risk guardrail before a challenger request is sent. It is not a workspace/monthly budget; invoices and the control plane remain aggregate authority. → src/reservation.ts

10. Added latency is measured, not asserted. See Overhead below. → src/overhead.bench.ts

What it does not do

• It never sends or stores prompts or outputs. The in-process SDK necessarily reads request and response objects transiently to canonicalize portable calls and validate a challenger. The journal carries token counts, latencies, outcomes and hashes, never content; the Finest control plane does not receive normal request or response payloads.
• It does not hold a provider key of its own. It resolves the key name you already have in your environment and hands it to your provider. Key material is never logged, hashed, fingerprinted or transmitted anywhere else.
• It does not route anything outside the portable subset. Tool calling, provider-managed state, image URLs and unbounded output are not canonicalizable, so those calls are observed and left alone. → src/shapes.ts
• It does not stop your agent loops. The budget governor clamps and demotes and emits events; it never fails a request to enforce a budget. Stopping a loop is your decision, not a shim's. → src/reflexes.ts

Install

Do not install finest from npm today: that name resolves to an unrelated package. Until an owned distribution name is selected or acquired, build the artifact from this repository and install the explicit tarball in the target project:

pnpm --filter finest-ai pack --pack-destination /tmp
# Run in the target project:
npm install /tmp/finest-0.1.0.tgz

Quickstart

import OpenAI from 'openai';
import { Finest } from 'finest-ai';

const finest = Finest.init({
  apiKey: process.env.FINEST_API_KEY,
  // Production must point at persistent storage; /tmp is never assumed.
  spoolDir: process.env.FINEST_SPOOL_DIR,
});
await finest.ready(); // authenticated bootstrap; never sits on the request path

const openai = finest.wrap(new OpenAI());

// Unannotated: observed, never routed.
await openai.chat.completions.create({
  model: 'your-model',
  messages: [{ role: 'user', content: ticket }],
  max_tokens: 256,
});

// Annotated: eligible for routing once a signed policy exists for this route.
// The `finest` key is stripped before the request reaches OpenAI.
await openai.chat.completions.create({
  model: 'your-model',
  messages: [{ role: 'user', content: ticket }],
  max_tokens: 256,
  finest: { route: 'classify_ticket', stickyKey: conversationId },
});

Anthropic and Google clients work the same way — Finest.wrap detects them structurally (client.messages.create, client.models.generateContent) and imports no provider SDK, so it adds nothing to your dependency tree and cannot break when a provider ships a major version.

import Anthropic from '@anthropic-ai/sdk';
const anthropic = Finest.wrap(new Anthropic());

That one Finest key bootstraps the authoritative workspace id, policy signing keys, revocation epoch, hard constraints, workspace-scoped pseudonym key, and the human-admitted runtime catalog. Provider requests still use your normal provider environment keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, and so on) and go directly to that provider. A missing provider key, exact tuple, adapter revision, or effective price epoch is an explicit observer/incumbent decision, never a guess.

For a local observer that must not contact Finest, say so explicitly:

const finest = Finest.init({ offline: true });

Route configuration (required for challenger routing)

// finest.routes.ts
import { artifactRevision, defineRoutes, validators } from 'finest-ai';

const classifyPrompt = { id: 'classify-ticket', version: 3, system: CLASSIFY_SYSTEM_PROMPT };
const classifySchema = { labels: ['billing', 'technical', 'account'] };

export default defineRoutes({
  classify_ticket: {
    promptRev: artifactRevision(classifyPrompt),
    schemaRev: artifactRevision(classifySchema),
    stickyKeyPath: 'metadata.conversation_id',
    releaseMode: 'buffered_validate',
    validators: [validators.oneOf(['billing', 'technical', 'account'])],
  },
  extract_financials: {
    promptRev: artifactRevision({ id: 'extract-financials', version: 2 }),
    schemaRev: artifactRevision(FINANCIALS_SCHEMA),
    stickyKeyPath: 'metadata.accession',
    validators: [validators.json(), validators.requiredFields(['cik', 'revenue'])],
  },
});

templateHash is only a structural discovery/drift signal and deliberately excludes user message content. It is not proof of prompt quality or routing authority. The explicit promptRev and schemaRev above must match both signed policy tuples or the SDK records a fallback and stays on the incumbent.

import routes from './finest.routes.js';
Finest.init({ apiKey: process.env.FINEST_API_KEY, routes });

// …or let the SDK find the file itself. Loading happens out of band and is
// entirely best-effort: a missing or broken file leaves you in observer mode
// rather than throwing at import time. Anything you pass as `routes` wins.
Finest.init({ apiKey: process.env.FINEST_API_KEY, routesFile: true });

Validators run on a challenger output only, in buffered_validate mode. On failure the challenger's output is discarded and the original input is re-run on your incumbent; both attempts are journalled. A validator that throws counts as a challenger failure, never as a failure of your request.

The hot path

Ten stages, in this order. Each is wrapped; each degrades to your own configuration.

| # | Stage | What it decides | Degrades to | |---|---|---|---| | 1 | kill_switch | FINEST_DISABLE / disable() | pure passthrough | | 2 | route_resolution | trusted annotation only | observe-only | | 3 | policy_lookup | installation routing authority + signed, lifecycle-valid, epoch-current policy | incumbent-direct | | 4 | constraint_assert | live constraints and explicit prompt/schema revisions match signed authority | incumbent + event | | 5 | sticky_assign | hash(HMAC(stickyKey, assignment-domain)) mod 10000 < pctBps | incumbent + event | | 6 | lever_apply | output caps and stop sequences, from an active signed policy only | no levers | | 7 | compile_reserve | adapter compile + worst-case cost reservation | incumbent + event | | 8 | dispatch | direct, on your key | incumbent | | 9 | release | exact model attribution and buffered validation; stream routes fail closed | incumbent | | 10 | journal | 100% attempt capture | logged gap |

The stage list is exported as PIPELINE_STAGES and the passthrough suite iterates it, so a stage cannot be added without a survivable failure path.

Sticky assignment is never a request id

Bucketing per request would split one conversation across two models, which makes every paired statistic downstream a measurement of noise and shows a user two different models mid-thread. The unit is a conversation, a document or an end user. A route with a canary and no sticky unit is not routed — it runs on your incumbent and emits an event asking for one. There is no fallback that quietly invents a unit.

Raw units are never persisted or sent to a provider. A stable workspace-owned HMAC key derives separate pseudonyms for assignment, journal evidence, and provider cache keys. Without that key, canary traffic stays on the incumbent and provider cache keys are omitted.

Local reflexes

All three work with the control plane unreachable, on an injected clock, with zero network.

• Failure-streak quarantine — N consecutive challenger failures demote the route to the incumbent for a cooldown.
• p95 latency breaker — challenger p95 above floored incumbent p95 × 1.25 over a rolling window demotes and reports. The floor stops a 4 ms route from tripping on a millisecond of noise.
• Agent budget governor — per-route token budgets and loop detection by request fingerprint, in flag and enforce modes. enforce clamps max_tokens and demotes to the incumbent; it never throws.

The journal

JournalSink is pluggable. The Node reference implementation (FileSpoolJournalSink) is an append-only NDJSON spool with an explicit fsync per batch, drained idempotently over HTTPS to POST /v1/ingest/journal.

• record() is synchronous and never blocks. An attempt is committed once the flush() covering it resolves ok.
• The sink starts non-durable and becomes routing-capable only after a real write plus fsync probe succeeds. Its short flush timer remains referenced so an ordinary early process exit cannot outrun the first scheduled write.
• A killed process loses nothing it committed — tested against a real SIGKILL, not a mock.
• A torn trailing line (the signature of a process killed mid-append) is truncated and reported as a process_exit gap.
• A full disk produces a visible gap marker, not a silent hole and not a thrown error. Same for a spool overflow or a serverless isolate frozen before a flush could finish.
• A gap freezes evidence and blocks fee eligibility for its interval. That is the point: we would rather bill you for less than claim savings we cannot prove.

Every simultaneously running process or replica must use its own persistent FINEST_SPOOL_DIR and FINEST_CACHE_DIR; never mount one mutable directory into multiple live processes. The SDK writes a private owner record and fails closed without touching journal or cache state on a live, foreign-host, or malformed owner. A complete dead same-host owner is reclaimed automatically.

Serverless. Pass waitUntil and the sink hands each flush to the platform instead of relying on a timer the runtime may freeze:

Finest.init({
  apiKey,
  spoolDir: process.env.FINEST_SPOOL_DIR,
  waitUntil: (p) => ctx.waitUntil(p),
});

Policies

A policy is a signed (ed25519, RFC 8785 canonical JSON) document from the control plane. @finest-ai/policy's evaluatePolicy is the single decision point, and this SDK calls it in exactly one place: admission, out of band.

Signature verification is real elliptic-curve work and does not belong in a <1 ms budget on every request, so the hot path re-checks only what changes between two calls — the lifecycle phase (policyPhase, imported, never re-implemented), the revocation epoch and the generation. A document that never passed evaluatePolicy is never in the cache; a document in the cache stops being usable the instant it leaves the usable phases.

Refresh is out of band and never blocks a request. A stale policy keeps serving through refresh_after → grace_until → hard_expires_at. Past hard expiry the answer is your own configuration, and no amount of control-plane unavailability changes that.

Overhead

Measured with a mocked provider, pairwise against an unwrapped call so process-level noise cancels. Run it yourself:

pnpm --filter finest-ai run bench:overhead

On the reference machine (Apple Silicon, Node 23.6, 10,000 iterations):

| path | p50 | p95 | p99 | |---|---|---|---| | observed (unannotated, journalled) | 0.0095 ms | 0.0126 ms | 0.0182 ms | | routed → incumbent (canary miss) | 0.0093 ms | 0.0115 ms | 0.0150 ms | | routed → challenger (full pipeline) | 0.0169 ms | 0.0208 ms | 0.0287 ms |

That is roughly 10–17 microseconds of added latency, against a 1 ms budget and a 5 ms p99 ceiling. Real provider calls take 300–3,000 ms, so the shim is three to five orders of magnitude below the thing it is measuring.

test/overhead.test.ts fails the build if any path's median exceeds 1 ms. It also asserts that each benchmarked path actually exercises the arm it claims to, so the numbers cannot quietly become a measurement of the wrong thing.

Observability

const state = Finest.state();
state.routingEnabled;          // false whenever routing is mechanically off
state.routingDisabledReason;   // and exactly why
state.journal.gaps;            // every interval we cannot account for
state.policies;                // route → policy id, lifecycle phase, generation
state.quarantinedRoutes;
state.observedOnlyCalls;

What this does not do yet

Stated here rather than discovered later:

• Streaming requests are observed on the incumbent and are not routed. The SDK preserves the provider stream surface and journals terminal outcomes, but it will not release challenger bytes without a crash-durable two-phase stream journal. A process kill between dispatch and terminal completion would otherwise leave an undetectable evidence hole.
• An intercepted method returns a native Promise. Provider SDKs return promise subclasses with extra helpers (APIPromise.withResponse(), .asResponse()); those helpers are not available on a wrapped call. Everything reachable by await is identical. Un-intercepted methods (messages.stream, beta.*, files.*) delegate to your originals. Methods are stably bound to the real provider object so private-field/WeakMap brand checks continue to work through the proxy.
• Routing needs an adapter resolver and a price resolver. Without them — and without a provider credential in the process — the SDK observes. It never guesses a price, and it never routes against a worst case it cannot compute.
• Runtime adapters are revision-pinned. Bootstrap names the provider-bound adapter revision (including compatibility dialects such as DeepSeek), and an SDK with a different revision refuses to execute that tuple.

API

| export | what it is | |---|---| | Finest.init(options) | configure; returns a FinestInstance and sets the default | | Finest.wrap(client) | wrap an OpenAI / Anthropic / Google client | | Finest.state() | the full install state, for the console | | Finest.disable() / .enable() | the runtime kill switch | | Finest.shutdown() | flush, drain, close | | defineRoutes({...}) | type-checked route configuration | | validators | json, requiredFields, oneOf, nonEmpty, matches | | FileSpoolJournalSink, MemoryJournalSink, NullJournalSink | sinks | | PolicyCache, SpendReservation, ConstraintAsserter | the pieces, individually | | FailureStreakQuarantine, LatencyBreaker, AgentBudgetGovernor | the reflexes |

Finest.init options

| option | default | notes | |---|---|---| | apiKey | FINEST_API_KEY | fnst_…; authenticates bootstrap, journal drain and policy fetch | | baseUrl | https://api.finest.so | hosted control plane; also FINEST_API_URL | | offline | false | explicit local-only observer; performs no bootstrap, policy fetch or journal upload | | environment | production | stamped on every attempt | | disabled | false | same effect as FINEST_DISABLE=1 | | routes | {} | usually defineRoutes({...}) | | routesFile | false | load finest.routes.ts (or its compiled output); true searches the working directory, a string names the file | | constraints / publicKeys | bootstrap | manual values are for embedded/offline control planes; hosted bootstrap is authoritative | | adapters / prices / credentials / transport | bootstrap catalog / provider env / direct fetch | advanced overrides; defaults are ready for the one-key path | | journalSink | file spool | pluggable | | waitUntil | — | serverless flush hook | | spoolDir / cacheDir | required in production | also FINEST_SPOOL_DIR, FINEST_CACHE_DIR; development may use $TMPDIR/finest | | stickyHmacKey | authenticated bootstrap, then FINEST_STICKY_HMAC_KEY | advanced stable-key override, minimum 32 bytes; raw sticky IDs are never journalled or sent | | clock | system | injected everywhere; nothing calls Date.now() directly | | logger | console.warn | rate-limited to one line per key per window | | onEvent | — | every degrade decision, as it happens |

Tests

pnpm --filter finest-ai test

The suite includes:

• test/passthrough.test.ts — a throw injected into each of the ten stages in turn; the caller still gets the provider's response, nothing escapes, exactly one log line per window.
• test/disable.test.ts — FINEST_DISABLE=1 returns the identical reference.
• test/sticky.test.ts — assignment is stable, monotonic in the canary percentage, and refuses to invent a unit.
• test/pipeline.test.ts — refusal does not fall back; a validator failure does, and journals both attempts.
• test/stream.test.ts — never retries after the first byte.
• test/journal.test.ts — survives SIGKILL; a full disk produces a gap.
• test/reservation.test.ts — refuses to dispatch when the cap is too small.
• test/routes-file.test.ts — a broken config file is never an exception.
• test/bootstrap.test.ts — one-key bootstrap, human admission, exact tuple, adapter revision, BYOK dispatch, and every fail-closed boundary.

License

Apache-2.0. See LICENSE.