@hawkeyexl/inference

Shared LLM inference layer for the docs-as-tests toolchain: schema-constrained completion across Anthropic, OpenAI-compatible, Claude CLI, and in-process local (llama.cpp) providers, with result caching, cost accounting, and an LLM-as-judge ensemble on top.

Extracted from three projects that had each grown their own copy — docevals, dockg, and agentevals — so a provider fix lands once instead of three times.

📖 Documentation

Install

npm install @hawkeyexl/inference

Requires Node 24+. Three runtime dependencies, plus one optional peer dependency for local models.

ESM only. The exports map has no require condition, so require("@hawkeyexl/inference") fails with ERR_PACKAGE_PATH_NOT_EXPORTED. From CommonJS, use await import("@hawkeyexl/inference").

What it does

Every consumer wants the same narrow thing: send a system prompt, a user prompt, and a JSON Schema; get back JSON that validates against that schema, or a recorded error.

(system, user, schema, temperature) -> JSON

No streaming, no multi-turn, no tool loops. If you need a conversation, this is the wrong package. Widening the provider contract requires an ADR.

Two layers, one entry point:

• Completion — the provider contract, five providers, a content-addressed cache, a price table, and a validate-and-retry wrapper. All that structured extraction needs.
• Judge — the canonical verdict schema, an N-run ensemble, consensus math, and confidence-zone routing. Built on the completion layer; ignore it if you do not need it.

Quick start

No API key required — MockProvider is exported for exactly this.

import { MockProvider, completeValidatedJSON } from "@hawkeyexl/inference";

const run = await completeValidatedJSON({
  provider: new MockProvider([{ json: { summary: "Covers authentication." } }]),
  system: "You summarize documentation pages.",
  user: pageBody,
  schema: {
    type: "object",
    required: ["summary"],
    properties: { summary: { type: "string" } },
    additionalProperties: false,
  },
});

if (run.error) console.error(run.error);
else console.log(run.result.summary, run.usage);

completeValidatedJSON never throws on a model failure and never coerces a bad response. It retries once, then returns a run with error set and result absent.

Point it at a real model by swapping the provider — or omit provider entirely and let the library detect one this machine can use, ending at the free local model:

const provider = await makeProviderAsync({});

Providers

| provider | Structured output via | Credential | Reports usage | |---|---|---|:---:| | anthropic | forced tool call | ANTHROPIC_API_KEY | yes | | openai | strict json_schema, falls back to json_object | OPENAI_API_KEY | yes | | claude-cli | schema in the prompt, --output-format json | local claude auth | no | | llama-cpp | GBNF grammar compiled from the schema | — (runs locally) | yes | | mock | scripted responses | — | synthetic |

Omit provider and the highest-priority one this machine can actually use is detected, ending at llama-cpp — which needs no key, and whose native binding is installed on demand into ~/.hawkeyexl-inference/runtime if it is missing. That install warns once and is refused by INFERENCE_NO_AUTO_INSTALL; it never touches your package.json, lockfile or node_modules.

Usage reporting is the column that decides whether cost accounting works: a provider that reports no tokens makes a budget gate inert. See Choose a provider.

Documentation

| Track | What it covers | |---|---| | Get started | Install, one validated call with no key, choosing a provider | | Judge & consensus | Ensembles, consensus math, confidence zones, caching, budgets | | Structured extraction | One schema-constrained call, honest failures, the subprocess seam | | Run models locally | GGUF weights in-process, model selection, managing weights on disk | | Keep it working | Testing without a network, upgrading without losing a cache | | Reference | Full signatures for every export |

Who the docs serve and why each page exists lives in docs/content-strategy/.

Design decisions

Recorded as ADRs in adrs/:

• 01000 — a library-owned ProviderSpec, not consumer config objects
• 01001 — one entry point; a canonical verdict schema with a per-consumer override seam
• 01002 — which fork won for each merged file, so the losing variants are not reintroduced
• 01003 — in-process local models via node-llama-cpp, why selectors need an async factory, and why the catalog pins exact blob paths
• 01004 — detect an available provider when none is specified, ending at the local model
• 01005 — a CUJ-first documentation set, with samples that CI executes
• 01006 — document failure and orchestration, and gate both against the source
• 01007 — harden two operational failure paths: non-JSON CLI output, and an unsupported Node
• 01008 — auto-install the local runtime into a library-owned prefix, why the shim beats createRequire, and why a model without a provider is now an error

License

MIT