corpus-engine

Structure, retrieval, and grounded answers over any body of media.

Give it a bounded corpus — one person's essays and talks, one research question's sources, one user's notes and streams — and it gives you back three things:

• Structure: sources → snapshots → packets. Raw evidence (chunks, quotes, timeline events) and distilled beliefs (claims, principles, decision criteria) live in the same index.
• Retrieval: hybrid lexical + semantic fusion, intent-aware ranking, query expansion, LLM reranking with guardrails, neighbor stitching, near-duplicate collapse.
• Grounded answers: every claim cited to a source, and an honest refusal when the corpus doesn't contain the answer.

The subject of the corpus is the variable. This is the engine behind researcher.now's personas (a mind of someone else), its run analysts (a mind of a question), and mind.store (a mind of you).

Why this exists

An agent answering from a body of media has one defining failure mode: fabrication. Chunk-RAG over someone's files will confidently invent aggregates, dates, and beliefs the corpus never contained — in our measurements, a stock BM25-RAG baseline fabricated answers to unanswerable probes while this engine's answer path held zero, treating refusal as a correct answer and measuring itself on it.

The second failure mode is subtler: retrieval that only knows chunks. "When did X happen" and "what does X believe" are different questions; the second is answered by distilled material (claims, principles) that raw chunk similarity routinely buries. corpus-engine indexes both kinds, classifies question intent, and guarantees distilled beliefs survive ranking on worldview questions.

Every mechanism here was extracted from a production persona engine where it was validated by measurement — and the constants carried verbatim. The in-repo benchmark exists so that stays true: no retrieval change ships if the scorecard regresses.

Install

npm install corpus-engine

Requirements: Node ≥ 20. Ships as ESM; require("corpus-engine") also works on Node ≥ 22.12 via require(esm). TypeScript users: "module": "nodenext" (or a bundler). Zero runtime dependencies.

Sixty seconds

import { createCorpus } from "corpus-engine";

const corpus = createCorpus({ subject: "Paul" });

// 1. add — drop in any text media
await corpus.add({ text: essay, title: "How to Start a Startup", kind: "essay" });
await corpus.add({ text: transcript, kind: "transcript", trustTier: "secondary" });

// 2. index — chunk + embed
await corpus.index();

// 3. retrieve — hybrid, intent-aware, works with zero config
const hits = await corpus.retrieve("What does Paul believe about startup ideas?");

That runs as-is: no API key, no database, no config. Retrieval is lexical (BM25 + rank fusion) until you plug in a model and embedder — then it gets query expansion, HyDE probes, semantic fusion, and reranking:

const corpus = createCorpus({
  subject: "Paul",
  model: {
    // one function — any provider fits
    complete: async ({ system, prompt, maxTokens, temperature }) => {
      const msg = await anthropic.messages.create({
        model: "claude-sonnet-4-6",
        max_tokens: maxTokens ?? 1024,
        temperature,
        system,
        messages: [{ role: "user", content: prompt }],
      });
      return msg.content[0].type === "text" ? msg.content[0].text : "";
    },
  },
  embedder: {
    embed: async (texts) => voyage.embed(texts), // one function
  },
});

// 4. answer — cited, or an honest refusal
const { answer, refused, citations } = await corpus.answer(
  "When was the company founded?",
);

// 5. distill — topic-stratified claims/principles, written back into the index
const { topics, claims } = await corpus.distill();

Five verbs. That's the whole API.

Benchmarks

Two tiers, both in-repo, both reproducible. All corpora are fictional — invented people, dates, and numbers — so no model can answer from world knowledge, and fixture integrity (gold verbatims, aggregation spanning, unanswerables) is enforced as tests in CI.

Tier 1 — retrieval (deterministic, no API key, runs in CI): the full pipeline vs stock RAG (single-probe BM25 top-k) over the identical index — only the retrieval strategy differs. Four corpora, 52 scored probes, including a 14-source/23k-word deep archive where every document spans multiple chunks. Scored at several evidence budgets because ranking quality shows most when the budget is tight (and tight budgets are the production reality — evidence windows are token-priced):

| k | corpus-engine | + semantic | stock RAG | | --- | --- | --- | --- | | hit@2 | 63% | 67% | 58% | | hit@4 | 83% | 81% | 85% | | hit@12 | 98% | 100% | 96% | | MRR | 0.74 | 0.75 | 0.71 |

+ semantic is the same pipeline with the vector lane lit by a keyless local embedder (BENCH_EMBEDDER=local npm run bench after a one-time npm i --no-save @huggingface/transformers; MiniLM, ~30MB weights) — the default two-system run stays downloadless for CI. Stock RAG's hit@4 edge is one probe; everywhere else the pipeline pays for itself, most at the tight head. Full per-fixture, per-type breakdown: bench/RESULTS.md.

Tier 2 — answers (needs a model): the full answer path — retrieval with expansion and reranking, grounded synthesis, refusal — for both systems with the identical synthesis prompt, including 12 out-of-corpus probes where answering at all is a fabrication. Latest run (claude-sonnet via CLI, k=12, 64 probes):

| system | correct (answerable) | fabricated (unanswerable) | honest refusals | missed refusals | | --- | --- | --- | --- | --- | | corpus-engine | 56% (29/52) | 0% (0/12) | 12/12 | 0/52 | | stock RAG | 54% (28/52) | 0% (0/12) | 12/12 | 0/52 |

Zero fabrications from either system — the refusal-sentinel prompt does that work, and it ships as the default answer path. Correctness is scored by strict verbatim substring, which undercounts both systems equally; per-probe failures are listed in bench/RESULTS-answers.md. Reproduce: ANTHROPIC_API_KEY=… npm run bench:answers (or BENCH_CLAUDE_CLI=1 with the Claude Code CLI, no key needed).

This tier has already paid for itself once: the reranker's first version shipped with an evidence-starvation bug that scored 38% with 9 false refusals here — caught by this benchmark, fixed, and pinned by tests before any release carried it.

API

createCorpus(config?)

const corpus = createCorpus({
  subject?: string;       // who/what the corpus is about; used in prompts. Default "the corpus".
  model?: LanguageModel;  // enables expansion, reranking, answer(), distill(). Optional.
  embedder?: Embedder;    // enables the semantic retrieval lane. Optional.
  store?: CorpusStore;    // durable storage. Default: in-memory reference store.
});

LanguageModel and Embedder are each a single function, so any provider fits without an adapter package:

interface LanguageModel {
  complete(args: {
    system?: string;
    prompt: string;
    maxTokens?: number;
    temperature?: number;
    json?: boolean; // hint: the response must be a single JSON value
  }): Promise<string>;
}

interface Embedder {
  embed(texts: string[]): Promise<number[][]>; // one vector per input, same order
}

corpus.add(input) → Source

Registers a source and snapshots its content. The source id is a stable citation handle forever; content lives in snapshots keyed by SHA-256 content hash, so identity is decoupled from freshness — re-adding changed content versions the snapshot, never the source.

await corpus.add({
  text: string;            // required — extracted text. Fetching URLs/media is the caller's job.
  title?: string;
  url?: string;
  kind?: string;           // "essay", "transcript", "post", … free-form
  trustTier?: "primary" | "secondary" | "tertiary";
  //           their own words | interviews of them | written about them
  authority?: string;      // "primary"/"official"/"self"/… earns a 1.1× retrieval weight
  metadata?: Record<string, unknown>;
});

corpus.index(options?) → { packetsAdded }

Chunks every un-indexed snapshot into chunk packets (and embeds them when an embedder is configured). Paragraph-aware windows: 650 words max, 100 words of overlap carried across boundaries, fragments under 90 words merged into their predecessor.

corpus.retrieve(question, options?) → RetrieveResult

The full pipeline: expand → probe → fuse → weight → rerank → floor → stitch → dedupe.

| option | default | what it does | |---|---|---| | limit | 12 | packets returned | | probeLimit | 24 | candidate depth per probe (floor: raised to limit+4 if set lower) | | rrfK | 60 | reciprocal-rank-fusion constant (the literature standard) | | neighborRadius | 1 | adjacent chunks stitched around each hit | | maxLexicalProbes | 6 | expansion phrases used as keyword probes | | beliefFloor | 3 | distilled-belief packets guaranteed on worldview questions | | noExpand | false | skip LLM query expansion | | noRerank | false | skip the LLM rerank stage | | rerankFloor | 0.08 | rerank relevance floor, applied adaptively as min(floor, top×0.35) | | rerankFusionBlend | 0.3 | weight of the fusion prior blended into rerank ordering |

Returns { packets, intent, probes, expanded, reranked } where each packet carries its score and stitched text. Works with no model and no embedder (lexical-only).

corpus.answer(question, options?) → AnswerResult

Retrieves, then synthesizes strictly from the evidence, citing every claim [1][2]. Returns:

{
  answer: string | null;   // null = the corpus doesn't contain the answer
  refused: boolean;        // honest refusal is a result, not an error
  citations: Citation[];   // packetId + sourceId + title/url
  retrieval: RetrieveResult;
}

Requires a model; throws a self-explaining error without one.

corpus.distill(options?) → DistillResult

Topic-stratified distillation: discovers the corpus's themes (72-packet strided sample), assigns evidence to topics deterministically by keyword match, gives each topic its own evidence budget (12–32 packets), and generates cited worldview + decision-criteria claims per topic. Claims are written back into the index as claim / decision_criterion packets — so worldview questions retrieve them, and the belief floor guarantees they survive ranking.

Why stratify: without per-topic budgets, distillation collapses to the corpus's most prolific theme. This mechanism took a production persona's held-out off-theme accuracy from 58% to 88% (measured upstream in the system this library was extracted from; not reproducible from this repo).

Packet types

| kind | examples | wins on | |---|---|---| | raw evidence | chunk, passage, quote, timeline_event | factual questions ("when", "who", "how many") | | distilled beliefs | claim, principle, decision_criterion, thesis, argument, prescription, summary | worldview questions ("believes", "thinks", "position on") |

A regex intent classifier routes each question; factual wins ties. On worldview questions the belief floor promotes distilled packets past incidentally-similar chunks.

corpus-engine/evals — the scorecard

import {
  scoreFaithfulness,          // do distilled claims match their citations?
  scoreGroundedness,          // fullyGroundedRate, meanSupportRate, fabricationRate
  scoreAgreement,             // vs held-out sources (interpolation) / domains (extrapolation)
  scoreConfidenceCalibration, // are high-confidence answers right more often?
  scoreOutOfCorpusCalibration,// unanswerable probes: refused | inferred_labeled | fabricated
  abVerdict,                  // ship/no-ship: keep only if extrapolation lifts (ε=0.02)
                              // while groundedness holds and fabrication doesn't rise
} from "corpus-engine/evals";

These are pure scorers over judge verdicts — running the LLM judges is the harness's job; the library defines the metrics so numbers are comparable across releases and consumers. fabricationRate (an answer whose core claim has no support) is the headline safety number; the in-repo benchmark measures it directly by asking questions the corpus provably cannot answer.

Bring your own store

createCorpus() defaults to an in-memory store (BM25 + exact cosine) — right for tests, scripts, and small corpora. For production, implement CorpusStore — twelve methods, six over sources/snapshots and six over packets — on pgvector, SQLite, or anything else:

import type { CorpusStore } from "corpus-engine";

Retrieval degrades gracefully wherever a capability is missing: no embedder → lexical-only; no model → no expansion or rerank, but retrieve() still works; searchPacketsByVector returning [] → the semantic lane silently sits out.

Why it retrieves well

Every mechanism is ported from a production system where it was validated by measurement, not intuition:

• Packets, not just chunks. Raw evidence and distilled beliefs share one index; a cheap intent classifier routes each question to the kinds that answer it.
• The belief floor. At least 3 distilled-belief packets survive ranking on worldview questions — "what does she always want" never loses to an incidentally-similar chunk.
• Topic-stratified distillation. Per-topic evidence budgets prevent collapse onto the corpus's loudest theme (58% → 88% held-out, measured upstream).
• Hybrid fusion. Lexical and semantic probe lists merge by reciprocal rank fusion (k=60), then intent and authority weighting.
• LLM reranking, blended. With a model configured, one batched call scores the candidate pool; the final order blends rerank (70%) with the fusion prior (30%) so a single noisy verdict can't kill a strong hybrid hit. Two guardrails learned by measurement: a candidate the model returned no verdict for keeps its fusion prior (no verdict is not a verdict), and the relevance floor never trims below the caller's evidence window. Degrades to fused ordering on any failure.
• Neighbor stitching. A hit chunk pulls its prev/next siblings, spliced without duplicated overlap, so lists split across boundaries don't truncate.
• Near-dup collapse. 8-token shingle containment (0.8) kills re-cuts of the same passage before they crowd out diverse evidence.
• Honest refusal. { answer: null, refused: true } is a first-class result, and the eval suite prices it: refusing an unanswerable probe scores; fabricating fails.

What this deliberately is not

A feature enters this library only when one of its production consumers (personas, run analysts, mind.store) already needed it. That excludes, permanently or until then:

• vector databases, embedding models, chunk-format zoos → bring your own, the interfaces are one function each
• chat-session memory, agent frameworks, orchestration
• hosted anything

Development

npm install
npm test               # vitest over the pure organs + five-verb integration + fixture integrity
npm run build          # tsc → dist/
npm run bench          # deterministic retrieval benchmark (no key, runs in CI)
npm run bench:answers  # answer-tier benchmark (ANTHROPIC_API_KEY, or BENCH_CLAUDE_CLI=1)

Provenance

Extracted from the production persona engine behind researcher.now (retrieval, evidence budgeting, topic synthesis, fidelity metrics) and its research-run indexer (chunking). Constants are carried verbatim from the production-validated versions; see inline comments for the reasoning behind each one. MIT.