contexel

Deterministic, dependency-free context shaping for code-writing agents — the TypeScript build of contexel.

Tool calls return verbose, duplicated, sometimes hostile data. contexel sits at the tool → context boundary and shapes it: select the fields that matter, drop duplicates, gate by provenance, quarantine injection patterns, rescore by relevance, truncate, rank, and trim to a token budget — every stage a pure function over Record<string, unknown>[], every run reproducible, every drop attributable in an audit record.

This package is a parity port, not a re-imagining. The Python implementation is canon; both languages run their test suites against the same golden vectors (parity/vectors.json, generated by Python) so that the same records, policy, and budget produce the same shaped output in either language. Versions track the Python release. Design and boundaries: ADR-001.

Install

npm install contexel     # zero runtime dependencies, Node >= 18, ESM

Sixty seconds

import { select, dedupe, rescore, truncateField, rank, trimToBudget, trace } from "contexel";

const raw = await searchTool(userQuery);       // a batch of verbose results

const [context, t] = trace({ idField: "url" }, () => {
  let r = select(raw, { fields: ["title", "url", "snippet", "published"] });
  r = dedupe(r, { key: "url" });
  r = rescore(r, { query: userQuery, fields: ["title", "snippet"] });
  r = truncateField(r, { field: "snippet", maxTokens: 120 });
  r = rank(r, { by: "score", desc: true });
  return trimToBudget(r, { maxTokens: 1500 });
});

console.log(t.report());   // per-stage: records in -> out, tokens before -> after
console.log(t.audit());    // policy fingerprints + which stage dropped which IDs

Or pin the policy at the tool boundary, so the model never sees raw output:

import { shaped, stage, select, dedupe, trimToBudget } from "contexel";

const searchCode = shaped([
  stage(select, { fields: ["path", "line", "snippet"] }),
  stage(dedupe, { key: ["path", "line"] }),
  stage(trimToBudget, { maxTokens: 2000 }),
])(async (query: string) => repo.search(query));  // async tools are awaited, then shaped

The eight stages

| Stage | What it does | |---|---| | select | keep only the listed fields | | dedupe | drop duplicates by key or whole-record fingerprint (type-aware: 1, "1", true stay distinct) | | allowlist | fail-closed provenance gate — keep records whose field is in the allowed set (a missing field reads as null, kept only if null is explicitly allowed) | | quarantine | drop or flag records matching injection patterns ("ignore previous instructions", …); custom patterns extend the built-ins — replacePatterns: true is the explicit opt-out | | rescore | batch BM25 relevance with word-boundary matching and in-order proximity bonus | | rank | stable sort by a field; records missing the field sort last | | truncateField | cut one field to a token budget, … suffix | | trimToBudget | keep records until the total token budget is spent; minRecords guarantees the best records survive a too-small budget instead of returning [] |

Plus merge for combining multiple sources under one schema, pipeline/stage for composition with a stable policy fingerprint, trace/audit for the governance record, and tokens.scoped for concurrency-safe (AsyncLocalStorage) tokenizer and serializer overrides per tenant.

Parity, precisely

• Shaped output is structurally identical to Python's for JSON-safe records — enforced by golden vectors covering every stage, the composed contract, the audit record, canonical serialization, and token counts.
• Token counting is per Unicode code point (matches Python len), not UTF-16 units — "café" costs the same in both languages.
• Documented boundaries (see ADR-001): JS collapses 1.0 to "1" in serialization; Unicode word classes differ at the margins; pipeline fingerprints are language-local in v1.

Links

• Docs: https://maheshvaikri-code.github.io/contexel/
• Benchmarks (methodology and limits stated): https://github.com/maheshvaikri-code/contexel/blob/main/benchmarks/COMPARISON.md
• Python package: https://pypi.org/project/contexel/
• License: MIT