langchunk

Turn a Universal Dependencies parse into sentences → clauses → phrases → words, with every unit traceable back to the exact characters it came from.

npm install langchunk

Clause extraction from a dependency parse is the piece the NLP ecosystem doesn't ship: parsers give you trees, and nothing turns a tree into the grammar a learner or teacher actually asks about — which clause is independent, what is its subject, where does the relative clause attach. langchunk is that layer: deterministic TypeScript over any UD tree, graded at 100% against hand-annotated gold treebanks — not a threshold, an exact bar — and measured end-to-end per language with real parsers.

import { parseConlluSentences, buildGoldDocument, buildDocument, packOrFallbackFor } from "langchunk";

// Bring any CoNLL-U — from Stanza, spaCy, UDPipe, a treebank, anywhere.
const sentences = parseConlluSentences(conlluText);
const gold = buildGoldDocument(sentences);
const pack = packOrFallbackFor("en");

const doc = buildDocument({
  text: gold.text,
  sentences: gold.sentences,
  language: { code: "en", tier: pack.tier, resolution: "declared" },
  analyzer: { id: "my-parser", version: "1" },
  options: pack.grammar,
});

// doc.sentences / doc.clauses / doc.phrases / doc.words — each with a
// span into the ORIGINAL text: text.slice(span.start, span.end) === unit.text.

What you get

• The taxonomy. Sentences, clauses (independent / coordinated / dependent with role), phrases (NP/VP/PP/AdjP/AdvP with heads), words (including multi-word units) — every unit carrying a span into the original string and a confidence.
• Five language packs built in — English, Russian, Persian, French, German — each with measured accuracy, plus a plugin mechanism where a new language is a JSON file, not a code change. A language with no pack still parses, honestly labelled broad-fallback.
• Segmentation (segmentSentences) with per-language, corpus-measured abbreviation handling.
• Exports (langchunk/export): CSV, JSONL, CoNLL-U, Anki decks.
• Analyzers: the gold CoNLL-U analyzer (browser-safe, shown above), and Node-only bridges — langchunk/analyzers/stanza (spawns Python Stanza, the highest-accuracy path) and langchunk/analyzers/onnx (pure Node, needs the optional onnxruntime-node peer).

Subpaths mirror the internals: langchunk/schema, /grammar, /segment, /lang, /conllu, /pipeline, /eval, /export, /validators, /lang-node, /analyzers/{gold,agreement,stanza,onnx}. The root export is the curated common path. Everything except lang-node and the stanza/onnx analyzers is browser-safe, and a boundary checker enforces that claim in CI.

The design in one table

| | | |---|---| | Tier 1 | A dependency parser produces a Universal Dependencies tree. Ambiguous, learned, replaceable — bring your own. | | Tier 2 | This library. Deterministic mapping from tree to taxonomy. Language-general, zero runtime dependencies beyond zod. |

Structural ambiguity belongs in a model; the taxonomy is a designed scheme, so mapping onto it is deterministic — which is why Tier 2 can be graded exactly. Clause boundaries come from advcl/acl/ccomp/conj/mark relations, never from keyword scanning.

Measured accuracy (strict F1, 300-sentence Gate 2 runs)

End-to-end with a real parser, against the same Tier 2 over gold trees — so every gap shown is parser error, not taxonomy error:

| language | parser | clause | phrase | word | |---|---|---|---|---| | Persian | Stanza perdt | 92.4 | 93.7 | 96.4 | | English | Stanza electra | 91.0 | 94.6 | 97.3 | | French | Stanza combined | 81.0 | 80.0 | 94.0 | | German | Stanza combined | 79.2 | 83.5 | 93.6 | | Russian | Stanza mixed ruBERT | 75.6 | 87.2 | 91.7 |

Segmentation is measured separately (Gate 3): of the sentence boundaries a writer actually marked, Persian finds 100%, German 98%, English and French 96.8%. Every number above has a committed report and a test that fails if a change regresses it.

The app

LangChunk the application — a local-first web app over this engine, with on-demand language installs and a local analysis service — lives at khizardevelops/langchunk-app.

Developing this repo

pnpm install
pnpm run ud:fetch en_ewt ru_taiga    # gold treebanks, gitignored
pnpm run verify                      # typecheck + boundaries + the full suite

The suite needs no model, no network, no GPU — a corpora-less clone skips the corpus suites and still passes. Gate 2 needs the Python reference parser:

python3 -m venv --system-site-packages .venv-stanza
.venv-stanza/bin/pip install stanza
pnpm run gate2 --lang en --limit 300
pnpm run gate3 --lang en               # the segmenter; needs no model

pnpm run build:npm builds the publishable package into npm/.

Every non-obvious choice in this repository has a written decision behind it — .agents/decisions.md is the log, and it is meant to be read. docs/ProjectInfo.md is the product contract; docs/UpdatedPlan.md the plan.

License

AGPL-3.0. Use it, modify it, ship it — but keep the source open. The evaluation treebanks and models this repo points at carry their own licenses and are downloaded by you, never distributed here; docs/UpdatedPlan.md Appendix B records them.