langtell
v0.6.0
Published
Tell me the language — evidence-fusion language detection for short strings, with an auditable confidence trail.
Maintainers
rejifaldReadme
langtell
Tell me the language.
langtell infers the language of short strings — titles, snippets, headlines —
by fusing evidence from many signals into a single weighted verdict with a
confidence score and an auditable trail. It reads the tells: the script and
distinctive letters of the text, the <html lang> / og:locale / meta tags of
the page it came from, the HTTP Content-Language header, and — optionally —
heavier statistical engines like franc or the
on-device Chrome AI language detector.
It is not another trigram detector competing with franc/cld3/tinyld. Those
answer "what language is this body of text?" from the characters alone.
langtell answers "what language is this title, given the page, transport,
and source it arrived in?" — and shows its work.
Status: early. The core detector (candidate-relative script/letter scoring, the BCP-47-aware fuser with the context-vs-script guard, and the opt-in franc and Chrome AI engines) is implemented and tested. The API below reflects the committed design.
Why
- Short strings beat statistical detectors. A two-word title gives franc too
little to chew on.
langtellleans on script ranges, distinctive letters, and out-of-band metadata that a pure text detector never sees. - Auditable, not magic. Every verdict carries the list of signals that
produced it (
evidence[]), each with its kind, language, confidence, and raw value — so you can debug why a title was classified the way it was. - Pay only for what you use. The zero-dependency core (script + HTML + header signals) is fully synchronous. Heavy engines (franc's trigram tables, the browser detector) live behind their own subpaths and only enter your bundle — and only run — when you opt in.
Quick start
import { compile } from "langtell";
import { uk, ru, en } from "langtell/profiles"; // ready-made roster data
// compile() does the per-roster setup once; call the returned fn many times.
const detect = compile({ candidates: [uk, ru, en] });
const result = detect({
text: "Їжак Сонік",
html, // optional: <html lang>, og:locale, meta content-language
responseHeaders, // optional: HTTP Content-Language
});
// → { language: "uk", confidence: 0.9x, evidence: [{ kind: "title-script", ... }, ...] }Add the franc engine — it stays behind its own import door so its trigram tables
never reach a bundle that doesn't use it. franc runs in-process and
synchronously, so detect stays synchronous:
import { compile } from "langtell";
import { uk, ru, en } from "langtell/profiles";
import { createFrancEngine } from "langtell/franc";
const candidates = [uk, ru, en];
const detect = compile({ candidates, engines: [createFrancEngine(candidates)] });
const result = detect({ text, html, responseHeaders });Register the on-device Chrome AI engine and the return type becomes Promise
automatically, because that engine is async:
import { compile } from "langtell";
import { uk, ru, en } from "langtell/profiles";
import { chromeAiEngine } from "langtell/chrome-ai";
const detect = compile({ candidates: [uk, ru, en], engines: [chromeAiEngine] });
const result = await detect({ text }); // Promise<Classification>Need more than "what language + how sure"? The default Classification collapses
the candidate-relative ladder into one confidence float. When you need the raw
structure — which rung decided (distinctive letters → function words → frequent
words → optional trigram backstop) and the integer margin (the winner's lead
over the runner-up) — reach for the opt-in langtell/classify door. It stays
zero-dependency and franc-free; scoring is relative to the roster you pass in.
import { classifyBySnippet } from "langtell/classify";
import { uk, ru } from "langtell/profiles";
classifyBySnippet("Слава Україні", [uk, ru]);
// → { language: "uk", margin: 2, rung: 1, discriminating: true } (a distinctive letter)
classifyBySnippet("Кофе и чай", [uk, ru]);
// → { language: "ru", margin: 1, rung: "2a", … } (a function-word marker)This powers per-rung safety gates ("act only when a weak rung clears a high
margin") and diagnostics — uses a single confidence number can't serve. The
high-level compile/detect/fuse output is unchanged; this is purely additive.
Roster-free Cyrillic fast-path
langtell/classify scores a snippet relative to a roster you pass in.
langtell/cyrillic is the opposite trade: a fixed, zero-config discriminator for
the four Cyrillic languages langtell profiles — Ukrainian, Russian, Belarusian,
Bulgarian — decided purely by distinctive letters, with no roster, no
tokenization, and no franc. Reach for it when you just need "is this Russian? is
this Ukrainian?" on a hot path and don't want to assemble a candidate set.
import { detectCyrillicLanguage, isUkrainian } from "langtell/cyrillic";
detectCyrillicLanguage("Їжак"); // → { language: "uk", … } ї is uniquely Ukrainian
detectCyrillicLanguage("жёлтый"); // → { language: "ru", … } ё / ы are Russian
detectCyrillicLanguage("съм българин"); // → { language: "bg", … } ъ used as a vowel, repeated
detectCyrillicLanguage("подъезд"); // → { language: "ru", … } lone ъ in a short word
isUkrainian("Слава Україні"); // → trueIt returns "unknown" rather than guessing when the signals are insufficient — no
Cyrillic at all, a uk/ru tie, or only an ambiguous э. The CyrillicVerdict also
carries the raw ukScore / ruScore tallies behind the call. Zero-dependency and
side-effect-free; escalate to classifyBySnippet or a franc-backed source when
letter signals aren't enough.
API at a glance
| Export | Role |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| compile(config) | Build a configured detect function (does the precompute once). |
| detect(input) | The compiled detector. Sync or Promise, by config — see below. |
| evidenceFromText(text, candidates?) | Producer: roster-relative script + distinctive-letter signals. Zero-dep, sync. |
| evidenceFromHtml(html) | Producer: <html lang>, meta content-language, og:locale. Zero-dep, sync. |
| evidenceFromHeaders(h) | Producer: HTTP Content-Language. Zero-dep, sync. |
| normalizeBCP47(tag) | Normalize a BCP-47 tag/alias to a canonical code (uk-UA/ua → uk). |
| fuse(evidence, opts?) | Weighted blend + "context never overrides clear script" guard. |
| langtell/profiles | Ready-made LanguageProfile data (uk/ru/be/bg/en). Opt-in (carries word data). |
| langtell/classify | Opt-in structured snippet verdict ({ language, margin, rung }). Zero-dep. |
| langtell/cyrillic | Opt-in roster-free Cyrillic fast-path (detectCyrillicLanguage, isRussian/isUkrainian). Zero-dep, sync. |
| langtell/franc | Opt-in franc engine (pulls trigram tables). Sync. |
| langtell/chrome-ai | Opt-in on-device Chrome AI engine (browser). Async. |
detect returns a plain Classification when every registered source is
synchronous, and Promise<Classification> the moment an async engine is in the
mix — the type reflects the config, so you never guess whether to await. See
DESIGN.md for the full architecture.
Prior art
franc— trigram detection over 400+ languages.langtellcan use it as one engine, but works on short strings where franc has too little signal, and fuses it with page/transport metadata.cld3,tinyld,languagedetect— statistical text-only detectors.langtelldiffers by combining script logic with out-of-band evidence and emitting an auditable trail.