indickit

Small, exact utilities for text in the languages of India, in Go and TypeScript. They fix the places where ordinary string code goes wrong on Indian text: a search that misses a word because it is spelled, encoded or inflected another way, a letter count that cuts a letter in two, a name that must be written in Latin letters, and Latin typing that must be written in an Indian script.

Try each utility in your browser, in all 22 languages: https://micahchoo.github.io/indickit/docs/demo/

stem      किताब  किताबें  किताबों              →  किताब
phonetic  राम  ರಾಮ  രാമ  ராம  رام  Ram       →  rn
segment   ಲಕ್ಷ್ಮಿ                              →  ಲ  ಕ್ಷ್ಮಿ
romanize  लक्ष्मी  ലക്ഷ്മി  லக்ஷ்மி              →  lakshmi
deromanize  namaste                         →  नमस्ते
wellformed  िहन्दी  हिन्दी                     →  broken  whole

Each utility gives the same output in Go and in TypeScript, and each was measured on data that its rules were never built on.

Which one you need

| You want to | Use | |---|---| | store text so that one look has one encoding | normalize | | let a search for किताब also find किताबों | normalize, then stem | | let a search for हिन्दी also find हिंदी, and keep words whole | fold | | find a person's name typed in another script | phonetic | | find a name in a long list, as people write it (titles, initials), or in running text | phonetic-search | | count letters, cut at a length, or move a cursor | segment | | write a name or a text in Latin letters (लक्ष्मी → lakshmi) | romanize | | let a search for "modi" find मोदी | romanize the index; see the recipe | | write Latin typing in an Indian script (namaste → नमस्ते) | deromanize | | reject or flag text that no font can draw whole (a PDF text layer) | wellformed |

Languages

The 22 languages of the Eighth Schedule. normalize, fold, segment and phonetic read every script below, so they serve every language; the others hold tables for some languages only.

| Language | Code | Script | stem | phonetic-search | romanize | deromanize | |---|---|---|---|---|---|---| | Assamese | as | Bengali-Assamese | ✓ | ✓ | words, names | words, names | | Bengali | bn | Bengali-Assamese | ✓ | ✓ | words, names | words, names | | Bodo | brx | Devanagari | ✓ | ✓ | words | words | | Dogri | doi | Devanagari | ✓ | ✓ | words | words | | Gujarati | gu | Gujarati | ✓ | ✓ | words, names | words, names | | Hindi | hi | Devanagari | ✓ | ✓ | words, names | words, names | | Kannada | kn | Kannada | ✓ | ✓ | words, names | words, names | | Kashmiri | ks | Perso-Arabic | ✓ | ✓ | words, names | words, names | | Konkani | gom | Devanagari | ✓ | ✓ | words, names | words, names | | Maithili | mai | Devanagari | ✓ | ✓ | words, names | words, names | | Malayalam | ml | Malayalam | ✓ | ✓ | words, names | words, names | | Manipuri | mni | Meetei Mayek, Bengali-Assamese | ✓ | ✓ | words, names | words, names | | Marathi | mr | Devanagari | ✓ | ✓ | words, names | words, names | | Nepali | ne | Devanagari | ✓ | ✓ | words, names | words, names | | Odia | or | Odia | ✓ | ✓ | words, names | words, names | | Punjabi | pa | Gurmukhi | ✓ | ✓ | words, names | words, names | | Sanskrit | sa | Devanagari | ✓ | ✓ | words, names | words, names | | Santali | sat | Ol Chiki | ✓ | keys only | names | names | | Sindhi | sd | Perso-Arabic | ✓ | ✓ | words, names | words, names | | Tamil | ta | Tamil | ✓ | ✓ | words, names | words, names | | Telugu | te | Telugu | ✓ | ✓ | words, names | words, names | | Urdu | ur | Perso-Arabic | ✓ | ✓ | words, names | words, names |

"words" and "names" are the two modes of romanize and deromanize; romanize has a third, "places", in all 22 languages (its own tables in nine, the words tables in the others). In deromanize, the names mode of Bodo and Dogri uses their words tables. Santali has no words tables: no dataset of Santali words in Latin letters was found. phonetic-search in Santali ranks by the keys alone. Measured quality differs by language; see docs/ for each utility.

Install

Go 1.22 or later:

go get github.com/micahchoo/indickit

JavaScript or TypeScript, on Node 20 or later, Bun or Deno:

npm install indickit
bun add indickit
deno add npm:indickit

The package is ES modules only; require() works from Node 20.19. In TypeScript, set moduleResolution to nodenext or bundler; the old node10 cannot find the utilities.

In a browser, with no build step:

import { normalize } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/normalize.js";
import { stem } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/stem.js";
import { match } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/phonetic.js";
import { segment } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/segment.js";
import { load } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/romanize.js";
import { loadSearch } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/phonetic-search.js";
import { load as loadDeromanize } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/deromanize.js";
import { isWellFormed } from "https://cdn.jsdelivr.net/gh/micahchoo/[email protected]/dist/wellformed.js";

Each utility is its own file, so a page loads only the rules it uses. romanize also loads one language's tables when you ask for them (0.2–1.8 MB), deromanize one language's tables and word list (0.3–3.1 MB), and phonetic-search one language's table (2–6 KB). The npm package carries no tables: load() reads them beside the module (a jsDelivr or GitHub-tag install), else from jsDelivr at the same version. Offline, pass your own fetch. In Go, go get downloads the tables of every language once (romanize 14 MB, deromanize 52 MB); a program carries only the languages it imports.

Recipes

Store text

Two strings can look the same and differ in their bytes: an invisible joiner, an old Malayalam chillu, a ज़ typed as one code point or two. normalize gives them one encoding, and it never changes what a reader sees. Store text after normalize.

import { normalize } from "indickit/normalize";

normalize("ಸಿಕಾರ್\u200c", "kn"); // "ಸಿಕಾರ್": a ZWNJ at the end draws nothing
normalize("അവന്\u200d", "ml"); // "അവൻ": the old chillu becomes the atomic one
normalize("र्\u200dय", "mr"); // "र्\u200dय": eyelash ra; this ZWJ is visible, so it stays

Reject text no font can draw

A PDF text layer or an OCR run gives words that no font can draw whole: a vowel sign typed before its letter, a conjunct with a member dropped. A renderer shows a dotted circle (◌) there. normalize cannot repair such a word, because it never moves a sign. wellformed finds it before you store it: 10.8% of the words in PDF text layers are broken, 0.012% in clean text.

import { isWellFormed, brokenAt } from "indickit/wellformed";

isWellFormed("\u093fहन्दी"); // false: the vowel sign ि has no letter before it
isWellFormed("हिन्दी"); // true
brokenAt("क्ि"); // 2: a vowel sign after a virama

Search words in their other forms

Give every word of a document a key, and give the query the same key. searchKey normalizes, stems, then folds accepted spellings, in the order that was measured best:

import { searchKey } from "indickit/search-key";

searchKey("किताबों", "hi"); // "किताब"
searchKey("हिन्दी", "hi"); // "हिंद"
searchKey("हिंदी", "hi"); // "हिंद"
searchKey("ஆண்டில்", "ta"); // "ஆண்ட"
searchKey("ஆண்டு", "ta"); // "ஆண்ட"
searchKey("ಸರ್ಕಾರಕ್ಕೆ", "kn"); // "ಸರ್ಕಾರ"
searchKey("حکومتوں", "ur"); // "حکومت"

In a search engine, index the key beside a trigram field (each word cut into its three-letter pieces), and add the two scores: over Wikipedia in Hindi, Bengali and Telugu, the two fields found 3.0 points more of the right passages than trigrams alone, and ranked them better. A setup for OpenSearch and Elasticsearch is in docs/stem.md. In Go: searchkey.Key(word, lang).

In 22 languages, a search with stems finds more of the right sentences than an exact search: 6–15 points in the 14 measured on FLORES+, 3.7–11.4 in the 8 measured on BPCC. In a search engine over Wikipedia passages, it finds 2–11.5 points more in 10 of 13 test sets, and nothing more in Hindi or Punjabi. It also finds some wrong ones: a stem merges forms by their endings, not by their meaning. Do not add fuzzy matching to stems: together they find less than stems alone.

If you call the steps yourself, stem first, then fold: fold before stem loses 3–10 points in Assamese, and fold after stem finds 2.4 points more in Hindi. fold alone merges accepted spellings of one word (हिन्दी and हिंदी) and keeps the word whole, for a search that must not merge forms:

import { fold } from "indickit/normalize";

fold("हिन्दी", "hi"); // "हिंदी"
fold("गाँव", "hi"); // "गांव"

Urdu and Sindhi often write two words as one (ہو گیا and ہوگیا). fold does not join them: joining the stored text glues 39% of Urdu words to the word before. Instead, search a phrase twice, as typed and with its spaces removed. On new Wikipedia text that finds 3–4% more of a phrase's hits in Urdu and 6–14% in Sindhi, with 0.6% and 1.3% wrong hits.

Find a name in any script

One person's name is typed in Devanagari on one form, in Malayalam on a second, and in English on a third. phonetic gives each spelling a key; equal keys mean "this sounds like the same name".

import { match } from "indickit/phonetic";

match("Mohanlal", "മോഹൻലാൽ"); // true
match("Mohan", "மோகன்"); // true
match("राम", "काम"); // false: Rām is not kām

To search a list of names, store every key of each name, and look up every key of the query. A name can have more than one key: Tamil writes both k and h with க, and a Tamil name ending in ன் is also keyed without it, so மோகன், Mohan, has four.

import { nameKeys, RULES_VERSION } from "indickit/phonetic";

for (const key of nameKeys("सचिन तेंदुलकर")) db.insert({ key, personId, rules: RULES_VERSION });
// later
const hits = nameKeys(query).flatMap((key) => db.find({ key }));

Equal keys mean "sounds alike", not "the same person". Before you merge two records, weigh the evidence: a shared rare name says more than a shared common one.

Search a list of names

phonetic-search uses the keys to find candidates, and a scorer learned from data to rank them from 0 to 100. It finds names as people write them, with titles, initials and joined words, and names in running text.

import { loadSearch, THRESHOLDS } from "indickit/phonetic-search";

const s = await loadSearch("hi"); // fetches the Hindi table (3 KB)
const ix = s.index(["नरेश मोदी", "नरेंद्र मोदी", "मनमोहन सिंह"]);
ix.search("Narendra Modi", THRESHOLDS.names); // [{ name: 1, score: 100 }]
ix.search("Manmohan Singh", THRESHOLDS.names); // [{ name: 2, score: 96 }]

Use the threshold for your task: THRESHOLDS.names (76) for a list of people, THRESHOLDS.villages (73) for place names, THRESHOLDS.text_10 (81) or text_20 (71) with s.index(words, "text") for a name among the words of a text. A score is relative to the other candidates of its search.

Count and cut letters

A cursor, a backspace, a letter count and a cut at a length limit must treat ಕ್ಷ್ಮಿ as one letter. Intl.Segmenter splits it in Kannada and Gurmukhi; segment keeps it whole. Each letter is an akshara, the orthographic syllable of the script.

import { segment, count } from "indickit/segment";

segment("ಲಕ್ಷ್ಮಿ"); // ["ಲ","ಕ್ಷ್ಮಿ"]
segment("ਪ੍ਰੀਤ"); // ["ਪ੍ਰੀ","ਤ"]
segment("ਕ੍ਕ"); // ["ਕ੍","ਕ"]: Gurmukhi shows this virama, so these are two letters
count("ಕನ್ನಡ"); // 3

Write it in Latin letters

romanize gives a ranked list of spellings, most likely first. Use the "words" mode for text, the "names" mode for a field that holds a person's name, and the "places" mode for a place name:

import { load } from "indickit/romanize";

const hi = await load("hi");          // fetches the Hindi tables
hi.word("लक्ष्मी", 2);                 // ["lakshmi","laxmi"]
hi.text("भारत के प्रधानमंत्री");          // "bharat ke pradhaanmantri"
const ur = await load("ur", "names");
ur.word("پرویز", 1);                   // ["parvez"]
const ta = await load("ta", "places");
ta.word("காஞ்சிபுரம்", 1);              // ["kanchipuram"]

In Go, import the languages you need; a program carries only their tables:

import (
	"github.com/micahchoo/indickit/romanize"
	_ "github.com/micahchoo/indickit/romanize/lang/hi"
)

romanize.Word("लक्ष्मी", "hi", romanize.Words, 2) // [lakshmi laxmi]

Find native text with a Latin query

A user types "modi" in a search box; the store holds मोदी. When you build the index, romanize each stored word in the "words" mode and store its four most likely spellings in a Latin field beside it:

import { load } from "indickit/romanize";

const hi = await load("hi");          // the "words" mode
hi.word("मोदी", 4);                    // ["modi","modee","moudi","mody"]
hi.word("दिल्ली", 4);                  // ["dilli","dili","dilly","dilley"]
hi.word("सरकार", 4);                   // ["sarkar","sarkaar","sarakar","sarakaar"]

At query time, lower-case the query and fuzzy-match it against that field: a hit is a word with a stored spelling one or two edits from the query (Elasticsearch: fuzziness: AUTO). The browser needs no model, and the index needs no download beyond the tables it was romanized with.

On Wikipedia sentences typed in Latin letters by native speakers (Dakshina, 11 languages, held out), the first hit is the right word 81.4% of the time, and one of the first four is 91.3%. IndicXlit, a 119 MB neural model that writes the query in the native script, gives 81.8% and 87.6%. The index leads in Bengali, Hindi, Kannada, Malayalam, Marathi, Tamil and Telugu, is level in Gujarati and Punjabi, and trails in Sindhi and Urdu. Its cost is a long list: 24 hits a search, and more than half of the first ten are other words. See docs/romanize.md.

Two other ways were measured. Do not put the phonetic key on both sides: it finds the right word first 38.0% of the time and floods the list; it is for names, not words. Do not write the query in the native script (deromanize, the next recipe) for a ranked list: that is the right way for a strict filter, where every hit must be the word itself, but for a ranked list the romanized index is as good and needs no second download.

Write Latin typing in an Indian script

deromanize is the reverse: it gives a ranked list of native spellings for a Latin-typed word, most likely first. It writes every word it is given in the Indian script, English words too. For Hindi typed with English words in it (Hinglish), load the English list and use textMixed: a word on the list stays as typed.

import { load } from "indickit/deromanize";

const hi = await load("hi");          // fetches the Hindi tables and word list
hi.word("namaste", 1);                // ["नमस्ते"]
hi.text("bharat ke pradhanmantri");   // "भारत के प्रधानमंत्री"
const ur = await load("ur", "names");
ur.word("ahmad", 1);                  // ["احمد"]
const mixed = await load("hi", "words", { english: true });  // also fetches the English list (22 KB)
mixed.textMixed("mera office band hai");  // "मेरा office बंद है"

In Go:

import (
	"github.com/micahchoo/indickit/deromanize"
	_ "github.com/micahchoo/indickit/deromanize/lang/hi"
)

deromanize.Word("namaste", "hi", deromanize.Words, 1) // [नमस्ते]

Match two Latin spellings of one word

People type one word in many ways: "zindagi" and "jindagi", "bahut" and "bohot". To match them, write each in the Indian script and fold it; the result is a key for search or for removing duplicates:

import { load } from "indickit/deromanize";
import { fold } from "indickit/normalize";

const hi = await load("hi");
const key = (w: string) => fold(hi.word(w, 1)[0] ?? w, "hi");
key("zindagi") === key("jindagi"); // true: ज़िंदगी and जिंदगी fold to one
key("bahut") === key("bohot");     // true: both are बहुत
key("kahaan") === key("kahan");    // false: कहाँ and कहा

On words deromanize never trained on (Dakshina's test lexicons, read once, 11 languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Sindhi, Tamil, Telugu and Urdu), this key matches 40–69% of the pairs of spellings of one word and gives 0–13% of spellings the key of another word. Fuzzy matching (one edit) matches 33–57% and gives 10–39% a wrong match. Keys from all four spellings match 54–84%, at 4–33% wrong. Malayalam is the weakest language: level with fuzzy matching, at a tenth of its wrong matches. The phonetic key is not for this: it drops vowels, which carry a Latin-typed word.

The same in Go

import (
	"github.com/micahchoo/indickit/normalize"
	"github.com/micahchoo/indickit/phonetic"
	"github.com/micahchoo/indickit/segment"
	"github.com/micahchoo/indickit/stem"
	"github.com/micahchoo/indickit/wellformed"
)

normalize.Text("അവന്\u200d", "ml")               // "അവൻ"
wellformed.Check("\u093fहन्दी")                   // false
stem.Stem(normalize.Text("किताबों", "hi"), "hi") // किताब
phonetic.Match("Mohanlal", "മോഹൻലാൽ")            // true
phonetic.NewIndex(names, "hi", phonetic.ProfileNames).Search("Narendra Modi", phonetic.Thresholds["names"])
segment.Segment("ಲಕ್ಷ್ಮಿ")                       // [ಲ ಕ್ಷ್ಮಿ]

Reference

| Go | TypeScript | Returns | |---|---|---| | normalize.Text(s, lang) | normalize(text, lang?) | one encoding; the look never changes | | normalize.Fold(s, lang) | fold(text, lang?) | normalize, then one spelling of each word | | stem.Stem(word, lang) | stem(word, lang) | the search key of a word | | stem.Languages() | LANGUAGES | the languages with a stem table | | phonetic.Keys(word) | keys(word) | the keys of one word | | phonetic.NameKeys(name) | nameKeys(name) | the keys of a whole name; index these | | phonetic.Match(a, b) | match(a, b) | titles dropped, the same word count, and each pair of words shares a key | | phonetic.Words(name) | words(name) | the words, split as the others split them | | phonetic.JoinedKeys(name) | joinedKeys(name) | the keys of the name with its words joined (6+ classes) | | phonetic.NewIndex(names, lang, profile) | (await loadSearch(lang)).index(names, profile?) | names keyed once for searching | | ix.Search(query, threshold) | ix.search(query, threshold?) | the names scoring at least threshold, best first | | phonetic.Score(query, names, lang, profile) | (await loadSearch(lang)).score(query, names, profile?) | one score (0..100) per candidate | | segment.Segment(s) | segment(text) | the letters, in order; joined, they give the text back | | segment.Count(s) | count(text) | how many letters a reader sees | | romanize.Word(w, lang, mode, n) | (await load(lang, mode)).word(w, n?) | up to n spellings, most likely first | | romanize.Text(s, lang, mode) | (await load(lang, mode)).text(text) | each word of the language's script in Latin letters | | romanize.Languages(mode) | languages(mode) | the languages with tables (Go: those imported) | | deromanize.Word(latin, lang, mode, n) | (await load(lang, mode)).word(latin, n?) | up to n native spellings, most likely first | | deromanize.Text(s, lang, mode) | (await load(lang, mode)).text(text) | each run of Latin letters in the language's script | | deromanize.Languages(mode) | languages(mode) | the languages with tables (Go: those imported) | | wellformed.Check(s) | isWellFormed(text) | true when a font draws the text with no dotted circle | | wellformed.BrokenAt(s) | brokenAt(text) | where the first dotted circle would be (Go: byte index; TypeScript: UTF-16 index), or -1 |

Three things hold for all of them:

• Language codes are ISO 639: hi, ta, as, and so on. phonetic needs none: it reads the script. normalize and fold use the code for a few rules that belong to one language (Assamese ৰ, for example). stem needs it, and returns the word unchanged for a language with no table.
• Keys are not text. A stem (ஆண்ட), a phonetic key (rn) and the output of fold are for comparing, not for showing or storing as text. Do not stem a stem: stem may cut it again, as other stemmers do.
• Rules change, and stored keys go stale. Each utility exports RULES_VERSION (Go: RulesVersion). Store it beside the keys you save, and compute them again when it changes. Before 1.0, a minor version may change the rules.

How good it is

Each result is on data that no rule was chosen on. The details, the other tools tried and each weak spot are in docs/.

| Utility | Result | Best other tool | Browser file | Details | |---|---|---|---|---| | normalize | makes 88.1% of look-alike spellings equal, and changes the look of no word | Indic NLP Library: 82.4%, and changes 0.68% of words | 23 KB | docs/normalize.md | | stem | finds 3.7–15 points more of the right sentences than exact search, in 22 languages | ahead in 5 languages, level in 2, a trade in 4, behind in Nepali; no other stemmer for the other 10 | 4 KB | docs/stem.md | | phonetic | finds 89% of place-name pairs across scripts (OpenStreetMap, held out, rules 2026-10-11.7; 91% on a second slice, rules 2026-10-11.8; rules 2026-10-12 add 0.34 points on a third); 84% of people on clean names in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a second roster as written (titles, initials; rules 2026-10-11); with phonetic-search, 93% of places (search 2026-10-11.7; 92% on the second slice, search 2026-10-11.8) and 86% of a roster as written (search 2026-10-09) | romanize + Soundex: 71% of place pairs, 23% of people in Tamil; cannot read Urdu; ICU + fuzzy match: 74% of places | 9 KB | docs/phonetic.md | | segment | cuts 0.05% of Kannada words wrongly | Intl.Segmenter: 50.3% | 8 KB | docs/segment.md | | romanize | writes 48.9% of the words of running text as people typed them, in 11 languages; 71.3% of names; in news, 48.1% of names and 39.8% of names it was not trained on; in Hindi, the route option (names it was not trained on get the words tables; 26 KB more on a page that has them, 2.5 MB otherwise) takes news names from 55.0% to 62.2% (Hindi only: in the other languages it loses on Wikidata's names) (tables: 0.2–1.8 MB a language) | IndicXlit (Python, a 119 MB model): 37.9% and 49.0%; in news 49.6%, 58.7% and 59.1% | 11 KB | docs/romanize.md | | deromanize | writes the right word first for 87.1% of the words of Hindi news typed in Latin, and in four for 97.4%; 81.1% of typed sentences in 11 languages (Dakshina, rules 2026-10-12); 69.7% of names in 19 languages; on development names 69.2% with one accepted spelling and 74.5% with every attested spelling, the key of 0.13.0; with its English list, 94.1% of Malayalam and Kannada typed words are right and 91.6% of Tamil (Hindi's list: 89.3%, 81.8% and 89.1%; Kannada calls 7.1% of its words English) (tables: 0.3–3.1 MB a language) | IndicXlit: 86.1%, 89.7%, 77.1% and 66.4%; 66.2% and 71.4%; behind on isolated words out of context (69.4% against 72.6% on Dakshina's lexicons) | 35 KB | docs/deromanize.md | | wellformed | agrees with HarfBuzz on 100% of 135,944 held-out PDF words (9,594 of them broken) and 2,146,974 new Wikipedia words | a check that the word starts with a combining sign: finds 84% of the broken words, with 4,667 false alarms | 6 KB | docs/wellformed.md |

(Browser files are gzipped.)

romanize is also ahead of every tool tried that runs without a neural model. The best of them, indic-trans (about 200 MB), gives 34.7% on running text against 51.2%, on the development data; it is ahead only in Urdu. See docs/romanize.md.

Limits

• deromanize writes English words in the Indian script too, unless you use textMixed (Hindi, Marathi, Kannada, Malayalam and Tamil). Its English list finds 62% of the English words of typed Hinglish and leaves 2.4% of the Hindi words in Latin, most of them chat spellings ("to" for तो, "he" for है). Kannada's list is too eager: it calls 7.1% of Kannada words typed in Latin English and leaves them in Latin (Malayalam 1.7%, Tamil 2.8%). On rare words out of context it is 10 points behind IndicXlit. See docs/deromanize.md.
• Only HarfBuzz was tested for "looks the same" (normalize, segment). Windows and Apple draw text with their own engines.
• The phonetic key reads names, not addresses. It drops a name's leading titles (Shri, Smt., श्रीमती, திரு) and reads one-letter initials (M = एम) from rules 2026-10-11; a title in the middle of a name, or any other honorific, stays a word. On one-word place names the key alone returns many wrong matches: use phonetic-search. See docs/phonetic.md.
• Tamil is the weakest language for phonetic (76%), and stem is behind Snowball's recall in Tamil: in Python or Java, Snowball's Tamil stemmer finds 2.8 points more in passage search. A table that matched its recall merged short Tamil words (இடம் "place" with இடையே "between") and broke the wrong-results budget.
• stem does not help passage search in Hindi or Punjabi. Short question words merge with other words (ਕਿਸ "which" with ਕਿਸਾਨ "farmer"), and names lose their ends (सीता → सी). In Hindi, Lucene's analyzer finds 4.8 points more, from its spelling folds. See docs/stem.md.
• stem damages some names. In news text, 1.3–3.0% of person-name tokens are swamped: most of their stem search's hits are other words (करीना → कर finds करने). For a field of names, pass minStem: 4: it halves that damage. See docs/stem.md.
• An n-gram field can find more than stem in passage search: in Malayalam, Kannada, Tamil, Bengali, Marathi and Hindi, indexing each word's three-letter pieces found 3–10 points more, at 2–3 times the index size. Index stem beside the trigram field, not instead of it: in Hindi, Bengali and Telugu, both fields together found 3.0 points more than trigrams alone and ranked better. See docs/stem.md.
• Thin evidence for the newest tables. Eight languages rest on one test source (BPCC); Manipuri in Bengali script on 1,012 FLORES+ sentences. Santali gains little (3.7 points).
• romanize is behind IndicXlit on OpenStreetMap place names and on rare single words: on a held-out slice of OpenStreetMap the places mode is right 44.4% of the time, IndicXlit 49.1% (it leads on village names, 57.6% against 52.8%; Kannada has no places tables). On names in news that its tables were not trained on, IndicXlit is ahead, 58.7% against 39.8% (Hindi's route option spells them with its words tables). Santali has names only; Bodo and Dogri have text only. Bodo writes ख थ फ as kh th ph, as typists do, not as Aksharantar's k t f. Manipuri in Bengali script loses syllables in the words mode; its tables were trained on Meetei Mayek. See docs/romanize.md.
• normalize keeps the direction marks LRM and RLM (U+200E, U+200F). Text copied from the web carries them, and they hide a match on 0.04% of Wikipedia redirect titles. Next to right-to-left text or digits, deleting one can change the order a reader sees, and the glyph oracle cannot see that. Strip them yourself before normalize when the text has no right-to-left part.
• wellformed gives HarfBuzz's verdict only. DirectWrite (Windows) and CoreText (Apple) insert dotted circles by their own grammars. Arabic and Ol Chiki text is always whole. A verdict that needs more than three characters of context (three or more invisible characters in a row) can be wrong. See docs/wellformed.md.
• Cap user input before phonetic-search. Its cost grows with (query words) x (candidate words): a 4,096-character query takes seconds. See docs/phonetic.md.
• A word of more than 256 letters gets one spelling from romanize and deromanize: each 256-letter piece is spelled alone and the pieces are joined. A run that long is not a word; the cap keeps the time linear.
• Go reads invalid UTF-8 as U+FFFD, so for such input the output does not join back into the original bytes.
• "The same output" holds inside the blocks indickit reads. There, both ports read one pinned table of Unicode 15.0 data (internal/unidata/unicode15.json), whatever Unicode version the host ships; a letter added since 15.0 is not a letter to indickit. Outside the blocks (Latin accents, symbols), Go reads x/text's Unicode 15.0 and TypeScript reads its host (node 24: 17.0), so a newer mark there can give another output: "Ram" plus U+0897 has the key rn in TypeScript and none in Go. Tests pin the table (unicode_test.go, js/unidata.test.ts). The table costs about 2.8 KB gzipped in each browser file that reads it (normalize, phonetic, phonetic-search, romanize, deromanize); without it, those files would follow each browser's Unicode version.
• Two promises have a scope. normalize twice gives what normalize once gives for text with at most 8 invisible characters; 9 BOMs before ૰ need a second call. normalize changes no phonetic key of a word with no invisible character; with one, it can (فاطمہ + ZWNJ).

Each utility's own weak spots are at the end of its file in docs/.

How it is built

Every table and switch of a utility is in its rules file (normalize/rules.json, stem/rules.json, phonetic/rules.json, segment/rules.json, wellformed/rules.json; romanize/rules.json and its tables in romanize/lang/; deromanize/rules.json and its tables and word lists in deromanize/lang/; phonetic-search's tables in phonetic/scorer/); the Go and TypeScript code is a short loop over it. Both are checked against a conformance file of inputs with the outputs that a reference implementation gave them: 345,276 inputs for normalize, 418,665 for stem, 199,992 for phonetic, 858,654 for segment, 3,347 for romanize, 3,119 for deromanize, 32,279 for wellformed, and 1,031 search rows for phonetic-search (*/testdata/*conformance.jsonl.gz). A change that makes any one disagree on any input fails the build.

Credits and licence

MIT. The test words are Wikidata labels (CC0). The rules were tuned on Wikidata and PIB Parallel (Press Information Bureau releases), and checked on AI4Bharat's Aksharantar and on Wikipedia text. No text of theirs ships here, except deromanize's word counts (below). normalize and segment were checked in fonts by Google (Noto), Ek Type (Anek), SIL, SMC and others. stem's endings were mined from PIB text; Wiktionary's inflection tables (CC BY-SA), Universal Dependencies treebanks and FLORES+ (CC BY-SA) were used only to measure them. It was compared with Snowball, Lucene and the Indic NLP Library.

romanize and deromanize learned from AI4Bharat's Aksharantar (CC BY for its hand-collected pairs, CC0 for its mined pairs), Wikidata labels (CC0), and IndicXlit's romanizations (MIT) of frequent Wikipedia words. deromanize's tables hold word lists: counts of the words of Wikipedia (text CC BY-SA 4.0) and, for Bodo and Dogri, of IndicCorp v2 (CC0). Both were measured against IndicXlit, on Google's Dakshina (CC BY-SA 4.0) and, for deromanize, on COMI-LINGUA (CC BY 4.0), used only to measure.