mixed-language-detector

Fast, lightweight, offline language identification for native scripts, Romanized / transliterated text, and code-mixed sentences across 11 major world and Indian languages in Node.js and TypeScript.

Highlights

• 11 Languages Supported:
  • English (EN): Standard Latin English
  • Hindi (HI): Devanagari script + Hinglish
  • Gujarati (GU): Gujarati script + Gujlish
  • Marathi (MR): Devanagari script + Marathlish
  • Tamil (TA): Tamil script + Tanglish
  • Telugu (TE): Telugu script + Telugish
  • Urdu (UR): Perso-Arabic Nastaliq script + Roman Urdu
  • Arabic (AR): Arabic script + Arabizi
  • Japanese (JA): Hiragana/Katakana/Kanji + Romaji
  • Korean (KO): Hangul script + Romanized Korean
  • Chinese (ZH): Simplified/Traditional Hanzi + Pinyin
• Code-Mixed & Code-Switching Detection: Accurately classifies sentences where English loanwords, greetings, and colloquial markers are intermingled with regional languages (e.g. "Hi, mera naam Jatin hein", "vanakkam epdi irukinga", "sabah al khair habibi", "konnichiwa arigato gozaimasu").
• Zero Runtime Dependencies: No Python runtime required in production. No LLMs, RAG, ChromaDB, FAISS, or external network APIs. Everything runs locally in pure TypeScript/JavaScript.
• Ultra-Fast Sub-Millisecond Inference: Employs an exported character $n$-gram TF-IDF Logistic Regression vectorizer and multi-signal ensemble running in microseconds per sentence with high throughput.
• Confidence Calibration & UNKNOWN Guardrails: Rejects noise, URLs, emojis, and isolated proper names (e.g. "Jatin" $\rightarrow$ UNKNOWN) when linguistic signal is ambiguous or insufficient.
• Dual CommonJS + ESM Support: Pre-bundled with TypeScript declarations (.d.ts), ready for Node.js, modern web bundlers, and serverless runtimes.

Installation

npm install mixed-language-detector

Works out of the box with both ES Modules (ESM) and CommonJS (CJS).

Quickstart

1. Basic Language Detection

import { detectLanguage } from "mixed-language-detector";

// English
detectLanguage("Hi. I am Jatin"); // "EN"

// Hindi: Native Devanagari & Hinglish
detectLanguage("मेरा नाम जतिन है"); // "HI"
detectLanguage("Hi, mera naam Jatin hein"); // "HI"
detectLanguage("Hello bhai kya kar raha hai"); // "HI"

// Gujarati: Native Script & Gujlish
detectLanguage("મારું નામ જતિન છે"); // "GU"
detectLanguage("Aaje hu office ma jais"); // "GU"
detectLanguage("kem cho maza ma chho"); // "GU"

// Marathi: Native Devanagari & Marathlish
detectLanguage("माझं नाव जतिन आहे"); // "MR"
detectLanguage("majhe naav jatin aahe"); // "MR"
detectLanguage("tu kasa ahes bhava"); // "MR"

// Tamil: Native Script & Tanglish
detectLanguage("வணக்கம் எப்படி இருக்கீங்க"); // "TA"
detectLanguage("vanakkam epdi irukinga"); // "TA"

// Telugu: Native Script & Telugish
detectLanguage("నమస్కారం బాగున్నారా"); // "TE"
detectLanguage("namaskaram bagunnara ela unnaru"); // "TE"

// Urdu: Native Perso-Arabic & Roman Urdu
detectLanguage("آپ کیسے ہیں جناب"); // "UR"
detectLanguage("kya haal hai janab sab kheriyat"); // "UR"

// Arabic: Native Script & Arabizi
detectLanguage("مرحبا كيف حالك اليوم"); // "AR"
detectLanguage("sabah al khair habibi"); // "AR"

// Japanese: Native Kana/Kanji & Romaji
detectLanguage("こんにちは、お元気ですか？"); // "JA"
detectLanguage("konnichiwa arigato gozaimasu"); // "JA"

// Korean: Native Hangul & Romanized Korean
detectLanguage("안녕하세요 만나서 반갑습니다"); // "KO"
detectLanguage("annyeonghaseyo kamsahamnida"); // "KO"

// Chinese: Native Hanzi & Pinyin
detectLanguage("你好，很高兴认识你"); // "ZH"
detectLanguage("ni hao xie xie zai jian"); // "ZH"

2. Detailed Confidence & Score Breakdown

import { detectLanguageDetailed } from "mixed-language-detector";

const result = detectLanguageDetailed("வணக்கம் எப்படி இருக்கீங்க");
console.log(result);

Output:

{
  "language": "TA",
  "confidence": 1.0,
  "scores": {
    "EN": 0.0,
    "HI": 0.0,
    "GU": 0.0,
    "MR": 0.0,
    "TA": 1.0,
    "TE": 0.0,
    "UR": 0.0,
    "AR": 0.0,
    "JA": 0.0,
    "KO": 0.0,
    "ZH": 0.0
  },
  "script": "TAMIL"
}

3. Handling Ambiguity & Proper Names

When input lacks sufficient linguistic evidence (e.g. isolated names, numbers, or symbols), UNKNOWN is returned rather than guessing:

detectLanguage("Jatin"); // "UNKNOWN"
detectLanguage("12345 67890"); // "UNKNOWN"
detectLanguage("😀🎉🚀"); // "UNKNOWN"

Supported Languages

| Code | Language | Native Scripts | Romanized Transliteration | | :--- | :--- | :--- | :--- | | EN | English | Latin (A-Z, a-z) | Standard English | | HI | Hindi | Devanagari (U+0900–U+097F) | Hinglish | | GU | Gujarati | Gujarati (U+0A80–U+0AFF) | Gujlish | | MR | Marathi | Devanagari (U+0900–U+097F) | Marathlish | | TA | Tamil | Tamil (U+0B80–U+0BFF) | Tanglish | | TE | Telugu | Telugu (U+0C00–U+0C7F) | Telugish | | UR | Urdu | Perso-Arabic (U+0600–U+06FF) | Roman Urdu | | AR | Arabic | Arabic (U+0600–U+06FF) | Arabizi | | JA | Japanese | Hiragana, Katakana, Kanji | Romaji | | KO | Korean | Hangul (U+AC00–U+D7AF) | Romanized Korean | | ZH | Chinese | Hanzi (U+4E00–U+9FFF) | Pinyin | | UNKNOWN | Unidentified | Any / Mixed Noise | Below confidence threshold |

Architecture

The detector employs a multi-signal tiered decision process:

                  Input Text
                      ↓
            Text Normalization
    (Whitespace, URLs, Emojis, Elongations)
                      ↓
           Unicode Script Detection
        (Devanagari, Gujarati, Latin)
                      ↓
                 Tokenization
      (Words, Subwords, Capitalization)
                      ↓
          Native-Script Disambiguation
   (Gujarati -> GU; Devanagari -> HI vs MR Lexicon)
                      ↓
        Romanized Word & Phrase Scoring
    (Grammatical Affixes, Strong Words, Slang)
                      ↓
       Machine Learning Char N-Gram Model
   (TF-IDF Sublinear Log-Reg Sparse Dot-Product)
                      ↓
           Multi-Signal Evidence Fusion
        (Log-Linear Pooling & Sharpening)
                      ↓
       Confidence Calibration & Thresholding
                      ↓
          Final Language Result ("EN" / "HI" / "GU" / "MR" / "UNKNOWN")

Script Disambiguation (Devanagari Hindi vs. Marathi)

Because both Hindi and Marathi share the Devanagari Unicode block (U+0900–U+097F), text cannot be classified by Unicode range alone. The engine uses language-specific grammatical particles and verb auxiliaries to disambiguate:

• Marathi: आहे, नाही, काय, कसा, करतोय, झाला, मला, तुला, उद्या
• Hindi: है, हैं, नहीं, क्या, कैसे, करता, हुआ, मुझे, तुझे, कल

Configuration Options

Both detectLanguage and detectLanguageDetailed accept an optional options object:

interface DetectorOptions {
  /**
   * Confidence threshold between 0.0 and 1.0.
   * If the top candidate confidence is below this threshold, UNKNOWN is returned.
   * Default: 0.65
   */
  confidenceThreshold?: number;

  /**
   * Minimum non-whitespace characters required.
   * Default: 2
   */
  minTextLength?: number;

  /**
   * Whether to include token-level breakdown in results.
   * Default: false
   */
  includeTokenAnalysis?: boolean;

  /**
   * Custom weight distribution for the multi-signal ensemble.
   */
  weights?: {
    script?: number;
    lexicon?: number;
    phrase?: number;
    ml?: number;
  };
}

Example with Options:

const result = detectLanguageDetailed("Hi, mera naam Jatin hein", {
  confidenceThreshold: 0.80,
  includeTokenAnalysis: true,
});

console.log(result.details.tokens);
/*
[
  { token: 'Hi', normalized: 'hi', language: 'UNKNOWN', weight: 0.2, reason: 'lexicon_shared' },
  { token: 'mera', normalized: 'mera', language: 'HI', weight: 2.0, reason: 'lexicon_strong' },
  { token: 'naam', normalized: 'naam', language: 'UNKNOWN', weight: 0.2, reason: 'lexicon_shared' },
  { token: 'Jatin', normalized: 'jatin', language: 'UNKNOWN', weight: 0, reason: 'name_or_entity' },
  { token: 'hein', normalized: 'hai', language: 'HI', weight: 2.0, reason: 'lexicon_strong' }
]
*/

Machine Learning Pipeline & Export

The ML model is trained via Python and scikit-learn using character $n$-grams and Logistic Regression:

1. Why Character $n$-grams? Romanized Indian text has wide spelling and transliteration variance (e.g. hai / hein / hain, che / chhe, aahe / ahe). Word-level dictionaries alone miss novel spellings; character sub-sequences capture these invariant phonetic roots.
2. Empirical Range Comparison:
   • char (2, 4): 5-Fold F1 Macro = 0.9887
   • char (2, 5): 5-Fold F1 Macro = 0.9904 (Selected as optimal)
   • char (3, 6): 5-Fold F1 Macro = 0.9853
3. Pure TypeScript Runtime: Model weights, vocabulary, IDFs, and coefficients are exported to a compact JSON file (src/models/model.json, ~190 KB). In Node.js, vectorization, sparse sublinear TF-IDF weighting, dot products, and softmax are implemented natively without external dependencies.

Evaluation Results

Evaluated on a stratified held-out test set ($N = 2,953$ sentences across all 11 languages) with zero train-test leakage:

Overall Metrics

• Accuracy: 98.54% on held-out test dataset
• Macro Precision: 97.72%
• Macro Recall: 98.75%
• Macro F1-Score: 98.19%
• Weighted F1-Score: 98.56%

Per-Language Classification Report

              precision    recall  f1-score   support

          AR     1.0000    0.9932    0.9966       147
          EN     0.9226    1.0000    0.9597       143
          GU     1.0000    0.9940    0.9970       166
          HI     0.9972    0.9828    0.9900      1455
          JA     1.0000    0.9524    0.9756       147
          KO     1.0000    0.9795    0.9896       146
          MR     0.9821    0.9880    0.9851       167
          TA     1.0000    1.0000    1.0000       146
          TE     1.0000    1.0000    1.0000       145
          UR     0.9467    0.9793    0.9627       145
          ZH     0.9006    0.9932    0.9446       146

    accuracy                         0.9854      2953
   macro avg     0.9772    0.9875    0.9819      2953
weighted avg     0.9863    0.9854    0.9856      2953

Sub-Category Breakdown

• Native Scripts: 1,260 samples | 99.05% Accuracy | 99.06% F1
• Romanized Text: 364 samples | 96.98% Accuracy | 95.96% F1
• Mixed-Language Text: 1,329 samples | 98.50% Accuracy

[!NOTE] While the model achieves >98.5% accuracy across all 11 languages, real-world arbitrary text with heavy slang, single isolated ambiguous words, or heavy code-mixing across three or more languages cannot be guaranteed to have 100% accuracy. The package uses confidence thresholds and returns UNKNOWN when evidence is ambiguous.

Performance Benchmark

Measured on a standard Node.js v20 runtime processing 60,000 sentences:

Total Sentences Processed:  60,000
Average Latency:            421.40 µs / sentence (0.42 ms)
Throughput:                 2,373 sentences / second
Heap Memory Used:           17.47 MB

Realistic Limitations & Trade-offs

1. Shared Vocabulary: Words like naam, office, time, bhai, and meeting exist across multiple languages. Without sufficient contextual words, the detector safely reports UNKNOWN.
2. Proper Names: Uncapitalized or unusual names (e.g. "Ramesh" vs "ramesh") without context cannot be definitively attributed to any language.
3. Dialectal Variation: Hyper-local colloquialisms not present in the character n-gram corpus or dictionary may have lower confidence.

Adding a New Language

The architecture is designed so additional languages (e.g. Tamil TA, Telugu TE, Bengali BN, Kannada KN, Punjabi PA) can be added cleanly:

1. Create Language Definition: Create src/languages/<language>.ts implementing LanguageDefinition:

   import { LanguageDefinition } from "../types";
   
   export const tamilDefinition: LanguageDefinition = {
     code: "TA",
     name: "Tamil",
     primaryScripts: ["OTHER", "LATIN"],
     strongWords: ["vanakkam", "epdi", "irukinga", "romba", "nandri"],
     mediumWords: ["inga", "anga", "ippo", "naan", "neenga"],
     sharedWords: ["office", "tea", "bhai"],
     phrases: ["epdi irukinga", "romba nandri"],
   };

2. Register in src/languages/index.ts:

   LanguageRegistry.getInstance().register(tamilDefinition);

3. Add Training Samples & Retrain: Add native and Romanized sentences to training/datasets/dataset.csv, then run:

   python training/preprocess.py
   python training/train.py

   The updated src/models/model.json is exported automatically.

Development & Testing

Prerequisites

• Node.js >= 18
• Python >= 3.10 (only needed for retraining models, not for runtime)

Build

npm run build

Typecheck

npm run typecheck

Run Tests

npm test
npm run test:coverage

Run Benchmark

npm run benchmark

Run Demo

npm run demo

License

MIT © Jatin