Mucellit 📖

High-performance unified document processing runtime for Node.js.

Mucellit (mücellit — named after the historic artisan bookbinders and manuscript restorers who bound, repaired, collated, and preserved delicate folios) is a single, unified runtime for orchestrating document analysis, text extraction, OCR, conversion, previews, thumbnails, search indexing, diffing, PII redaction, and in-memory editing across PDF, DOCX, PPTX, XLSX, Markdown, and Plain Text documents.

Instead of writing parsers from scratch or bundling fragile native C++ bindings, Mucellit acts as a high-performance document tooling orchestrator. It integrates PDF.js (pdfjs-dist), modern office engines, and system tools (tesseract, ghostscript, libreoffice, sips).

⚡ Highlights

• Single npm package: Published cleanly as mucellit with a minimalistic user API and TypeScript definitions.
• High-Performance Architecture: Zero-copy slab memory buffer pooling and compact intermediate representation (IR) to prevent V8 heap bloat across large documents.
• PDF.js Core: Robust, standard-compliant PDF parsing and text layout extraction via pdfjs-dist.
• Modern Office Engine: Fast, secure parsing for DOCX, PPTX, and XLSX without heavy native bindings.
• PII & KVKK Redaction Engine: Automated detection and masking of TCKN (checksum verified), credit cards (Luhn algorithm), IBAN (Mod-97), email, phone, and VKN.
• RAG & LLM Chunking Engine: Semantic, hierarchical (markdown sections), page, and sliding window chunking with metadata enrichment and token estimation.
• Document Editing Engine: In-memory text find & replace, page manipulation (removal, reordering, rotation, insertion), watermarks, headers & footers.
• 60+ Language Detection: Sub-millisecond N-gram document language detection and confidence scoring powered by eld.
• Extensive Operations: Analysis, flow/raw/block text extraction, OCR, vector SVG and HTML previews, raster thumbnails, format conversion, compression, full-text BM25 search indexing, and structural diffing.
• Canonical Document IR & Binary Serialization (.mir): Format-independent canonical intermediate representation with stable block identities, multi-level fingerprints, and 12x smaller compact binary storage with schema migrations.
• Incremental Document Processing & Changeset API: Granular dependency invalidation for RAG chunking (>99% work avoided on minor document edits), incremental BM25 search index updates, and page-level OCR/preview caching.
• Fidelity & Quality Engine: Deterministic multi-dimensional scoring (character/word preservation, layout bbox density, table structure, images, fonts) and optional pixel-level visual regression mode.
• Corpus & Compatibility Runner: Automated test harness for real-world document compatibility across 19 categories (scanned, large, complex layout, corrupted, RTL, CJK) with CI quality gates and failure regression fixtures.
• Deterministic Fixture Generator: Generate reproducible synthetic test documents (PDF, DOCX, XLSX, Markdown, Plain Text) with seeded PRNG.
• Powerful CLI: Full terminal CLI with mucellit doctor, analyze, text, chunk, redact, edit, preview, convert, ocr, diff, index, corpus, fidelity, fixture, and mcp.

📦 Installation

npm install mucellit

🚀 Quick Start

import { open, capabilities } from "mucellit";

// Open any document (PDF, DOCX, PPTX, XLSX, Markdown, TXT)
const doc = await open("./report.pdf");

// Basic metadata and page count
const info = await doc.info();

// Extract text
const text = await doc.text({ layout: "flow" });

// Generate page preview (SVG, HTML, or PNG)
const svgPreview = await doc.preview({ page: 1, format: "svg" });

// Generate a thumbnail
const thumb = await doc.thumbnail({ size: 200 });

// Convert to other formats
const pdfBuffer = await doc.convert("pdf");

// Free allocated memory
doc.close();

🔍 Document Analysis

const doc = await open("./invoice.pdf");
const analysis = await doc.analyze();

Output:

{
  "type": "pdf",
  "mime": "application/pdf",
  "size": 1048576,
  "pages": 12,
  "language": "en",
  "hasText": true,
  "needsOcr": false,
  "encrypted": false,
  "metadata": {
    "title": "Quarterly Invoice",
    "author": "Acme Corp"
  },
  "hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}

Language Detection (ELD)

Mucellit includes built-in N-gram language detection powered by eld, covering 60+ ISO 639-1 languages with sub-millisecond execution and confidence scoring:

import { detectLanguage, detectLanguageDetails } from "mucellit";

// Fast ISO 639-1 code detection
const lang = detectLanguage("Bu bir Türkçe dokümandır."); // 'tr'

// Detailed scores & reliability
const details = detectLanguageDetails("Bu bir Türkçe teknik dokümandır.");
// {
//   language: "tr",
//   isReliable: true,
//   scores: { tr: 0.81, ... },
//   supported: true
// }

✏️ Document Editing & Manipulation

Mucellit provides a chainable, fluent document editing API for modifying pages, text content, watermarks, headers, footers, and metadata:

import { open } from "mucellit";

const doc = await open("./report.pdf");

// 1. Find & replace text (supports strings or RegExp)
doc.replaceText("Draft Version", "Official Release v1.0");

// 2. Page manipulation (remove, reorder, rotate, insert)
doc.removePage(2)                           // Remove page 2
   .reorderPages([3, 1, 2])                 // Reorder pages
   .rotatePage(1, 90)                       // Rotate page 1 by 90°
   .insertPage(1, { text: "Cover Title" }); // Insert cover page

// 3. Watermarks, headers & footers
doc.addWatermark("CONFIDENTIAL", { opacity: 0.2 })
   .addHeader("Mucellit Internal Audit")
   .addFooter("Page {page} of {total}");

// 4. Update metadata and save
doc.setMetadata({ author: "Antigravity Team", title: "Audit 2026" });
await doc.save("./report-final.pdf");

doc.close();

🛡️ PII & KVKK Redaction Engine

Mucellit provides built-in, zero-dependency PII detection, auditing, and redaction for compliance with KVKK and GDPR regulations before documents are indexed into vector databases or passed into LLM pipelines:

• T.C. Kimlik No (TCKN): 11-digit Turkish identification numbers verified with the official 11-digit checksum algorithm.
• Credit & Debit Cards: 13-19 digit card numbers verified with the Luhn Algorithm (Mod 10).
• IBAN: Turkish and international bank account numbers verified with ISO 7064 Modulo-97.
• Contact Info: RFC-compliant email addresses and international/Turkish phone numbers.
• Tax Identification (VKN): Turkish Vergi Kimlik Numarası with official Ministry of Finance checksums.
• Custom Rules & Keywords: Support for user-defined regex rules and custom keywords/project codenames.

import { open } from "mucellit";

const doc = await open("./customer-contract.pdf");

// 1. Audit / Scan non-destructively for sensitive information
const audit = doc.scanPII();
console.log(`Found ${audit.totalFound} sensitive records:`, audit.byType);
// { tckn: 2, creditCard: 1, email: 3, iban: 1 }

// 2. Redact with type replacement (e.g. [TCKN], [KREDİ KARTI], [IBAN])
doc.redactPII({
  mode: "replace" // replaces matches with [TCKN], [KREDİ KARTI], etc.
});

// 3. Or mask partially (preserving format: 100******46, 4012 **** **** 1881, a***[email protected])
doc.redactPII({
  mode: "mask",
  types: ["tckn", "creditCard", "iban"]
});

// 4. Save sanitized document to disk
await doc.save("./customer-contract-clean.pdf");
doc.close();

🧩 RAG & LLM Document Chunking

Mucellit features an intelligent chunking engine designed specifically for vector databases (Chroma, Pinecone, Qdrant, Milvus) and LLM contexts (OpenAI, Claude, Gemini, Llama). Unlike naive text splitters, Mucellit preserves section hierarchies, sentence integrity, and table layouts:

import { open } from "mucellit";

const doc = await open("./technical-manual.pdf");

// 1. Semantic Chunking (sentence & paragraph aware)
const chunks = await doc.chunk({
  strategy: "semantic",
  maxTokens: 400,
  overlapTokens: 50
});

// 2. Hierarchical / Markdown Section Chunking (with contextual path)
const sectionChunks = await doc.chunk({
  strategy: "hierarchical",
  prependSectionPath: true, // Prepends "[Context: Architecture > Storage]" to chunk text
  maxTokens: 300
});

// 3. Page-based Chunking (1 chunk per page for precise citations)
const pageChunks = await doc.chunk({ strategy: "page" });

console.log(chunks[0]);
// {
//   id: "chunk_0",
//   text: "Mucellit Architecture...",
//   metadata: {
//     documentTitle: "technical-manual.pdf",
//     format: "pdf",
//     language: "en",
//     pageNumber: 1,
//     pageNumbers: [1],
//     section: "Introduction",
//     sectionPath: ["Technical Manual", "Introduction"],
//     index: 0,
//     charCount: 1420,
//     wordCount: 220,
//     tokenCountEstimate: 290,
//     hasTable: false,
//     hasHeading: true
//   }
// }

doc.close();

🏛️ Ecosystem Integration

1. cemalloc (Memory Management)

• Applied to large byte buffer operations (reading 100MB+ documents, PDF decompression, image resizing).
• Reduces V8 garbage collection latency spikes using slab allocation and scratch arenas.
• Provides seamless fallback to Node's BufferPool if cemalloc is not installed.

2. muttafa (Compact Document IR)

• Document Intermediate Representation (IR) is buffered in Muttafa mutation engines and compact stores instead of creating thousands of loose V8 objects.
• Stores pages, text blocks, bounding box coordinates, tables, images, and annotations.

Document
 ├── metadata
 ├── pages
 │    ├── text blocks
 │    ├── coordinates
 │    ├── images
 │    ├── tables
 │    └── annotations
 ├── assets
 └── indexes

3. raptiye (Worker Orchestration)

• Offloads heavy jobs (batch OCR, multi-page conversion, preview generation) through a distributed worker transport.

import { mucellit } from "mucellit";

// Local mode (default worker pool)
await mucellit.process(file, { action: "analyze" });

// Distributed mode (dispatched via Raptiye transport)
await mucellit.process(file, {
  transport: "raptiye",
  action: "ocr"
});

💾 Canonical Document IR & Compact Serialization (.mir)

Mucellit provides a format-independent, schema-versioned Canonical Document Intermediate Representation (IR). Each page and block receives a stable, position-independent identity and multi-level fingerprint.

import { open, DocumentIR } from "mucellit";

const doc = await open("./contract.pdf");

// 1. Build canonical IR
const ir = await doc.ir();
console.log(`Pages: ${ir.pageCount}, Sections: ${ir.sections.length}`);

// 2. Multi-level fingerprint (binary, content, structure, and per-page)
const fp = doc.fingerprint();
// { binary: "...", content: "...", structure: "...", pages: [{ page: 1, content: "...", structure: "..." }] }

// 3. Compact binary serialization (.mir) - 12x smaller than JSON
const binary = await ir.serialize();
await ir.save("./contract.mir");

// 4. Restore or load persisted IR
const restored = await DocumentIR.load("./contract.mir");
// restored.pages, restored.tables, restored.sections available immediately without re-parsing!

⚡ Incremental Document Processing (DocHost Engine)

When a new version of a document arrives (e.g. contract-v2.pdf), Mucellit avoids re-processing unchanged pages through granular dependency invalidation.

import { open, DocumentIR } from "mucellit";

const prev = await open("./contract-v1.pdf");
const curr = await open("./contract-v2.pdf");

// 1. Granular structural changeset
const changes = await curr.changesSince(prev);
console.log(changes);
// {
//   documentChanged: true,
//   pages: { added: [], removed: [], changed: [4], unchanged: [1, 2, 3, 5] },
//   blocks: { added: 1, removed: 1, changed: 0 },
//   text: { addedCharacters: 142, removedCharacters: 38 }
// }

// 2. Incremental RAG Chunking (>99% work avoided on minor revisions)
const delta = await curr.chunkIncremental(prev, { strategy: "semantic" });
// {
//   keep: ["chunk_p1_...", "chunk_p2_..."],
//   remove: ["chunk_p4_..."],
//   add: [{ id: "chunk_p4_new", text: "..." }],
//   workAvoided: 0.98
// }

// 3. Incremental BM25 Search Index update
await curr.updateIndex(prev);

// 4. Incremental Pipeline Execution
const result = await curr.process({
  previous: prev, // or previous DocumentIR
  text: true,
  ocr: "auto",
  chunk: true,
  index: true
});

📐 Fidelity & Quality Measurement

Deterministic heuristic evaluation of character preservation, layout bounding boxes, table structures, images, and fonts, with optional visual regression diffing:

import { fidelity } from "mucellit";

const score = await fidelity("./original.docx", "./converted.pdf");
console.log(score);
// {
//   overall: 0.94,
//   text: 0.99,
//   layout: 0.92,
//   tables: 0.89,
//   images: 0.96,
//   fonts: 0.91
// }

// Optional pixel visual regression mode with difference map
const visualScore = await fidelity("./original.pdf", "./rendered.pdf", {
  visual: true,
  diffOutput: "./diff-output"
});

🛠️ System Capabilities & Doctor

Probe system tools and runtime features programmatically:

import { capabilities } from "mucellit";

console.log(await capabilities());
// {
//   pdf: true,
//   office: true,
//   ocr: true,
//   storage: true,
//   distributed: true,
//   details: { ... }
// }

Or via CLI:

🏭 Production Usage

Mucellit is architected as an industrial-grade document processing runtime designed for predictable memory usage under high concurrency and multi-gigabyte workloads.

Composable Pipeline API

Run complex, dependency-aware document workflows with zero boilerplate. Intermediate dependencies (like text extraction for chunking, or analysis for OCR) are automatically resolved and deduplicated:

import { mucellit } from "mucellit";

const result = await mucellit.pipeline("./large-contract.pdf")
  .analyze()
  .extractText()
  .ocr({ mode: "auto" })
  .redactPII({ mode: "mask" })
  .chunk({ strategy: "semantic" })
  .withLimits({ memory: "512mb", cpuTime: 30_000 })
  .onProgress((ev) => console.log(`[${ev.operation}] ${Math.round(ev.progress * 100)}% - ${ev.step}`))
  .run();

Dry-Run Execution Planner

Inspect execution graphs, required engine steps, and estimated memory before committing resources:

const plan = await mucellit.plan("./scanned-archive.pdf", {
  preview: true,
  ocr: "auto",
  chunk: true
});

console.log(plan);
// {
//   detected: "pdf",
//   pages: 218,
//   operations: ["pdf.inspect", "page.render", "ocr", "text.normalize", "chunk.semantic"],
//   estimatedMemory: 220000000,
//   estimatedWorkUnits: 218
// }

CLI equivalent:

mucellit plan ./scanned-archive.pdf --json

Isolated Process Workers

Run risky external tools (Tesseract, LibreOffice, Ghostscript) in an isolated child process with hard memory ceilings and auto-kill timeouts. An engine crash or memory leak will never bring down your main Node.js process:

const result = await mucellit.process("./untrusted-upload.docx", {
  action: "text",
  isolation: "process", // Child process fork with --max-old-space-size
  limits: { memory: "256mb", cpuTime: 15_000 }
});

📦 Large Documents & Streaming

Mucellit allows processing files from 1 MB up to 1 GB+ without forcing the entire payload into V8 heap memory.

Universal Input Streams

Open documents from Buffer, Uint8Array, Node.js Readable, Web ReadableStream, or filesystem path:

import fs from "node:fs";
import { open } from "mucellit";

// Filesystem path (zero unnecessary buffer allocation)
const doc1 = await open("/data/huge-1gb-book.pdf");

// Node.js stream
const fileStream = fs.createReadStream("./dump.pdf");
const doc2 = await open(fileStream);

// Web Streams API (fetch/fetch-like environments)
const response = await fetch("https://example.com/spec.pdf");
const doc3 = await open(response.body);

Incremental Processing Generators

Iterate through large documents page-by-page or chunk-by-chunk without massive array allocations:

// Stream text page-by-page
for await (const page of doc.streamText()) {
  console.log(`Page ${page.pageNumber}: ${page.text.length} chars`);
}

// Stream RAG chunks incrementally
for await (const chunk of doc.streamChunks({ strategy: "semantic" })) {
  await vectorDb.upsert(chunk);
}

// Real-time progress events generator
for await (const event of doc.processStream({ text: true, chunk: true })) {
  console.log(`[${event.operation}] Progress: ${(event.progress * 100).toFixed(1)}%`);
}

⚡ Resource Limits & Cancellation

Set strict resource budgets per document job. When limits are exceeded, Mucellit aborts cleanly with typed errors instead of crashing the process:

import { open } from "mucellit";

try {
  const doc = await open("./incoming.pdf", {
    limits: {
      memory: "512mb",      // Max memory budget ("256mb", "1gb", or bytes)
      cpuTime: 30_000,      // Max wall/CPU time in ms
      maxPages: 500,        // Max allowed page count
      maxOutputSize: "1gb", // Max output size
      maxOcrPages: 50       // Max pages permitted for OCR
    }
  });

  await doc.process({ text: true, chunk: true });
} catch (err) {
  // Strongly typed Sahaf/Mucellit errors:
  // err.code: MEMORY_LIMIT | PAGE_LIMIT | TIMEOUT | OUTPUT_LIMIT | OCR_LIMIT
  console.error(`Job aborted: [${err.code}] ${err.message}`);
}

First-Class Cancellation (AbortSignal)

Every long-running operation, including external child processes, supports AbortSignal for instant cancellation:

const controller = new AbortController();

setTimeout(() => controller.abort(), 5000); // 5s timeout

await doc.ocr({
  signal: controller.signal,
  timeout: 30_000 // Shortcut timeout option
});

🛡️ Security & Document Safety

Mucellit assumes untrusted user uploads and enforces strict security defenses:

• Decompression Bomb Protection: Zip extraction validates compression ratios (capped at 150:1), total uncompressed output (<512 MB), and maximum entries (<10,000).
• XML Entity & XXE Shield: All OOXML parsers inspect XML headers for DTD, ENTITY, and recursive expansion attacks.
• Path Traversal Guard: Prevents malicious zip archive entries from escaping temp boundaries (../).
• Centralized Temp Lifecycle: All scratch files (LibreOffice, OCR PNGs, Ghostscript outputs) are registered with tempManager and automatically deleted upon process exit, termination signals (SIGINT, SIGTERM), or via CLI:

# Purge stale temporary artifacts older than 1 hour
mucellit cleanup --hours 1

💾 Content-Addressed Caching & Fingerprints

Avoid recomputing expensive operations (OCR, previews, thumbnails, text layout, semantic chunking) using deterministic content addressing:

import { createMucellit } from "mucellit";

const mucellit = createMucellit({
  cache: {
    directory: "./.mucellit-cache", // FileSystem or memory
    maxSize: "10gb"                 // LRU eviction ceiling
  }
});

// Cache key: SHA256(document) + ":" + operation + ":" + SHA256(options) + ":v" + version
const doc = await mucellit.open("./quarterly.pdf");
const text = await doc.text(); // 1st run: computed and cached
const cachedText = await doc.text(); // 2nd run: instant cache hit

3-Way Document Fingerprinting

Detect duplicate documents even when binary metadata (like creation date or producer) changes:

const fp = await doc.fingerprint();
// {
//   binary: "sha256...",    // Exact file byte hash
//   content: "sha256...",   // Normalized textual content hash
//   structure: "sha256..."  // Page layout, headings, and table structure hash
// }

📈 Observability & Telemetry

Capture deep performance metrics for every operation without external runtime dependencies:

const result = await doc.process({
  text: true,
  chunk: true,
  telemetry: true
});

console.log(result.telemetry);
// {
//   totalMs: 42,
//   cpuTimeMs: 38,
//   peakMemory: 142000000,
//   bytesRead: 1048576,
//   cacheHits: 1,
//   cacheMisses: 1,
//   operations: [
//     { name: "text", durationMs: 14, cpuTimeMs: 12, cacheHit: true },
//     { name: "chunk", durationMs: 28, cpuTimeMs: 26, cacheHit: false }
//   ]
// }

OpenTelemetry Bridge Hook

Easily stream telemetry spans into your existing OpenTelemetry collector:

import { registerOtelHook } from "mucellit";

registerOtelHook({
  onSpanStart: (spanName, attributes) => tracer.startSpan(spanName, { attributes }),
  onSpanEnd: (spanName, attributes, metrics) => mySpan.end()
});

💻 CLI Usage

# Analyze document
mucellit analyze report.pdf
mucellit analyze report.pdf --json

# Extract text
mucellit text report.pdf
mucellit text report.docx --page 1 --output page1.txt

# Preview generation
mucellit preview report.docx --page 1 --format svg
mucellit preview report.pdf --page 1 --format png --output preview.png

# Format conversion
mucellit convert report.docx --to pdf
mucellit convert data.xlsx --to csv
mucellit convert notes.md --to html

# OCR scanned documents
mucellit ocr scan.pdf --lang eng --output ocr.txt

# Edit documents (replace text, remove/reorder pages, watermark)
mucellit edit document.pdf --replace "Draft:Final" --output final.pdf
mucellit edit manual.pdf --remove-pages 2,4 --watermark "CONFIDENTIAL" --output out.pdf
mucellit edit report.pdf --reorder-pages 3,1,2 --set-title "New Title" --output reordered.pdf

# Detect and redact PII / KVKK (TCKN, cards, IBAN, email, phone)
mucellit redact customer.pdf --scan-only
mucellit redact customer.pdf --mode mask --output sanitized.pdf
mucellit redact contract.docx --types tckn,creditCard,iban --mode replace --output clean.docx

# Compare two documents (structural diff)
mucellit diff version1.docx version2.docx

# Full-text BM25 search
mucellit index manual.pdf --query "authentication"

# Corpus compatibility testing & quality gates
mucellit corpus ./tests/corpus --min-open 99 --min-text 98 --max-crashes 0

# Measure fidelity between original and rendered document
mucellit fidelity original.docx rendered.pdf --min-overall 0.90
mucellit fidelity original.pdf rendered.pdf --visual --diff-output ./diffs

# Generate deterministic synthetic fixtures
mucellit fixture pdf --pages 100 --tables 10 --seed 42 --out fixture.pdf
mucellit fixture docx --pages 20 --seed 123 --out fixture.docx

# Start Model Context Protocol (MCP) Server
mucellit mcp

🤖 Model Context Protocol (MCP) Server

Mucellit includes built-in MCP server capabilities powered by mcponce. AI assistants (Claude Desktop, Cursor, Antigravity, VS Code) can directly inspect, parse, search, convert, and diff documents.

Client Configuration (Claude Desktop / Cursor / Antigravity)

{
  "mcpServers": {
    "mucellit": {
      "command": "npx",
      "args": ["-y", "mucellit", "mcp"]
    }
  }
}

Exposing MCP Tools

| Tool | Description | |---|---| | mucellit_analyze | Extract document format, metadata, pages, word count, and OCR status | | mucellit_extract_text | Extract text with layout modes (flow, blocks, raw) | | mucellit_chunk | Chunk document for RAG & LLMs (semantic, markdown, page, sliding) | | mucellit_redact_pii | Detect and redact sensitive personal information (TCKN, IBAN, cards, email, phone, VKN) | | mucellit_edit | Edit document text (find & replace), reorder/remove/rotate pages, add watermark, headers, footers, and metadata | | mucellit_preview | Generate vector SVG or HTML page preview | | mucellit_convert | Convert documents between formats (PDF, DOCX, XLSX, HTML, Markdown, CSV, JSON) | | mucellit_search | BM25 full-text keyword search across pages with highlighted snippets | | mucellit_diff | Compare two documents and return similarity score and line changes | | mucellit_ocr | Recognize text in scanned documents or images via Tesseract | | mucellit_capabilities | Inspect runtime engines and available formats |

📊 Benchmarks

Run benchmarks locally:

npm run bench

Results on Apple Silicon (Node.js v25): | Format | Speed | Throughput | Average Latency | |---|---|---|---| | Text (Plain) | 4,300 docs/sec | 0.64 MB/sec | 0.23 ms/doc | | Markdown (AST) | 3,580 docs/sec | 1.49 MB/sec | 0.28 ms/doc | | PPTX (Slides) | 2,190 docs/sec | 2.09 MB/sec | 0.46 ms/doc | | XLSX (Sheets) | 1,660 docs/sec | 1.48 MB/sec | 0.60 ms/doc | | DOCX (OOXML) | 950 docs/sec | 0.93 MB/sec | 1.05 ms/doc | | PDF (PDF.js) | 100 docs/sec | 0.10 MB/sec | 9.80 ms/doc |

📜 License

MIT