@msbayindir/rag-chunker

PDF → Mistral OCR → Deterministic AST Chunker → Contextual RAG

A production-ready pipeline that turns PDFs into retrieval-optimized chunks. Uses Mistral OCR for accurate text extraction, a deterministic AST-based chunker that respects document structure, and optionally enriches each chunk with a context summary following Anthropic's Contextual Retrieval method — reducing retrieval failures by up to 49%.

Highlights

• Mistral OCR 3 as primary OCR provider; Gemini Vision as automatic fallback
• Deterministic AST chunker powered by remark/mdast — no LLM required for chunking, fast and reproducible
• Anthropic Contextual Retrieval — optional batch or per-chunk context enrichment via Gemini
• Heading normalization — two-phase Gemini pipeline fixes inconsistent heading levels from OCR
• Large PDF auto-split — PDFs over 50 MB are automatically split into 40 MB batches and merged seamlessly
• OCR cache — results cached locally (7-day TTL) so the same PDF is never re-processed
• CLI + programmatic API — use as a command-line tool or import into your own pipeline

Installation

npm install @msbayindir/rag-chunker

@google/genai is a required peer dependency and is installed automatically. If you plan to use OpenAI embeddings:

npm install openai

API Keys — Free to Get Started

You can process hundreds of documents without spending a cent.

Mistral AI — Primary OCR

1. Go to console.mistral.ai → Sign Up → API Keys
2. New accounts receive $5 free credit (no credit card required on sign-up)
3. Mistral OCR pricing: ~$1 per 1,000 pages → $5 gets you ~5,000 pages
4. More than enough to evaluate and prototype

If you only have a Mistral key, OCR works. Context enrichment and heading normalization require a Gemini key.

Google Gemini — Context Enrichment, Heading Normalization, Fallback OCR

1. Go to aistudio.google.com → Get API Key
2. Completely free on the Google AI Studio free tier — no credit card required
3. Free tier limits:
   • gemini-2.0-flash (context default): 1,500 requests/day
   • gemini-2.5-flash (context alternative): 500 requests/day
   • gemini-2.5-pro (heading normalization phase 1): 50 requests/day

For most documents, context enrichment uses ~20–30 batch requests. You can process 50+ documents per day on the free tier.

Quick Start

# Basic — OCR + chunk, no context enrichment
npx rag-chunker process document.pdf -m YOUR_MISTRAL_KEY -o ./output

# With Anthropic-style context enrichment (recommended for RAG)
npx rag-chunker process document.pdf \
  -m YOUR_MISTRAL_KEY \
  -k YOUR_GEMINI_KEY \
  --context-mode batch \
  -o ./output

# With heading normalization (useful when OCR produces inconsistent heading levels)
npx rag-chunker process document.pdf \
  -m YOUR_MISTRAL_KEY \
  -k YOUR_GEMINI_KEY \
  --heading-normalization \
  -o ./output

# Using environment variables (recommended)
export MISTRAL_API_KEY=your_key
export GEMINI_API_KEY=your_key
npx rag-chunker process document.pdf -o ./output

After processing, ./output/ will contain:

output/
  chunks.jsonl      ← one chunk per line, ready for embedding
  document.md       ← full markdown with page markers
  structure.json    ← headings, tables, page count
  manifest.json     ← processing stats and metadata

CLI Reference

rag-chunker process <pdf>

Full pipeline: OCR → chunk → optional context enrichment → save.

rag-chunker process <pdf> [options]

| Option | Default | Description | |--------|---------|-------------| | -o, --output <dir> | — | Output directory. If omitted, results are not saved to disk. | | -m, --mistral-api-key <key> | MISTRAL_API_KEY env | Mistral API key — primary OCR provider | | -k, --gemini-api-key <key> | GEMINI_API_KEY env | Gemini API key — context enrichment, heading fix, fallback OCR | | --context-mode <mode> | none | none | batch | per-chunk | | --context-model <model> | gemini-2.0-flash | Gemini model for context summaries | | --context-batch-size <n> | 10 | Chunks per batch in batch mode | | --max-chunk-tokens <n> | 512 | Max tokens per chunk | | --min-chunk-tokens <n> | 50 | Min tokens for a chunk to be emitted | | --overlap-tokens <n> | 0 | Tokens prepended from the previous chunk | | --no-preserve-tables | — | Do not keep tables in their own chunk | | --no-preserve-code | — | Do not keep code blocks in their own chunk | | --heading-normalization | — | Fix inconsistent heading levels (requires --gemini-api-key) | | --ocr-cache-path <path> | ~/.rag-chunker/ocr-cache.json | Custom OCR cache file path | | --ocr-cache-ttl <days> | 7 | OCR cache TTL in days | | --no-ocr-cache | — | Disable OCR caching | | --warn-large-chunk <n> | 2000 | Warn when a table/code chunk exceeds N tokens | | --verbose | — | Show verbose pipeline logs |

Large PDF handling: PDFs over 50 MB automatically trigger a confirmation prompt. If confirmed, the file is split into 40 MB batches, each batch is OCR'd sequentially, and results are merged — page numbers and heading hierarchy are preserved across the split.

rag-chunker ocr <pdf>

Debug command. Runs OCR and prints each page's markdown to stdout.

rag-chunker ocr document.pdf -m YOUR_MISTRAL_KEY

| Option | Description | |--------|-------------| | -m, --mistral-api-key <key> | Mistral API key | | -k, --gemini-api-key <key> | Gemini API key (for Vision fallback) |

rag-chunker chunk <md>

Debug command. Runs the AST chunker on a .md file and prints chunk boundaries. No API key needed.

rag-chunker chunk document.md --max-tokens 512

| Option | Default | Description | |--------|---------|-------------| | --max-tokens <n> | 512 | Max tokens per chunk | | --min-tokens <n> | 50 | Min tokens per chunk | | --overlap-tokens <n> | 0 | Overlap tokens from previous chunk | | --no-preserve-tables | — | Do not isolate tables | | --no-preserve-code | — | Do not isolate code blocks |

rag-chunker inspect <output-dir>

Reads an output directory and prints a summary of its manifest and structure.

rag-chunker inspect ./output

rag-chunker cache list / clear

Manage the local OCR cache.

rag-chunker cache list
rag-chunker cache clear --expired    # remove entries older than TTL
rag-chunker cache clear --all        # wipe entire cache

| Option | Description | |--------|-------------| | --cache-path <path> | Custom cache file path | | --ttl <days> | TTL for expired check (default: 7) |

Programmatic API

process(pdfInput, config)

Full pipeline. Returns a ProcessResult with chunks, markdown, structure, manifest, and a save() method.

import { process } from '@msbayindir/rag-chunker'

const result = await process('document.pdf', {
  mistralApiKey: process.env.MISTRAL_API_KEY,
  geminiApiKey: process.env.GEMINI_API_KEY,
  contextMode: 'batch',
  maxChunkTokens: 512,
})

// Save all output files
await result.save('./output')

// Or work with chunks directly
for (const chunk of result.chunks) {
  console.log(chunk.content)     // embed this
  console.log(chunk.sectionPath) // breadcrumb path
  console.log(chunk.pageNumber)
}

chunk(pdfInput, config)

Convenience wrapper. Forces contextMode: 'none' and returns only Chunk[].

import { chunk } from '@msbayindir/rag-chunker'

const chunks = await chunk('document.pdf', {
  mistralApiKey: process.env.MISTRAL_API_KEY,
  maxChunkTokens: 512,
})

Full ChunkerConfig Reference

| Field | Type | Default | Description | |-------|------|---------|-------------| | mistralApiKey | string | — | Mistral API key — primary OCR | | geminiApiKey | string | — | Gemini API key — context, heading fix, fallback OCR | | contextMode | 'none' \| 'batch' \| 'per-chunk' | 'none' | Context enrichment mode | | contextModel | string | 'gemini-2.0-flash' | Gemini model for context summaries | | contextBatchSize | number | 10 | Chunks per Gemini batch call | | contextConcurrency | number | 2 | Max concurrent calls in per-chunk mode | | maxChunkTokens | number | 512 | Max tokens per chunk | | minChunkTokens | number | 50 | Min tokens per chunk | | overlapTokens | number | 0 | Overlap tokens prepended from previous chunk | | preserveTables | boolean | true | Keep tables in their own chunk | | preserveCodeBlocks | boolean | true | Keep code blocks in their own chunk | | ocrCachePath | string \| false | ~/.rag-chunker/ocr-cache.json | Cache path. false to disable. | | ocrCacheTtlDays | number | 7 | OCR cache TTL in days | | headingNormalization | boolean | false | Fix OCR heading levels via Gemini | | headingFixPhase1Model | string | 'gemini-2.5-pro' | Model for structure discovery phase | | headingFixPhase2Model | string | 'gemini-2.5-flash-preview-05-20' | Model for correction phase | | warnLargeChunkTokens | number | 2000 | Warn threshold for oversized preserved chunks | | embeddingProvider | IEmbeddingProvider | — | Embedding provider (optional) | | logger | ILogger | pino at INFO | Custom logger |

Embedding Providers

Built-in providers can optionally generate embeddings inline during the pipeline.

import {
  createGeminiEmbeddingProvider,
  createOpenAiEmbeddingProvider,
  createNullEmbeddingProvider,
} from '@msbayindir/rag-chunker'

// Gemini — 1536 dimensions
const geminiProvider = createGeminiEmbeddingProvider({
  apiKey: process.env.GEMINI_API_KEY!,
})

// OpenAI text-embedding-3-large — 3072 dimensions (requires: npm install openai)
const openaiProvider = createOpenAiEmbeddingProvider({
  apiKey: process.env.OPENAI_API_KEY!,
})

// Null — returns empty vectors, useful for testing
const nullProvider = createNullEmbeddingProvider()

const result = await process('document.pdf', {
  mistralApiKey: process.env.MISTRAL_API_KEY,
  embeddingProvider: openaiProvider,
})
// result.chunks[0].embedding → number[]

Note: Embedding is non-fatal. If the provider throws, the chunk is still returned with embedding: [].

Output Files

chunks.jsonl

One JSON object per line. Each line is a Chunk:

{
  "chunkId": "a3f7c2d1...",        // SHA-256(rawContent), first 32 hex chars — deterministic
  "index": 0,                       // 0-based position in chunk array
  "content": "Context: This section covers sodium metabolism in the electrolyte chapter.\n\n## Sodium Metabolism\n\nSodium is...",
  "rawContent": "## Sodium Metabolism\n\nSodium is...",   // pure markdown, no context
  "contextSummary": "This section covers sodium metabolism in the electrolyte chapter.",
  "tokenCount": 412,
  "contentType": "text",            // "text" | "table" | "code" | "mixed"
  "sectionPath": ["Electrolyte Disorders", "Sodium Metabolism"],
  "pageNumber": 14,                 // 1-based, from OCR page markers
  "prevChunkId": "9f1a...",
  "nextChunkId": "c82b...",
  "mustPreserve": false,            // true for table/code chunks that can't be split
  "embedding": []                   // number[] if embeddingProvider was configured
}

document.md

Full document markdown with page markers:

<!-- page 1 -->

# Chapter 1

Content...

<!-- page 2 -->

## Section 1.1

structure.json

Document structure extracted from markdown:

{
  "headings": [
    { "level": 1, "text": "Chapter 1", "pageNumber": 1, "markdownLine": 3 }
  ],
  "tables": [
    { "index": 0, "caption": "Table 1. Electrolyte values", "pageNumber": 5, "rowCount": 8, "columnCount": 3 }
  ],
  "tableCount": 12,
  "codeBlockCount": 0,
  "pageCount": 192,
  "totalTokens": 68432
}

manifest.json

Processing metadata:

{
  "version": "3.0",
  "processedAt": "2026-03-16T00:20:51.118Z",
  "pdfHash": "0f2860b4...",
  "ocrModel": "mistral-ocr-latest",
  "contextModel": "gemini-2.0-flash",
  "contextMode": "batch",
  "chunkStats": {
    "total": 183, "avgTokens": 378,
    "minTokens": 50, "maxTokens": 2662,
    "tableChunks": 25, "codeChunks": 0, "textChunks": 158, "mixedChunks": 0
  },
  "durationMs": 146689,
  "ocrCacheHit": true,
  "headingFix": null,               // populated if --heading-normalization was used
  "contextEnrichment": {
    "model": "gemini-2.0-flash",
    "chunksEnriched": 176,
    "chunksSkipped": 7,             // low-quality OCR artifacts skipped
    "batchCalls": 19,
    "durationMs": 146005,
    "cacheUsed": true               // whether Gemini CachedContent was used
  }
}

Contextual Retrieval — What to Embed and Why

The Problem with Naive Chunking

When you split a document into chunks and embed them independently, each chunk loses its context. A chunk containing "It was founded in 1987 and has since expanded to 42 countries" provides no signal for a query about a specific organization — the surrounding context that identifies the subject is gone.

This is the core retrieval failure in most RAG systems.

Anthropic's Solution — and the Numbers

In September 2024, Anthropic published Contextual Retrieval, showing that prepending a short, situating summary to each chunk before embedding significantly improves retrieval:

| Method | Retrieval Failure Reduction | |--------|-----------------------------| | Basic semantic embedding | baseline | | Contextual embedding | 49% fewer failures | | Contextual embedding + BM25 hybrid | 67% fewer failures |

The approach: for each chunk, generate a 1–2 sentence summary that situates it within the document — what section it belongs to, what topic it covers. Prepend this to the chunk content before embedding.

How This Package Implements It

When --context-mode batch (or per-chunk) is used:

1. The full document markdown is sent to Gemini as context
2. For each chunk, a 1–2 sentence summary is generated in the same language as the document
3. The contextSummary field is stored separately
4. content is assembled as: "Context: <summary>\n\n<rawContent>"

This matches Anthropic's exact format.

Which Field Should You Embed?

| Your setup | Embed this field | |-----------|-----------------| | --context-mode batch or per-chunk (recommended) | content — includes the context summary | | --context-mode none (default) | content or rawContent — they are identical | | Hybrid search: vector + BM25/keyword | content for vector index, rawContent for keyword index |

// Reading chunks.jsonl and embedding
import { createReadStream } from 'fs'
import { createInterface } from 'readline'

const rl = createInterface({ input: createReadStream('./output/chunks.jsonl') })
for await (const line of rl) {
  const chunk = JSON.parse(line)
  const textToEmbed = chunk.content  // always use content
  // → send to your vector database
}

Should You Strip Markdown Before Embedding?

Modern embedding models (OpenAI text-embedding-3-large, Gemini embedding-001, Cohere embed-v3) are trained on web-scale data that includes markdown. They handle #, **, | and similar syntax gracefully — stripping is generally not necessary.

Research findings:

• Nussbaum et al. (2024) — Nomic Embed: Training a Reproducible Long Context Text Embedder — shows retrieval quality is robust to markdown formatting in general-purpose embedders
• Muennighoff et al. (2022) — MTEB: Massive Text Embedding Benchmark — the benchmark includes mixed-format text; top models perform well without preprocessing
• For table-heavy content, stripping | separators can marginally improve semantic similarity scores; for prose, the effect is negligible

This package does not strip markdown. If your downstream embedding model or use case requires clean text, strip it yourself before embedding — this is intentionally left to the caller:

// Optional: strip markdown before embedding (your responsibility)
function stripMarkdown(text: string): string {
  return text
    .replace(/#{1,6}\s+/g, '')          // headings
    .replace(/\*\*([^*]+)\*\*/g, '$1')  // bold
    .replace(/_([^_]+)_/g, '$1')        // italic
    .replace(/`([^`]+)`/g, '$1')        // inline code
    .replace(/\|/g, ' ')                // table separators
    .replace(/\s+/g, ' ').trim()
}

const textToEmbed = stripMarkdown(chunk.content)

Advanced Usage

Custom Logger

Replace the default pino logger with your own:

import { process } from '@msbayindir/rag-chunker'

const result = await process('document.pdf', {
  mistralApiKey: process.env.MISTRAL_API_KEY,
  logger: {
    debug: (msg, meta) => console.debug('[rag]', msg, meta),
    info:  (msg, meta) => console.info('[rag]', msg, meta),
    warn:  (msg, meta) => console.warn('[rag]', msg, meta),
    error: (msg, meta) => console.error('[rag]', msg, meta),
  }
})

Disable or Customize OCR Cache

// Disable caching entirely
const result = await process('document.pdf', {
  mistralApiKey: process.env.MISTRAL_API_KEY,
  ocrCachePath: false,
})

// Use a project-local cache instead of ~/.rag-chunker/
const result = await process('document.pdf', {
  mistralApiKey: process.env.MISTRAL_API_KEY,
  ocrCachePath: './.rag-cache/ocr.json',
  ocrCacheTtlDays: 30,
})

Heading Normalization

OCR pipelines often produce inconsistent heading levels — a section title becomes # Title on one page and ## Title on another, or numbered sections (1.1, 1.2) get assigned the wrong level.

The two-phase normalization process:

1. Phase 1 (Gemini Pro): Sends the full document to discover structure — document type, main sections, numbering patterns
2. Phase 2 (Gemini Flash): Sends the discovered structure + heading list (not the full document) and corrects each heading's level

Enable it when:

• The source PDF has complex, multi-level structure (textbooks, technical reports)
• OCR produces mismatched heading levels
• You need accurate sectionPath breadcrumbs in your chunks

rag-chunker process document.pdf \
  -m YOUR_MISTRAL_KEY \
  -k YOUR_GEMINI_KEY \
  --heading-normalization \
  -o ./output

Phase 1 uses gemini-2.5-pro (50 requests/day on free tier). For a 200-page document, this is 1 request. You can process up to 50 documents per day with heading normalization on the free tier.

Process a Buffer Instead of a File Path

import { readFileSync } from 'fs'
import { process } from '@msbayindir/rag-chunker'

const pdfBuffer = readFileSync('document.pdf')
const result = await process(pdfBuffer, {
  mistralApiKey: process.env.MISTRAL_API_KEY,
})

Cost Reference

| Setup | Approx. cost per 200-page document | |-------|-------------------------------------| | OCR only (Mistral) | ~$0.05 | | OCR + context enrichment (batch) | ~$0.15–0.22 | | OCR + heading normalization | ~$0.06 | | OCR + context + heading normalization | ~$0.23 | | v2 equivalent (LLM-based OCR + chunking) | ~$0.95 |

Requirements

• Node.js ≥ 20
• At least one of: MISTRAL_API_KEY or GEMINI_API_KEY

License

MIT