DeepPipe

DeepPipe — Document Ingestion & AI Search Pipeline · Copyright © Jinan Kordab 2026

A pure-TypeScript document ingestion, full-text search, and retrieval-augmented chat engine — packaged as a single library. No frontend, no HTTP server: just the engine. You drive it from your own code.

• Ingests PDF, Office (OOXML + legacy CFB), HTML, email (MIME/EML/MHT), ZIP, and plain text using in-house, pure-TypeScript parsers (no pdf.js, tika, or mammoth).
• Indexes into a local SQLite database with FTS5 full-text search and BM25 ranking.
• Builds grounded, cited retrieval contexts for "chat with your documents" (lexical retrieval — no embeddings/vectors).

Install

npm install @kordabjinan/deeppipe

Requires Node.js >= 18. The only native dependency is better-sqlite3, which builds a prebuilt binary on install for most platforms.

Quick start

import { openPipeline } from '@kordabjinan/deeppipe';

// 1. Open (or create) an index. Use ':memory:' for an ephemeral one.
const opened = openPipeline({ location: './data/deeppipe.db' });
if (!opened.ok) throw opened.error;
const pipe = opened.value;

// 2. Ingest a file from disk...
const ingested = await pipe.ingestFile('./contract.pdf');
if (ingested.ok) {
  console.log('Indexed doc', ingested.value.documentId, ingested.value.wordCount, 'words');
}

// ...or ingest raw bytes you already have in memory.
import { readFile } from 'node:fs/promises';
const bytes = await readFile('./report.docx');
pipe.ingestBytes(bytes, 'report.docx');

// 3. Search (BM25-ranked, with optional snippet highlighting).
const found = pipe.search('payment terms', { limit: 10, snippets: true });
if (found.ok) {
  for (const hit of found.value.hits) {
    console.log(hit.score, hit.source, hit.snippet);
  }
}

// 4. Build a grounded context for an LLM ("chat with your documents").
const ctx = pipe.chatContext('What are the payment terms?');
if (ctx.ok) {
  // ctx.value.sources / ctx.value.messages → send to any OpenAI-compatible model.
}

The Result<T, E> contract

Fallible methods never throw. They return ok(value) or err(error), so you branch on .ok:

const r = pipe.search('invoice');
if (r.ok) {
  use(r.value);
} else {
  console.error(r.error.code, r.error.message);
}

Helpers ok, err, isOk, isErr, mapOk, unwrapOr are exported.

Pipeline API

| Method | Description | |---|---| | openPipeline({ location }) | Open/create an index. location is a file path or ':memory:'. | | ingestBytes(data, source?, options?) | Extract + index in-memory bytes. | | ingestFile(path, options?) | Read a file from disk, then ingest (async). | | ingestBatch(documents, options?) | Ingest many docs; isolates per-doc failures. | | search(query, options?) | BM25 full-text search; supports field:value, "phrases", prefix*, AND/OR/NOT. | | chatContext(question, options?) | Retrieve grounded passages + messages for RAG. | | listDocuments(limit?, offset?) | List indexed documents. | | getDocument(id) / documentText(id) | Fetch a document's metadata / reconstructed text. | | removeDocument(id) | Delete a document and its chunks from the index. |

Lower-level building blocks

The barrel also exports the pieces under the Pipeline, for finer control:

• Extraction: extractText, filterHtml, filterMime, detectFormat
• Containers: CompoundFile / readCompoundFile, ZipArchive / readZipArchive
• Text: decodeText, detectCharset, detectLanguage, breakWords, tokenize
• Search: IndexStore / openIndexStore, writeDocument, parseQuery, planQuery, executeQuery
• RAG: buildChatContext, extractiveAnswer, chatSystemPrompt
• Errors: DeepPipeError and subclasses (FormatError, EncodingError, QueryError, IOError, …)

See src/index.ts for the complete exported surface.

Build from source

npm install
npm run build      # compiles src/ → dist/
npm run typecheck  # type-check only

License

MIT © Jinan Kordab 2026