easy-rag-pipeline

A lightweight, zero-dependency TypeScript library for text chunking, embedding generation, in-memory vector similarity retrieval, and RAG (Retrieval-Augmented Generation) prompt context building.

Designed for modern Node.js, Next.js, Bun, and browser applications. Framework-agnostic and provider-agnostic.

Features

• Recursive Text Chunking: Smart boundary-aware text splitting (\n\n, . , etc.) with customizable chunk size & overlap.
• Vector Similarity Search: Built-in cosine similarity calculation and in-memory vector index.
• Provider Agnostic: Works seamlessly with Vercel AI SDK, OpenAI, OpenRouter, Cohere, HuggingFace, or any custom embedding model.
• Prompt Context Builder: Formats retrieved document chunks directly into clean, structured context strings for LLM prompts.
• Fully Typed: Written in 100% TypeScript with complete type definitions included (.d.ts).
• Zero Runtime Dependencies: Ultra-lightweight and fast.

Installation

npm install easy-rag-pipeline
# or
bun add easy-rag-pipeline
# or
pnpm add easy-rag-pipeline

Quickstart

1. Simple Usage with Vercel AI SDK / OpenAI

import { EasyRAGPipeline } from 'easy-rag-pipeline';
import { embedMany, embed } from 'ai';
import { openai } from '@ai-sdk/openai';

// 1. Initialize pipeline with your preferred embedding model
const rag = new EasyRAGPipeline({
  embedder: async (texts: string[]) => {
    const { embeddings } = await embedMany({
      model: openai.embedding('text-embedding-3-small'),
      values: texts,
    });
    return embeddings;
  },
  chunkOptions: {
    chunkSize: 500,
    chunkOverlap: 100,
  },
});

// 2. Ingest documents (CVs, articles, FAQs, docs)
await rag.ingest({
  id: 'resume_01',
  text: `John Doe is a Senior Software Engineer with 5 years experience in React, Next.js, and RAG architectures.`,
  metadata: { author: 'John Doe' },
});

// 3. Search relevant chunks by query
const results = await rag.retrieve('What framework does John use?', {
  limit: 3,
  minSimilarity: 0.2,
});

console.log(results[0].chunk.text);
// Output: "John Doe is a Senior Software Engineer with 5 years experience in React..."

// 4. Build ready-to-use context string for LLM prompts
const promptContext = await rag.buildContext('What framework does John use?');
console.log(promptContext);
/*
Retrieved Context:

[Chunk #1]:
John Doe is a Senior Software Engineer with 5 years experience in React, Next.js, and RAG architectures.
*/

API Reference

EasyRAGPipeline

const rag = new EasyRAGPipeline({
  embedder: (texts: string[]) => Promise<number[][]>,
  chunkOptions?: {
    chunkSize?: number;    // Default: 1000
    chunkOverlap?: number; // Default: 200
    separators?: string[]; // Natural boundary separators
  }
});

Methods

• ingest(doc: string | DocumentInput): Promise<DocumentChunk[]> Splits text into chunks, generates vector embeddings, and indexes them.
• ingestBatch(docs: (string | DocumentInput)[]): Promise<DocumentChunk[]> Ingests multiple documents in batch.
• retrieve(query: string, options?: SearchOptions): Promise<SearchResult[]> Returns top-K matching chunks sorted by cosine similarity.
• buildContext(query: string, options?: RAGContextOptions): Promise<string> Retrieves top matching chunks and formats them into a prompt context block.
• getStore(): InMemoryVectorStore Returns internal vector store instance.

License

MIT © KidiXDev