@operor/knowledge

Knowledge base system with vector search, retrieval pipeline, and document ingestors for Operor.

Overview

@operor/knowledge provides a complete knowledge base solution for AI agents, enabling them to retrieve relevant context from documents, FAQs, and web content. The system uses vector embeddings for semantic search and includes specialized pipelines for ingestion and retrieval.

Inspired by Astra Agent KB design and OpenClaw's extensible plugin architecture, this package provides a modular, production-ready knowledge base with pluggable ingestors.

Quick Start

Installation

pnpm add @operor/knowledge

Basic Usage

import {
  SQLiteKnowledgeStore,
  EmbeddingService,
  TextChunker,
  IngestionPipeline,
  RetrievalPipeline,
} from '@operor/knowledge';

// 1. Initialize components
const store = new SQLiteKnowledgeStore('./knowledge.db', 1536);
await store.initialize();

const embedder = new EmbeddingService({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  model: 'text-embedding-3-small',
});

const chunker = new TextChunker({ chunkSize: 500, chunkOverlap: 50 });
const ingestion = new IngestionPipeline(store, embedder, chunker);
const retrieval = new RetrievalPipeline(store, embedder, 0.85);

// 2. Ingest documents
await ingestion.ingest({
  sourceType: 'url',
  content: 'Your document content here...',
  title: 'Getting Started Guide',
});

// 3. Query the knowledge base
const result = await retrieval.retrieve('How do I get started?', { limit: 5 });
console.log(result.context); // Formatted context for LLM injection
console.log(result.isFaqMatch); // true if FAQ fast-path matched

Architecture

Core Components

• KnowledgeStore: SQLite-backed storage with sqlite-vec for vector search
• EmbeddingService: Multi-provider embeddings via Vercel AI SDK (OpenAI, Google, Mistral, Cohere, Ollama)
• TextChunker: Document chunking using LangChain text splitters
• IngestionPipeline: Orchestrates document ingestion (chunk → embed → store)
• RetrievalPipeline: Query-time retrieval with FAQ fast-path optimization

Ingestors (Extensible)

Following OpenClaw's plugin pattern, ingestors are modular and extensible:

• UrlIngestor: Web crawling with Readability extraction
• FileIngestor: PDF, DOCX, XLSX, CSV, TXT, MD, HTML parsing
• WatiFaqSync: FAQ extraction from WATI conversations via LLM

Technology Stack

All dependencies use the latest stable versions:

• Vector Search: sqlite-vec (0.1.7-alpha.2) — Fast vector search in SQLite
• Embeddings: ai (6.x) + @ai-sdk/* (3.x) — Vercel AI SDK with multi-provider support
• Text Splitting: @langchain/textsplitters (1.x) — Recursive and markdown splitters
• Document Parsing:
  • PDF: unpdf (1.4.0)
  • DOCX: mammoth (1.11.0)
  • XLSX: xlsx (0.18.5)
  • HTML: @mozilla/readability (0.6.0) + linkedom (0.18.12)
• Storage: better-sqlite3 (12.x) — Fast, synchronous SQLite

Embedding Providers

The EmbeddingService supports multiple providers via the Vercel AI SDK:

| Provider | Model (default) | Dimensions | API Key Required | |----------|----------------|------------|------------------| | OpenAI | text-embedding-3-small | 1536 | Yes | | Google | text-embedding-004 | 768 | Yes | | Mistral | mistral-embed | 1024 | Yes | | Cohere | embed-english-v3.0 | 1024 | Yes | | Ollama | nomic-embed-text | 768 | No (local) |

Important: When switching embedding providers, you must re-ingest all documents. The vector dimensions must match what the store was initialized with, or search will fail.

Provider Examples

// OpenAI (default)
const embedder = new EmbeddingService({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  model: 'text-embedding-3-small', // optional
  dimensions: 1536, // optional, defaults per provider
});

// Google Gemini
const embedder = new EmbeddingService({
  provider: 'google',
  apiKey: process.env.GOOGLE_API_KEY,
  model: 'text-embedding-004',
});

// Mistral
const embedder = new EmbeddingService({
  provider: 'mistral',
  apiKey: process.env.MISTRAL_API_KEY,
});

// Cohere
const embedder = new EmbeddingService({
  provider: 'cohere',
  apiKey: process.env.COHERE_API_KEY,
});

// Ollama (local, no API key)
const embedder = new EmbeddingService({
  provider: 'ollama',
  model: 'nomic-embed-text',
  baseURL: 'http://localhost:11434/v1', // optional
});

Usage Guide

Ingesting Documents

From URL

import { UrlIngestor } from '@operor/knowledge';

const urlIngestor = new UrlIngestor(ingestion);

// Single URL
await urlIngestor.ingestUrl('https://docs.example.com/guide');

// Sitemap (batch ingest)
await urlIngestor.ingestSitemap('https://example.com/sitemap.xml', {
  maxPages: 50,
});

// Crawl website (BFS traversal)
await urlIngestor.crawl('https://docs.example.com', {
  maxPages: 20,
  maxDepth: 2,
});

From File

import { FileIngestor } from '@operor/knowledge';

const fileIngestor = new FileIngestor(ingestion);

// Supports: PDF, DOCX, XLSX, CSV, TXT, MD, HTML
await fileIngestor.ingestFile('./docs/manual.pdf', 'User Manual');
await fileIngestor.ingestFile('./data/products.xlsx', 'Product Catalog');

FAQ Entries

// Manual FAQ entry
await ingestion.ingestFaq(
  'What is the return policy?',
  'You can return items within 30 days of purchase.'
);

// Batch FAQ sync from WATI conversations
import { WatiFaqSync } from '@operor/knowledge';

const faqSync = new WatiFaqSync(ingestion, async (conversation) => {
  // Your LLM extraction logic here
  return [
    { question: 'How do I reset my password?', answer: 'Click Forgot Password...' },
  ];
});

await faqSync.syncFromConversations(conversations, {
  minAnswerLength: 20,
  maxPairs: 100,
});

Retrieving Context

// Basic retrieval
const result = await retrieval.retrieve('What is the return policy?', {
  limit: 5,
  scoreThreshold: 0.7,
});

console.log(result.isFaqMatch); // true if FAQ fast-path matched
console.log(result.context); // Formatted context for LLM injection
console.log(result.results); // Raw search results with scores

// Filter by source type
const result = await retrieval.retrieve('pricing info', {
  limit: 3,
  sourceTypes: ['url', 'file'], // exclude FAQs
});

FAQ Fast-Path

The retrieval pipeline includes an optimization for FAQs:

1. First searches FAQ documents only
2. If score ≥ 0.85 (configurable), returns immediately (fast-path)
3. Otherwise, searches full knowledge base

This ensures instant responses for common questions while maintaining comprehensive search for complex queries.

// Adjust FAQ threshold (default: 0.85)
const retrieval = new RetrievalPipeline(store, embedder, 0.90);

CLI Usage

The @operor/cli package provides commands for managing the knowledge base:

# Add documents
operor kb add-url https://docs.example.com/guide
operor kb add-file ./manual.pdf
operor kb add-faq "What are your hours?" "We are open 9-5 Mon-Fri"

# Search
operor kb search "return policy" -n 5

# List all documents
operor kb list

# Delete a document
operor kb delete <doc-id>

# Show statistics
operor kb stats

API Reference

EmbeddingService

class EmbeddingService {
  constructor(config: EmbeddingServiceConfig);

  embed(text: string): Promise<number[]>;
  embedMany(texts: string[]): Promise<number[][]>;

  get dimensions(): number;
  get provider(): string;

  static defaultDimensions(provider: string, model?: string): number;
}

interface EmbeddingServiceConfig {
  provider: 'openai' | 'google' | 'mistral' | 'cohere' | 'ollama';
  apiKey?: string;
  model?: string;
  baseURL?: string;
  dimensions?: number;
}

SQLiteKnowledgeStore

class SQLiteKnowledgeStore implements KnowledgeStore {
  constructor(dbPath?: string, dimensions?: number);

  initialize(): Promise<void>;
  close(): Promise<void>;

  addDocument(doc: KBDocument): Promise<void>;
  getDocument(id: string): Promise<KBDocument | null>;
  listDocuments(): Promise<KBDocument[]>;
  deleteDocument(id: string): Promise<void>;

  addChunks(chunks: KBChunk[]): Promise<void>;
  search(query: string, embedding: number[], options?: KBSearchOptions): Promise<KBSearchResult[]>;
  searchByEmbedding(embedding: number[], options?: KBSearchOptions): Promise<KBSearchResult[]>;

  getDimensions(): number;
}

TextChunker

class TextChunker {
  constructor(options?: ChunkOptions);

  chunk(text: string, options?: ChunkOptions): Promise<string[]>;
  chunkMarkdown(markdown: string, options?: ChunkOptions): Promise<string[]>;
}

interface ChunkOptions {
  chunkSize?: number; // default: 500
  chunkOverlap?: number; // default: 50
}

IngestionPipeline

class IngestionPipeline {
  constructor(store: KnowledgeStore, embedder: EmbeddingService, chunker: TextChunker);

  ingest(input: IngestInput): Promise<KBDocument>;
  ingestFaq(question: string, answer: string, metadata?: Record<string, any>): Promise<KBDocument>;
}

interface IngestInput {
  sourceType: 'url' | 'file' | 'faq' | 'annotation';
  content: string;
  title?: string;
  sourceUrl?: string;
  fileName?: string;
  metadata?: Record<string, any>;
}

RetrievalPipeline

class RetrievalPipeline {
  constructor(store: KnowledgeStore, embedder: EmbeddingService, faqThreshold?: number);

  retrieve(query: string, options?: KBSearchOptions): Promise<RetrievalResult>;
}

interface RetrievalResult {
  results: KBSearchResult[];
  context: string; // Formatted for LLM injection
  isFaqMatch: boolean; // true if FAQ fast-path matched
}

Types

interface KBDocument {
  id: string;
  sourceType: 'url' | 'file' | 'faq' | 'annotation';
  sourceUrl?: string;
  fileName?: string;
  title?: string;
  content: string;
  metadata?: Record<string, any>;
  createdAt: number;
  updatedAt: number;
}

interface KBChunk {
  id: string;
  documentId: string;
  content: string;
  chunkIndex: number;
  embedding?: number[];
  metadata?: Record<string, any>;
}

interface KBSearchResult {
  chunk: KBChunk;
  document: KBDocument;
  score: number; // 0-1, higher is better
  distance: number; // raw vector distance
}

interface KBSearchOptions {
  limit?: number;
  scoreThreshold?: number;
  sourceTypes?: ('url' | 'file' | 'faq' | 'annotation')[];
  metadata?: Record<string, any>;
}

Extending with Custom Ingestors

Following OpenClaw's plugin architecture pattern, you can create custom ingestors:

export class CustomIngestor {
  private pipeline: IngestionPipeline;

  constructor(pipeline: IngestionPipeline) {
    this.pipeline = pipeline;
  }

  async ingestCustomSource(source: string): Promise<KBDocument> {
    // 1. Extract content from your source
    const content = await this.extractContent(source);

    // 2. Use pipeline to ingest
    return this.pipeline.ingest({
      sourceType: 'url', // or 'file', 'annotation'
      content,
      title: 'Custom Source',
      metadata: { source: 'custom' },
    });
  }

  private async extractContent(source: string): Promise<string> {
    // Your extraction logic
    return 'extracted content';
  }
}

Troubleshooting

Dimension Mismatch Errors

If you see errors like "Dimension mismatch for inserted vector", it means:

• You switched embedding providers without re-ingesting documents
• The store was initialized with different dimensions than the embedder produces

Solution: Delete the KB database and re-ingest all documents with the new provider.

rm knowledge.db knowledge.db-shm knowledge.db-wal
operor kb add-url https://docs.example.com  # re-ingest

Performance Tips

• Use batch ingestion for multiple documents (sitemap, crawl)
• Adjust chunk size based on your content (smaller for FAQs, larger for docs)
• Set appropriate scoreThreshold to filter low-quality results
• Use sourceTypes filter to narrow search scope

License

MIT