htRAG

Hypertext-native RAG system built with Effect TS

htRAG combines semantic retrieval (vectors) with relational reasoning (graph traversal) to provide context-aware, explainable answers. Documents are stored as human-readable hypertext pages (anchors) connected by explicit and inferred links.

Features

• 🔗 Hypertext-first: Documents as readable pages with links, not opaque vectors
• 🧠 Hybrid retrieval: Combines vector search + graph traversal + entity matching
• 🎯 Neural reranking: Optional cross-encoder models for improved accuracy (Cohere, Jina, Voyage)
• 🏠 Local embeddings: Run models locally via Ollama (QWEN, Nomic, etc.) for privacy
• 📊 Explainable: Every answer comes with path provenance
• ⚡ Platform-agnostic: Runs on Cloudflare Workers, Node.js, Deno, Bun
• 🔧 Effect TS: Built with Effect for composable, type-safe operations
• 🔌 Multi-provider: OpenAI, Google, Ollama for embeddings and LLMs

Quick Start

Installation

bun install htrag

Basic Usage

import { Effect, Layer } from "effect";
import {
  query,
  createNodeId,
  createNodeUrl,
  Storage,
  VectorSearch,
  Embeddings,
  InMemoryStorageLayer,
  InMemoryVectorSearchLayer,
} from "htrag";
import type { Node } from "htrag";

// Create a node
const node: Node = {
  id: createNodeId("doc1"),
  url: createNodeUrl("/docs/getting-started"),
  title: "Getting Started",
  content: "This is a guide to getting started with htRAG...",
  embeddings: [0.1, 0.2, 0.3, ...], // Your embedding
  entities: { topics: ["rag", "hypertext"] },
  linksOut: [],
  linksIn: [],
  typedEdges: [],
  meta: {
    source: "docs/getting-started.md",
    updated: new Date(),
  },
};

// Set up services
const program = Effect.gen(function* () {
  const storage = yield* Storage;
  
  // Store the node
  yield* storage.putNode(node);
  
  // Query
  const results = yield* query("How do I get started?");
  
  console.log(results);
});

// Run with in-memory services (for development)
const runnable = Effect.provide(
  program,
  Layer.mergeAll(InMemoryStorageLayer, InMemoryVectorSearchLayer)
);

Effect.runPromise(runnable);

Platform Adapters

Cloudflare Workers (with Drizzle ORM)

import { makeCloudflareLayer } from "htrag/cloudflare";
import { query } from "htrag";
import { Effect } from "effect";

export default {
  async fetch(request: Request, env: Env) {
    const program = query("What are the payment terms for PO-113?");
    
    const layer = makeCloudflareLayer(env);
    const runnable = Effect.provide(program, layer);
    
    const results = await Effect.runPromise(runnable);
    
    return Response.json(results);
  },
};

Node.js / Bun

import { NodeLayer } from "htrag/node";
import { retrieve } from "htrag";

const program = retrieve({
  text: "Find invoices for Project HQ-Refit",
  entities: { project: ["HQ-Refit"] },
});

const runnable = Effect.provide(program, NodeLayer);
const results = await Effect.runPromise(runnable);

Architecture

htRAG implements a 6-step retrieval pipeline:

1. Seed Recall: Vector search finds top-k similar nodes
2. Expand: Graph traversal explores connected nodes (beam search)
3. Re-rank: Hybrid scoring combines multiple signals
4. Assemble: Select coherent context along paths
5. Cite: Return results with transparent provenance
6. Learn: Store usage patterns for future queries

Hybrid Scoring Formula

score = α·cosine + β·edge_proximity + γ·authority + δ·entity_overlap + ε·recency

Where:

• α (0.3): Semantic similarity weight
• β (0.25): Graph distance weight
• γ (0.15): Authority/PageRank weight
• δ (0.2): Entity overlap weight
• ε (0.1): Recency weight

Core Concepts

Nodes

Atomic units of knowledge - a paragraph, table, or figure with:

• Content (100-500 words)
• Vector embedding
• Extracted entities
• Links to other nodes
• Metadata (source, version, timestamp)

Edges

Connections between nodes:

• Explicit: Human-authored hyperlinks
• Inferred: LLM-extracted relationships (e.g., is_about:ProjectX)
• Usage: Learned from query patterns

Retrieval

Multi-hop reasoning through the graph:

const results = await retrieve({
  text: "Who approved PO-113?",
  sessionContext: {
    organization: "Acme Corp",
    channel: "work",
  },
});

// Returns nodes with path provenance
results.forEach(result => {
  console.log(result.node.title);
  console.log(result.path.nodes); // Path through graph
  console.log(result.explanation); // Why this was selected
});

Development

# Install dependencies
bun install

# Type check
bun run typecheck

# Build
bun run build

# Test
bun test

# Format
bun run format

# Drizzle commands
bun run db:generate  # Generate migrations from schema
bun run db:push      # Push schema directly (dev only)
bun run db:studio    # Open Drizzle Studio (GUI)

Cloudflare Setup

1. Create a Cloudflare Worker project:

bun create cloudflare my-htrag-worker
cd my-htrag-worker
bun add htrag

2. Set up wrangler.jsonc:

{
  "name": "htrag-worker",
  "main": "src/index.ts",
  "compatibility_date": "2024-01-01",
  "d1_databases": [
    {
      "binding": "DB",
      "database_name": "htrag",
      "database_id": "your-database-id"
    }
  ],
  "vectorize": [
    {
      "binding": "VECTORIZE",
      "index_name": "htrag-embeddings"
    }
  ],
  "ai": {
    "binding": "AI"
  }
}

3. Generate and apply Drizzle migrations:

# Generate migration from schema
bun run db:generate

# Apply to D1
wrangler d1 migrations apply htrag --local  # for local testing
wrangler d1 migrations apply htrag          # for production

4. Deploy:

bun run deploy

Use Cases

Document Q&A

const results = await query(
  "What are the payment terms for PO-113?"
);

Temporal Queries

const results = await retrieve({
  text: "Show invoices approved after the contract addendum",
  sessionContext: {
    timeframe: {
      start: new Date("2023-01-01"),
      end: new Date("2023-12-31"),
    },
  },
});

Entity-based Search

const results = await retrieve({
  text: "Latest vendor certificate",
  entities: {
    project: ["HQ-Refit"],
    vendor: ["Acme Ltd"],
  },
});

Comparison to Other Approaches

| Feature | htRAG | Traditional RAG | GraphRAG | |---------|-------|-----------------|----------| | Document structure | Hypertext (readable) | Chunks | Knowledge graph | | Retrieval | Hybrid (vector + graph) | Vector only | Graph only | | Multi-hop | ✅ Yes | ❌ No | ✅ Yes | | Human editable | ✅ Yes | ❌ No | ⚠️ Limited | | Provenance | ✅ Full paths | ⚠️ Chunks | ⚠️ Triples | | Compliance-ready | ✅ Yes | ❌ No | ⚠️ Limited |

Documentation

• QUICKSTART.md - Get started in 5 minutes
• CONFIGURATION.md - Configure providers and models
• RERANKING.md - Neural reranking guide
• OLLAMA.md - Local embeddings with Ollama
• SQLITE_EXAMPLES.md - SQLite usage examples
• DESIGN.md - Architecture and design decisions
• INGESTION.md - Document ingestion pipeline
• SCORING.md - Hybrid scoring explained

License

MIT

Contributing

Contributions welcome! See DESIGN.md for architecture details.