@ruvector/ruvllm

Self-learning LLM runtime for Node.js — GGUF inference, TurboQuant KV-cache compression (6-8x memory savings), SONA adaptive learning, FlashAttention, speculative decoding, and SIMD-optimized kernels. Built in Rust, runs everywhere.

Inference at 88-135 tok/s on M4 Pro | <1ms SONA adaptation | 6-8x KV-cache compression via TurboQuant

Installation

npm install @ruvector/ruvllm

Quick Start

import { RuvLLM, RuvLLMConfig } from '@ruvector/ruvllm';

// Initialize with default configuration
const llm = new RuvLLM();

// Or with custom configuration
const llm = new RuvLLM({
  modelPath: './models/ruvltra-small-q4km.gguf',
  sonaEnabled: true,
  flashAttention: true,
  maxTokens: 256,
});

// Generate text
const response = await llm.query('Explain quantum computing');
console.log(response.text);

// Stream generation
for await (const token of llm.stream('Write a haiku about Rust')) {
  process.stdout.write(token);
}

What's New in v2.5

| Feature | Description | |---------|-------------| | TurboQuant KV-Cache | 2-4 bit asymmetric quantization with per-channel scale/zero-point — 6-8x memory reduction, <0.5% perplexity loss | | TurboQuant Embedding Store | Quantized vector storage with compressed search — 10-30x memory savings | | H2O / PyramidKV Eviction | Intelligent cache eviction policies for long-context inference | | Optimized Inner Product | Asymmetric distance on quantized data — skip decompression for 2-4x faster search | | RuvLTRA Models | Purpose-built 0.5B & 3B models for Claude Flow | | Task-Specific LoRA | 5 pre-trained adapters (coder, researcher, security, architect, reviewer) | | HuggingFace Hub | Download/upload models directly | | Adapter Merging | TIES, DARE, SLERP strategies | | HNSW Routing | 150x faster semantic matching | | Evaluation Harness | SWE-Bench testing with 5 ablation modes | | mistral-rs Backend | Production serving with PagedAttention, X-LoRA, ISQ |

TurboQuant — KV-Cache Compression

Reduce inference memory by 6-8x with <0.5% quality loss:

import { simd } from '@ruvector/ruvllm/simd';

// TurboQuant compresses KV-cache entries at 2-4 bit precision
// with per-channel asymmetric quantization (scale + zero-point).
// Eviction policies (H2O, Sliding Window, PyramidKV) keep the
// most important tokens in cache during long-context generation.

// Supported bit widths: 2-bit (32x), 3-bit (10.7x), 4-bit (8x), 8-bit (4x)

| Bits | Compression | Perplexity Loss | Use Case | |------|-------------|-----------------|----------| | 2-bit | 32x | ~2% | Maximum compression, edge devices | | 3-bit | 10.7x | <1% | Balanced — recommended for most uses | | 4-bit | 8x | <0.5% | High quality, long-context inference | | 8-bit | 4x | ~0% | Baseline quantization |

CLI Usage

# Query a model
ruvllm query "What is machine learning?"

# Stream output
ruvllm query --stream "Write a poem"

# Download a model
ruvllm download ruvector/ruvltra-small-q4km

# Benchmark
ruvllm bench ./models/model.gguf

# Run evaluation (SWE-Bench)
ruvllm eval --model ./models/model.gguf --subset lite --max-tasks 50

API Reference

RuvLLM Class

class RuvLLM {
  constructor(config?: RuvLLMConfig);

  // Generate text
  query(prompt: string, params?: GenerateParams): Promise<Response>;

  // Stream generation
  stream(prompt: string, params?: GenerateParams): AsyncIterable<string>;

  // Load a model
  loadModel(path: string): Promise<void>;

  // Get SONA learning stats
  sonaStats(): SonaStats | null;

  // Adapt on feedback
  adapt(input: Float32Array, quality: number): void;
}

Configuration

interface RuvLLMConfig {
  modelPath?: string;       // Path to GGUF model
  sonaEnabled?: boolean;    // Enable SONA learning (default: true)
  flashAttention?: boolean; // Use Flash Attention 2 (default: true)
  maxTokens?: number;       // Max generation tokens (default: 256)
  temperature?: number;     // Sampling temperature (default: 0.7)
  topP?: number;            // Top-p sampling (default: 0.9)
}

Generate Parameters

interface GenerateParams {
  maxTokens?: number;
  temperature?: number;
  topP?: number;
  topK?: number;
  repetitionPenalty?: number;
  stopSequences?: string[];
}

SIMD Module

For direct access to optimized SIMD kernels:

import { simd } from '@ruvector/ruvllm/simd';

// Dot product
const result = simd.dotProduct(vecA, vecB);

// Matrix multiplication
const output = simd.matmul(matrix, vector);

// Flash Attention
const attended = simd.flashAttention(query, key, value, scale);

// RMS Normalization
simd.rmsNorm(hidden, weights, epsilon);

Performance (M4 Pro)

| Operation | Performance | |-----------|-------------| | Inference | 88-135 tok/s | | Flash Attention | 320µs (seq=2048) | | HNSW Search | 17-62µs | | SONA Adapt | <1ms | | Evaluation | 5 ablation modes |

Evaluation Harness

Run model evaluations with SWE-Bench integration:

import { RuvLLM, EvaluationHarness, AblationMode } from '@ruvector/ruvllm';

const harness = new EvaluationHarness({
  modelPath: './models/model.gguf',
  enableHnsw: true,
  enableSona: true,
});

// Run single evaluation
const result = await harness.evaluate(
  'Fix the null pointer exception',
  'def process(data): return data.split()',
  AblationMode.Full
);

console.log(`Success: ${result.success}, Quality: ${result.qualityScore}`);

// Run ablation study (Baseline, RetrievalOnly, AdaptersOnly, R+A, Full)
const report = await harness.runAblationStudy(tasks);
for (const [mode, metrics] of Object.entries(report.modeMetrics)) {
  console.log(`${mode}: ${metrics.successRate * 100}% success`);
}

mistral-rs Backend (Production Serving)

For production deployments with 10-100+ concurrent users, use the mistral-rs backend:

import { RuvLLM, MistralBackend, PagedAttentionConfig } from '@ruvector/ruvllm';

// Configure for production serving
const backend = new MistralBackend({
  // PagedAttention: 5-10x more concurrent users
  pagedAttention: {
    blockSize: 16,
    maxBlocks: 4096,
    gpuMemoryFraction: 0.9,
    prefixCaching: true,
  },
  // X-LoRA: Per-token adapter routing
  xlora: {
    adapters: ['./adapters/coder', './adapters/researcher'],
    topK: 2,
  },
  // ISQ: Runtime quantization
  isq: {
    bits: 4,
    method: 'awq',
  },
});

const llm = new RuvLLM({ backend });
await llm.loadModel('mistralai/Mistral-7B-Instruct-v0.2');

// Serve multiple concurrent requests
const response = await llm.query('Write production code');

Note: mistral-rs features require the Rust backend with mistral-rs feature enabled. Native bindings will use mistral-rs when available.

Supported Models

• RuvLTRA-Small (494M) - Q4K, Q5K, Q8
• RuvLTRA-Medium (3B) - Q4K, Q5K, Q8
• Qwen 2.5 (0.5B-72B)
• Llama 3.x (8B-70B)
• Mistral (7B-22B)
• Phi-3 (3.8B-14B)
• Gemma-2 (2B-27B)

Platform Support

| Platform | Architecture | Status | |----------|--------------|--------| | macOS | arm64 (M1-M4) | ✅ Full support | | macOS | x64 | ✅ Supported | | Linux | x64 | ✅ Supported | | Linux | arm64 | ✅ Supported | | Windows | x64 | ✅ Supported |

Related Packages

• @ruvector/core - Vector operations
• @ruvector/sona - SONA learning engine
• @ruvector/ruvector - Full Ruvector SDK

Links

• GitHub Repository
• API Documentation
• Crate (Rust)

License

MIT OR Apache-2.0