@santhoshdasari/local-llm

A high-performance, browser-native LLM library providing OpenAI API and Google Gemini API compatible layers to run open-source language models (SmolLM2, Qwen2.5, Gemma 2, Llama 3.2, etc.) 100% locally in the browser using WebAssembly (WASM) and WebGPU.

Includes a standalone, zero-config, beautiful React Chat Component (<LocalChat />) and React hooks for seamless integration.

⚡ Highlights

• 🔒 100% Private & In-Browser: Zero server calls, zero API keys, no telemetry. Models run completely client-side.
• 🔄 OpenAI-Compatible Client: Drop-in API mimicking client.chat.completions.create({ model, messages, stream }).
• ♊ Gemini-Compatible Client: Drop-in API mimicking ai.getGenerativeModel().generateContent() and generateContentStream().
• ⚙️ Dual WASM & WebGPU Engine: Automatic WebGPU hardware acceleration on modern browsers with instant fallback to WebAssembly (WASM CPU) for broad browser support across current + last 5 browser versions.
• 🧵 Off-Thread Web Worker: Model inference runs in a background Web Worker — your UI stays at a smooth 60fps with zero stutter.
• 🎨 Standalone <LocalChat /> Component: Plug-and-play glassmorphism chat UI with token-streaming, live speed indicator (tokens/sec), model switching, download progress, and markdown formatting with code block copy buttons.
• 🪝 Reusable React Hooks: useLocalChat and useLocalLLM for custom UI building.

📦 Installation

npm install @santhoshdasari/local-llm

Peer Dependencies

React and React DOM (^18.0.0 || ^19.0.0) are peer dependencies for the UI components and hooks:

npm install react react-dom

🚀 Quick Start

1. OpenAI-Compatible API Layer

import { createOpenAICompatibleClient } from '@santhoshdasari/local-llm';

const openai = createOpenAICompatibleClient({
  defaultModel: 'onnx-community/SmolLM2-135M-Instruct',
  device: 'auto', // 'auto' | 'webgpu' | 'wasm'
  hfToken: 'hf_...', // Optional: Hugging Face token (for gated models or rate limit bypass)
});

// Non-streaming completion
const response = await openai.chat.completions.create({
  model: 'onnx-community/SmolLM2-135M-Instruct',
  messages: [
    { role: 'system', content: 'You are a helpful coding assistant.' },
    { role: 'user', content: 'Explain WebAssembly in one sentence.' },
  ],
  temperature: 0.7,
  max_tokens: 128,
});

console.log(response.choices[0].message.content);

// Streaming tokens (AsyncIterable)
const stream = await openai.chat.completions.create({
  model: 'onnx-community/SmolLM2-135M-Instruct',
  messages: [{ role: 'user', content: 'Write a poem about the ocean.' }],
  stream: true,
});

for await (const chunk of stream) {
  const token = chunk.choices[0]?.delta?.content || '';
  process.stdout.write(token);
}

2. Google Gemini-Compatible API Layer

import { createGeminiCompatibleClient } from '@santhoshdasari/local-llm';

const ai = createGeminiCompatibleClient({ device: 'auto' });

const model = ai.getGenerativeModel({
  model: 'onnx-community/SmolLM2-135M-Instruct',
  systemInstruction: 'You are an expert physics tutor.',
  generationConfig: {
    temperature: 0.6,
    maxOutputTokens: 256,
  },
});

// Standard generateContent
const result = await model.generateContent('Explain how black holes form.');
console.log(result.response.text());

// Real-time stream
const streamResult = await model.generateContentStream(
  'What is quantum entanglement?',
);
for await (const chunk of streamResult.stream) {
  console.log(chunk.text());
}

// Final assembled response
const finalResponse = await streamResult.response;
console.log('Full text:', finalResponse.text());

3. Standalone React <LocalChat /> Component

Drop an independent, interactive chat widget into any React / Next.js / Vite application in one line:

import { LocalChat } from '@santhoshdasari/local-llm';

export function App() {
  return (
    <div
      style={{
        width: '100%',
        height: '100vh',
        padding: '20px',
        boxSizing: 'border-box',
      }}
    >
      <LocalChat
        defaultModel="onnx-community/Qwen2.5-0.5B-Instruct"
        theme="dark" // 'dark' | 'light' | 'auto'
        showModelSelector={true}
        showDeviceBadge={true}
        title="Local AI Assistant"
        placeholder="Ask anything (runs 100% in your browser)..."
      />
    </div>
  );
}

4. Custom React Chat using useLocalChat Hook

import { useLocalChat } from '@santhoshdasari/local-llm';

export function CustomChat() {
  const {
    messages,
    input,
    setInput,
    sendMessage,
    stop,
    clearMessages,
    isStreaming,
    isModelLoading,
    loadingProgress,
    activeModel,
    device,
    speed,
  } = useLocalChat({
    defaultModel: 'onnx-community/SmolLM2-135M-Instruct',
    systemPrompt: 'You are an AI assistant.',
  });

  return (
    <div>
      <div>
        <span>Hardware: {device.toUpperCase()}</span>
        {speed && <span> | {speed} tok/s</span>}
      </div>

      {isModelLoading && loadingProgress && (
        <p>Loading model: {loadingProgress.progress}%</p>
      )}

      <div>
        {messages.map((m) => (
          <div key={m.id}>
            <strong>{m.role}: </strong>
            <span>{m.content}</span>
          </div>
        ))}
      </div>

      <input
        value={input}
        onChange={(e) => setInput(e.target.value)}
        onKeyDown={(e) => e.key === 'Enter' && sendMessage()}
        placeholder="Type a message..."
      />
      {isStreaming ? (
        <button onClick={stop}>Stop</button>
      ) : (
        <button onClick={() => sendMessage()}>Send</button>
      )}
    </div>
  );
}

🔗 Local Development & Linking to Another Project

When developing or testing @santhoshdasari/local-llm locally with another consumer application (e.g. Next.js, Vite, or Create React App), follow these steps:

Method 1: Using npm link (Standard)

1. In the local-llm directory:

# Build the library bundle
npm run build

# (Optional) Run in watch mode so changes rebuild automatically
npm run dev

# Register the global symlink
npm link

2. In your consuming project directory:

# Link the local package
npm link @santhoshdasari/local-llm

3. How to Unlink:

In your consuming project:

# Remove the symlink
npm unlink @santhoshdasari/local-llm

# Reinstall the package from npm registry if needed
npm install @santhoshdasari/local-llm

In the local-llm directory:

npm unlink

Method 2: Using Relative File Path

You can install the local package directly using relative path in your consumer project:

# In the consuming project:
npm install ../path/to/local-llm

# To remove and restore registry version:
npm install @santhoshdasari/local-llm

Method 3: Using yalc (Recommended for React projects)

yalc copies the build output into a local offline store, preventing common symlink issues like duplicate React instances (Invalid hook call):

# Install yalc globally (once)
npm install -g yalc

# In local-llm repository:
npm run build
yalc publish

# In consuming project:
yalc add @santhoshdasari/local-llm
npm install

# To remove / unlink:
yalc remove @santhoshdasari/local-llm
npm install

🚀 How to Publish to npm

Method 1: Automated Release via GitHub Actions (Recommended)

CI publishes to npm automatically when a version tag matching v*.*.* is pushed to GitHub:

# 1. Ensure all tests and builds pass
npm run typecheck
npm run test
npm run build

# 2. Create git tag and push
git tag v1.0.0
git push origin v1.0.0

The GitHub Actions workflow will automatically:

1. Verify linting, type-checking, and tests.
2. Build the production dist/ bundle (ESM + CJS + .d.ts).
3. Publish to npm with provenance under public access (npm publish --provenance --access public).

Method 2: Manual Publish via npm CLI

# 1. Login to your npm account (if not already logged in)
npm login

# 2. Bump version in package.json (e.g. patch: 0.1.0 -> 0.1.1)
npm version patch

# 3. Build the library
npm run build

# 4. Publish to npm
npm publish --access public

One-Time Setup: NPM_TOKEN in GitHub Secrets

For automated GitHub Actions publishing:

1. Generate an npm access token (Granular or Classic with Publish permissions) at npmjs.com.
2. In your GitHub repository: Go to Settings → Secrets and variables → Actions.
3. Create a secret named NPM_TOKEN and paste your npm token value.

🌐 Browser Compatibility Matrix

Tested and compatible with modern browsers and the last 5 versions:

| Browser | WebGPU Acceleration | WASM CPU Fallback | Minimum Supported | | :------------------ | :------------------: | :---------------: | :---------------: | | Google Chrome | ✅ (Chrome 113+) | ✅ Full (SIMD) | Chrome 90+ | | Microsoft Edge | ✅ (Edge 113+) | ✅ Full (SIMD) | Edge 90+ | | Apple Safari | ✅ (Safari 18+) | ✅ Full | Safari 15+ | | Mozilla Firefox | ⚙️ (Nightly / Flags) | ✅ Full (SIMD) | Firefox 90+ | | iOS Safari | ✅ (iOS 18+) | ✅ Full | iOS 15+ | | Android Chrome | ✅ (Android 12+) | ✅ Full | Chrome 90+ |

Diagnostic Utility

You can check browser capabilities directly in code:

import { checkBrowserCapabilities } from '@santhoshdasari/local-llm';

const caps = await checkBrowserCapabilities();
console.log(caps);
// {
//   webgpu: true,
//   wasm: true,
//   wasmSimd: true,
//   webWorkers: true,
//   recommendedDevice: 'webgpu'
// }

🧠 Curated Preset Models

| Alias | Hugging Face ID | Size | Target Device | Best For | | :------------- | :------------------------------------- | :------ | :-----------: | :-------------------------------------- | | smollm2-135m | onnx-community/SmolLM2-135M-Instruct | ~135 MB | WASM / WebGPU | Instant load, lightweight CPU testing | | smollm2-360m | onnx-community/SmolLM2-360M-Instruct | ~360 MB | WASM / WebGPU | Balanced speed & conversational quality | | qwen2.5-0.5b | onnx-community/Qwen2.5-0.5B-Instruct | ~400 MB | WebGPU | Multilingual, coding, reasoning | | gemma-2-2b | onnx-community/gemma-2-2b-it | ~1.5 GB | WebGPU | High quality instruction following | | llama-3.2-1b | onnx-community/Llama-3.2-1B-Instruct | ~800 MB | WebGPU | Instruction following & general tasks |

Custom ONNX models can also be passed directly by Hugging Face repository ID (e.g. your-org/your-onnx-model).

🛠️ API Reference

Client Creators

• createOpenAICompatibleClient(options?: OpenAIClientOptions)
• createGeminiCompatibleClient(options?: GoogleGenAIOptions)
• createLocalLLM(options?: LocalLLMOptions)

React Components

• <LocalChat />
• <MessageList />
• <ChatInput />
• <ModelSelector />
• <StatusBadge />
• <ProgressBar />

React Hooks

• useLocalChat(options?: UseLocalChatOptions)
• useLocalLLM(options?: UseLocalLLMOptions)

📜 License

MIT License. Free for commercial and open-source use.