nanobridge

v0.2.4

Published

16 days ago

Chrome Built-in AI CLI bridge to on-device language models via CDP

0High
0Medium
0Low

cond

chrome built-in-ai prompt-api gemini-nano cli cdp on-device-ai local-ai

NanoBridge

Use Chrome's built-in on-device AI from your terminal and OpenAI-compatible tools.

NanoBridge is a lightweight Node.js CLI & server that bridges Chrome's Built-in AI / on-device language models (LanguageModel / Gemini Nano) to your terminal and local developer tools via the Chrome DevTools Protocol (CDP).

Features

On-Device Inference: Runs entirely on your local machine using Chrome's built-in Gemini Nano model.
Zero External AI Services: No API keys, no third-party model runtimes (no Ollama, no LM Studio), no cloud fallbacks.
Sub-Second Warm Daemon: Keeps Chrome and model weights warm in RAM for instantaneous response times.
Interactive Chat REPL: Interactive multi-turn conversation with memory and /clear, /system, /history, /help commands.
File Reading & @file Mentions: Attach local files via -f / --file or inline @filepath mentions.
OpenAI-Compatible API: Mounts /v1/chat/completions and /v1/models for plug-and-play use in Cursor, Continue, LangChain, OpenAI SDK, Open WebUI, LiteLLM, and curl.
Full Unix Pipe & Stream Support: First-class streaming output and standard input (stdin) piping.
Hardware Acceleration: Automatic macOS Metal and GPU rasterization flags for maximum inference throughput.
Microsecond Benchmarking: Measures Chrome startup, session initialization, Time To First Token (TTFT), and generation rate.
Isolated & Safe: Uses isolated temporary profiles for each execution without interfering with your personal Chrome sessions.

Installation

npm install -g nanobridge

Or run directly with npx:

npx nanobridge status

Quick Start

1. Check AI & Model Status

Detect your Chrome version, Built-in AI availability, model readiness, and daemon state:

nanobridge status

Example output:

NanoBridge

Chrome          151.0.7922.174
Platform        macOS arm64
Built-in AI     available
LanguageModel   available
Model           ready
Execution       on-device
Daemon          running (PID: 14947, port: 56340)

Output as structured JSON:

nanobridge status --json

2. Download Model Components

If the model is in a downloadable state, trigger the official Chrome model preparation and monitor download progress:

nanobridge download

3. Interactive Chat Mode & Runtime Settings

Start an interactive multi-turn conversation with memory and live runtime configuration:

nanobridge chat

Slash Commands & Settings:

/sampling <mode> — Change sampling mode (creative, predictable, balanced, etc.). Recreates session and preserves history.
/system <prompt> — Set or update system instructions (recreates session and preserves conversation).
/json on / /json off — Toggle structured JSON output constraint on the fly.
/schema <file.json> — Load and validate a JSON schema for structured model outputs (/schema clear to remove).
/context — Inspect token usage, total window size, percentage, and message count.
/settings — Display the current session configuration, language, and runtime capabilities.
/clear — Clear conversation history while preserving your active system prompt and sampling settings.
/new — Reset both conversation history and all settings back to default.
/read <filepath> (or @file) — Load local files directly into the conversation context.
/history — View active conversation turns.
/help — List available commands based on runtime capabilities.
/exit or Ctrl+C — Quit the chat session.

4. Ask Questions & Prompt the Model

Quick Prompts & Unix Pipes

Ask questions directly from your terminal or pipe content:

# Simple question
nanobridge ask "Explain Web Workers in one sentence"

# Pipe text from any Unix command
cat README.md | nanobridge ask "Summarize this document in 3 bullet points"
git diff | nanobridge ask "Generate a conventional commit message for these changes"
tail -n 50 /var/log/system.log | nanobridge ask "Find any errors or warnings in this log"

File Reading & Code Context

Attach local files to your prompts easily:

# Attach file using -f / --file flag
nanobridge ask -f package.json "What dependencies are used?"

# Attach multiple files
nanobridge ask -f src/cli.js -f package.json "How is the CLI structured?"

# Use inline @file mentions
nanobridge ask "Review and find bugs in @src/chrome/launch.js"

Tip: nanobridge ask automatically starts the background daemon on your first query so subsequent questions respond in < 1 second.

5. OpenAI-Compatible API Server

Start a dedicated OpenAI-compatible local server:

nanobridge serve --port 8000

Endpoints Provided

GET /v1/models — Lists available local models (gemini-nano, chrome, gpt-4o-mini alias)
POST /v1/chat/completions — Standard chat completions (Supports both JSON and Streaming SSE)
POST /v1/completions — Text completions endpoint

Example: Streaming with `curl`

curl -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-nano",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is WebAssembly?"}
    ],
    "stream": true
  }'

Example: Python OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="nanobridge"  # Any string
)

response = client.chat.completions.create(
    model="gemini-nano",
    messages=[
        {"role": "user", "content": "Explain async/await in Python"}
    ],
    stream=True
)

for chunk in response:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
print()

Example: Node.js OpenAI SDK

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: 'http://localhost:8000/v1',
  apiKey: 'nanobridge',
});

const stream = await openai.chat.completions.create({
  model: 'gemini-nano',
  messages: [{ role: 'user', content: 'Explain Node.js event loop' }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

6. Background Daemon Management & Context Reset

NanoBridge runs a warm Chrome background daemon to eliminate the ~2.5s cold-start penalty on every query:

# Start the background daemon
nanobridge daemon start

# Check daemon status (PID, Port, Uptime)
nanobridge daemon status

# Reset model session and clear KV context cache (<100ms)
nanobridge reset

# Restart the background daemon process
nanobridge restart
# or:
nanobridge daemon restart

# Stop the running background daemon
nanobridge stop
# or:
nanobridge daemon stop

Memory & Halucination Reset: If you have long-running chat sessions or heavy multi-file prompts, running nanobridge reset (or typing /reset in nanobridge chat) instantly clears Chrome's on-device session and KV cache without needing to cold-relaunch Chrome.

7. Benchmarking On-Device Performance

Measure the exact millisecond latencies of your machine's hardware and Chrome runtime:

# Single pass
nanobridge bench

# Multi-run statistical benchmark with percentiles (p50, p90, p99)
nanobridge bench --runs 5