react-native-local-llm

One interface for every on-device LLM runtime in React Native.

Run Apple Foundation Models, Google's Gemma (via ExecuTorch), and llama.cpp-based Cactus models (Qwen and friends) behind a single LLMEngine interface — with device-aware model selection, unified token streaming, download management, and OOM guards. Fully on-device: no server, no cloud handoff.

Extracted from Solmari, a private on-device AI assistant on the App Store. Every guard and workaround in this library exists because something broke in production first.

The problem

On-device inference in React Native is fragmented across runtimes that share nothing:

| | Apple Foundation Models | ExecuTorch (Gemma) | Cactus (llama.cpp) | |---|---|---|---| | API style | Vercel AI SDK provider | imperative module + token callback | promise + token callback | | Availability | iOS 26+ Apple-Intelligence devices only | iOS/Android, physical devices with enough RAM | iOS/Android | | Download | none (ships with OS) | ~4 GB weights | 0.5–1.3 GB weights | | Images | no | yes | model-dependent | | Embeddings | no | no | yes | | maxTokens | yes | no (must be enforced client-side) | yes |

Supporting more than one means three lifecycles, three streaming styles, and three failure modes scattered through your app. This library collapses them into one interface, so switching models — including across runtimes — is a one-line change.

Quickstart

import { autoSelectEngine } from "react-native-local-llm";

// Picks the best model this device can actually hold:
// 12 GB phone → Gemma 4 E2B (multimodal) · Apple-Intelligence device → instant
// Apple model · 4 GB phone → Qwen3.5 0.8B
const engine = autoSelectEngine();

await engine.ensureReady((p) => console.log(`downloading… ${Math.round(p * 100)}%`));

const { textStream } = await engine.streamChat([
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "Explain how tides work in two sentences." },
]);

for await (const delta of textStream) {
  process.stdout.write(delta); // append to your UI state
}

Everything else in your app talks to getEngine() — chat, summarization, semantic search — and never imports a runtime directly:

import { getEngine } from "react-native-local-llm";

const title = await getEngine().generateText("Give this note a 4-word title: …");
const vector = await getEngine().embed?.("query text"); // null/undefined → fall back to keyword search

Installation

npm i react-native-local-llm

Then install only the engines you want — each is an optional peer dependency:

| Engine | Install | Runs | |---|---|---| | Apple | npm i @react-native-ai/apple ai | Apple's built-in Foundation Model | | ExecuTorch | npm i react-native-executorch react-native-executorch-expo-resource-fetcher | Gemma 4 E2B multimodal (text + image + audio) | | Cactus | npm i cactus-react-native | Qwen3.5 2B / 0.8B and other llama.cpp models | | Device detection | npm i expo-device | RAM tiering + simulator guard (recommended) |

Missing engines degrade cleanly: isSupported() returns false instead of throwing, and autoSelectEngine simply skips them.

Device-aware model selection

A 4 GB phone that tries to load a 4 GB model doesn't get an error — it gets jetsam-killed. The catalog encodes what each model needs, and selection respects it:

import { DEFAULT_CATALOG, resolveDefaultModel } from "react-native-local-llm";

const spec = resolveDefaultModel(DEFAULT_CATALOG, {
  appleAvailable: true,
  totalMemoryBytes: 6 * 1024 ** 3,
});
// → Apple (instant), because 6 GB is below Gemma's 8 GiB default gate

Two gates per model, deliberately different:

• defaultMinRamGib — RAM needed to be picked by default. Conservative: unknown RAM skips gated models unless allowUnknownRam is set.
• hardMinRamGib — the absolute floor, checked at load time. A 6 GB power user may opt into Gemma and accept the risk; a 4 GB device is hard-blocked because the load is a guaranteed OOM crash. Unknown RAM is allowed through — the user asked for the model and the device can't be measured.

Define your own catalog with your own models and gates — DEFAULT_CATALOG is just a curated, production-tuned starting point.

Born in production

Details this library handles that you'd otherwise rediscover the hard way:

• The double-answer race. Native runtimes stream tokens through a callback and resolve a promise with the full text. If you count tokens on the consumer side, a fast completion appends the full response on top of the queued tokens — rendering the whole answer twice. createTokenStream counts on arrival and only falls back to the resolved text when nothing streamed. (It's exported — use it to bridge any callback-streaming API.)
• Repeat-forever loops. Quantized small models degenerate into repeating one sentence. The default catalog ships the fix: repetitionPenalty: 1.15 + minP: 0.05 for Gemma, a tighter nucleus (topK: 20, topP: 0.8) for Qwen.
• No native maxTokens on ExecuTorch. A degenerate generation runs until the context fills. The cap is enforced client-side: when streamed output exceeds the budget, the runner is interrupted and resolves normally with partial text.
• Percent-encoded image URIs. Image pickers hand over file:// URIs that are percent-encoded; Cactus loads plain paths. Passed as-is, the image is silently dropped and the model hallucinates an unrelated scene.
• Simulator crashes. GPU-delegated builds (iOS MLX / Android Vulkan) can't initialize on a simulator. The load bails cleanly so your app falls back instead of stack-tracing on every simulator launch.
• Mid-download switching. Switching models while a 4 GB download is in flight doesn't orphan or restart the transfer — the download finishes in the background, the weights are cached, and the module just isn't kept resident next to the other runtime. Deleting, by contrast, cancels the download outright.
• Single-flight loads. Concurrent ensureReady calls share one download; progress reports to the latest caller, not whichever closure happened to start the fetch.

Privacy

• Cactus telemetry is forced off on every call. Cactus defaults telemetry ON, which transmits usage off-device — this library never lets that happen.
• Apple Foundation Models run entirely on-device by design.
• Heads-up: ExecuTorch's downloader (upstream, not this library) fires one analytics request on first model download. If you make a strict no-network promise, patch or fork the resource fetcher.

API sketch

interface LLMEngine {
  id: string;
  label: string;
  supportsImages: boolean;
  isSupported(): boolean;
  ensureReady(onProgress?: (p: number) => void): Promise<boolean>;
  streamChat(messages, { temperature, topK, topP, minP, repetitionPenalty,
                         maxTokens, stopSequences, enableThinking, abortSignal }):
    Promise<{ textStream: AsyncIterable<string> }>;
  generateText(prompt, options): Promise<string>;
  embed?(text): Promise<number[] | null>; // Cactus only
}

Model management is exported per engine: isCactusModelDownloaded / deleteCactusModel / unloadActiveCactusModel, isExecutorchModelDownloaded / deleteExecutorchModel / unloadExecutorchModel — so a Settings screen can show download state and free disk without loading anything.

Abort works the same everywhere:

const controller = new AbortController();
const { textStream } = await engine.streamChat(messages, { abortSignal: controller.signal });
// later: controller.abort() — the runtime is interrupted, the stream ends cleanly

Roadmap

• On-device RAG helpers (sqlite-vec store + chunking) — extracted from the same app
• More engines: llama.rn, MLC, Google AI Edge (Gemini Nano)
• Per-conversation KV-cache reuse where runtimes expose it

License

MIT © Binaya Dhakal