@bendyline/gezel-inference

The on-device inference runtime shared by Gezel and by applications that embed it. It runs local AI models on the user's own computer and answers them through OpenAI-compatible routes:

• Native engines — starting, sizing, and stopping llama.cpp, MLX, and the other on-device engines, with admission control so models fit in memory.
• Model management — listing, downloading, verifying, and preparing local chat models.
• Embeddings and knowledge — the on-device text embedder (model: "gezel:embeddings") and search over installed knowledge catalogs, each in its own worker.
• OpenAI-compatible routes — /v1/chat, /v1/responses, /v1/embeddings, /v1/models, /v1/models/ensure, /v1/knowledge, and the engine routes.

It holds no product state: no gezels, projects, chats, tools, cloud providers, or UI. Those belong to the full Gezel service, @bendyline/gezel-service, which uses this same runtime.

Install

npm install @bendyline/gezel-app-sdk @bendyline/gezel-inference

Node 24 or newer. On-device text embeddings and the optional relevance model also need the @huggingface/transformers peer:

npm install @huggingface/transformers@^4.3.1

Use it through the app SDK

Most applications should not import this package directly. Install it beside @bendyline/gezel-app-sdk and ask the SDK for an inference-only host. The SDK loads this runtime inside your process, so there is no second process to supervise, no separate Node executable to ship, and no loopback network connection:

import { connectOrHost } from '@bendyline/gezel-app-sdk/host';

const gezel = await connectOrHost({
  appId: 'acme.editor',
  appName: 'Acme Editor',
  scopes: ['openai'],
  host: { mode: 'in-process', inferenceOnly: true },
});

const { chatModel } = await gezel.ensureModel({ model: 'qwen3.5-0.8b-q4' });

const reply = await gezel.openai.chat({
  model: chatModel, // 'mlx:qwen3.5-0.8b-q4' on Apple Silicon, 'llama-cpp:…' elsewhere
  messages: [{ role: 'user', content: 'Suggest a title for these notes: …' }],
});
console.log(reply.choices[0].message.content);

await gezel.close();

The runtime never downloads a model on its own: weights are fetched only when your app asks for them, for example with ensureModel. See the app SDK README for consent, hosting options, shipping engines and models with your app, and knowledge catalogs.

Entry points

| Import | What it provides | |---|---| | @bendyline/gezel-inference | startService(), the in-process host the app SDK calls, and verifyNativeBinaries() | | @bendyline/gezel-inference/native-release | The pinned native engine release and its archive digests, as data. Importing it starts nothing | | @bendyline/gezel-inference/packaging | Build-time helpers that stage the pinned native engines for an Electron app | | @bendyline/gezel-inference/speech | The Kokoro text-to-speech frontend and its pronunciation-dictionary lookup |

The package also includes chat-model and knowledge-catalog metadata generated from the pinned @bendyline/gilde catalog. The pronunciation dictionaries' license is at dist/kokoro-lexicon/LICENSE.

Stability

The entry points above are public API under semver. @bendyline/gezel-inference/internal/* is not: it is how the Gezel service reaches the runtime, always installed at exactly the same version, and it can change in any release. Do not import it.

Documentation

• Repository and full documentation
• Embedding guide

MIT © Bendyline