@sovea/node-llm-gateway

Gateway for local LLM backends in NodeJs.

@sovea/node-llm-gateway is a backend-first runtime and protocol adapter for local model access. The current built-in protocol is OpenAI-compatible Chat Completions. You can call the gateway in-process through a fetch-style facade, or expose the same routes through an optional localhost HTTP server.

Install

pnpm add @sovea/node-llm-gateway

This package is ESM-only and requires Node.js 20 or newer.

Quick Start

Create backend instances, bind public model ids to those backends, then create a gateway from the runtime.

import {
  createGateway,
  createModelRuntime,
  defineModel,
  llamaCppArtifacts,
  nodeLlamaCppBackend,
} from '@sovea/node-llm-gateway'

const runtime = createModelRuntime({
  backends: [
    nodeLlamaCppBackend({
      id: 'node-llama',
      cacheDir: './models',
      artifacts: llamaCppArtifacts({
        model: { source: 'hf:user/model:Q4_K_M' },
      }),
    }),
  ],
  models: [
    defineModel({
      id: 'local-llama',
      aliases: ['default'],
      backend: 'node-llama',
      backendModelId: 'local-llama',
    }),
  ],
})

const gateway = createGateway({
  runtime,
  protocols: ['openai'],
})

const response = await gateway.fetch('/v1/chat/completions', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({
    model: 'default',
    messages: [{ role: 'user', content: 'Say hello' }],
  }),
})

console.log(await response.json())

To expose the same gateway over localhost, use serveGateway(...):

import { serveGateway } from '@sovea/node-llm-gateway'

const served = serveGateway({
  runtime,
  port: 8080,
  protocols: ['openai'],
})

await served.ready
console.log(served.url)
await served.dispose()

served.close() closes only the HTTP server by default. served.dispose() closes the HTTP server and disposes the runtime. Pass disposeRuntimeOnClose: true when close() should also dispose the runtime.

Current Support

Implemented protocol support is limited to OpenAI-compatible Chat Completions:

• GET /v1/models
• GET /v1/models/:id
• POST /v1/chat/completions
• GET /gateway/models
• GET /gateway/models/:id

Model Bindings

The public model id is separate from the backend model id:

defineModel({
  id: 'public-name',
  aliases: ['default'],
  backend: 'llama-remote',
  backendModelId: 'actual-llama-server-model-name',
})

Backends

Built-in backend helpers are backend-first. They implement model loading, generation, streaming, disposal, and capability declarations. Protocol route semantics stay in the OpenAI protocol adapter.

node-llama-cpp

nodeLlamaCppBackend({
  id: 'node-llama',
  cacheDir: './models',
  artifacts: llamaCppArtifacts({
    model: { source: 'hf:user/model:Q4_K_M' },
  }),
})

This backend resolves artifacts through node-llama-cpp, lazy-loads the model, enforces text-only input, supports streaming, function tool calls, JSON mode, and Structured Outputs, and serializes generation per backend instance to protect mutable sessions.

The gateway does not execute tools. Tool definitions are passed to the backend so the model can request function tool calls, which are returned to the caller.

llama-server

Managed mode starts one private llama-server process for the backend instance:

llamaCppServerBackend({
  id: 'vision-local',
  binaryPath: '/usr/local/bin/llama-server',
  cacheDir: './models',
  artifacts: llamaCppArtifacts({
    model: { source: '/models/vision-model.gguf' },
    mmproj: { source: '/models/mmproj.gguf' },
  }),
  args: ['--ctx-size', '8192'],
  apiKey: 'auto',
  capabilities: {
    inputModalities: ['text', 'image'],
    outputModalities: ['text'],
    features: ['streaming'],
  },
})

Remote Runtime

Electron apps can expose a model runtime over a transport and use it from another process:

import {
  createGateway,
  createInMemoryRuntimeTransport,
  createRemoteModelRuntime,
  exposeModelRuntime,
} from '@sovea/node-llm-gateway'

const transport = createInMemoryRuntimeTransport()
exposeModelRuntime(runtime, transport)

const gateway = createGateway({
  runtime: createRemoteModelRuntime({ transport }),
  protocols: ['openai'],
})

Custom transports implement GatewayRuntimeTransport and GatewayRuntimeServerTransport.

Development

pnpm install
pnpm test -- --run
pnpm run typecheck
pnpm run build

License

MIT