Sipp Server for Node.js

lib/node is the Node.js package source for the public @sipphq/sipp-server package. It exposes Sipp's native client API to Node server processes for local GGUF inference, gateway-backed inference, provider descriptors, and token streaming.

Source builds use the workspace manifest in this directory. Public docs use the @sipphq/sipp-server package target. Applications own framework routes and call client.query(), client.chat(), or client.embed() inside those routes.

Source Checkout

From the repository root, after source ./setup.sh:

sipp build node --backend cpu && node examples/node/query.mjs <model.gguf> "Explain Sipp."

sipp forwards to cargo xtask; use cargo xtask ... with the same arguments if the launcher is not active.

Set SIPP_NODE_BACKEND=cpu|vulkan|cuda|metal to choose a native backend.

Local GGUF Query

import { EndpointDescriptor, SippClient } from '@sipphq/sipp-server';

const client = new SippClient({ storageRoot: '.sipp-models' });
const model = await client.models.add([process.argv[2]]);
await client.add(
  'default',
  EndpointDescriptor.local(model.id, {
    runtime: {
      context: { n_ctx: 2048 },
      scheduler: { continuous_batching: true, prefill_chunk_size: 0 },
      cache: { mode: 'live_slot_prefix' },
      observability: { runtime_metrics: true },
    },
  })
);

const run = client.query({
  prompt: 'Explain Sipp in one sentence.',
  emitTokens: true,
  options: { maxTokens: 64, temperature: 0.7 },
  local: { contextKey: 'node-local' },
});

let streamed = '';
for await (const batch of run) {
  streamed += batch.text;
}
const response = await run.response;
console.log(streamed || response.text);
await client.close();

Gateway clients use EndpointDescriptor.gateway(...) when a Node process calls a separate Sipp gateway.

Gateway Profile Helpers

Use the gateway profile helpers when a Node route should behave like a first-party gateway endpoint for browser gateway clients:

import {
  EndpointDescriptor,
  SippClient,
  decodeGatewayQueryBody,
  gatewayErrorResponse,
  gatewayTextResponseBody,
  gatewayTextStreamResponse,
} from '@sipphq/sipp-server';

export async function handleQuery(request: Request): Promise<Response> {
  try {
    const decoded = decodeGatewayQueryBody(await request.json());
    const client = new SippClient();
    const endpoint = await client.add('gateway', EndpointDescriptor.gateway({
      target: decoded.target,
      baseUrl: process.env.SIPP_GATEWAY_URL!,
      authentication: {
        kind: 'bearer',
        value: process.env.SIPP_GATEWAY_TOKEN!,
      },
    }));
    const run = client.query({ ...decoded.request, endpoint });
    return decoded.stream
      ? gatewayTextStreamResponse(run)
      : Response.json(
          gatewayTextResponseBody(decoded.target, await run.response),
        );
  } catch (error) {
    const response = gatewayErrorResponse(error);
    return Response.json(response.body, response.init);
  }
}

decodeGatewayChatBody(), decodeGatewayEmbedBody(), gatewayTextResponseBody(), and gatewayEmbeddingResponseBody() mirror the first-party gateway JSON profile used by Sipp clients.

Learn More

• Node.js package docs
• Local inference
• Gateway and hybrid inference
• Node examples