liter-llm (TypeScript/Node.js)

High-performance LLM client library for TypeScript and Node.js. Unified interface for streaming completions, tool calling, and provider routing across OpenAI, Anthropic, and 142+ LLM providers. Powered by Rust core via NAPI-RS bindings with full TypeScript type definitions and native async/Promise support.

Installation

npm:

npm install liter-llm

pnpm:

pnpm add liter-llm

Quick Start

import { LlmClient } from "liter-llm";

const client = new LlmClient({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat({
  model: "openai/gpt-4o",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(response.choices[0].message.content);

Features

| Feature | Supported | |---------|-----------| | Provider Routing | 142+ providers via "provider/model" prefix | | Chat Completions | OpenAI-compatible unified API | | Streaming | Server-sent events, token-by-token | | Tool Calling | Function definitions, structured outputs | | Async | Native async/await | | Provider Auth | Automatic key injection from environment variables | | Retry Logic | Configurable retries with exponential backoff | | Timeouts | Per-request configurable timeouts |

Streaming

Stream tokens as they are generated for responsive user experiences:

import { LlmClient } from "liter-llm";

const client = new LlmClient({ apiKey: process.env.OPENAI_API_KEY });
const stream = client.chatStream({
  model: "openai/gpt-4o",
  messages: [{ role: "user", content: "Tell me a story" }],
});

for await (const chunk of stream) {
  process.stdout.write(chunk.delta ?? "");
}
console.log();

Tool Calling

Define tools for the model to call with structured outputs:

import { LlmClient, type Tool } from "liter-llm";

const client = new LlmClient({ apiKey: process.env.OPENAI_API_KEY });

const tools: Tool[] = [
  {
    name: "get_weather",
    description: "Get the current weather for a location",
    parameters: {
      type: "object",
      properties: {
        location: { type: "string", description: "City name" },
      },
      required: ["location"],
    },
  },
];

const response = await client.chat({
  model: "openai/gpt-4o",
  messages: [{ role: "user", content: "What is the weather in Berlin?" }],
  tools,
});

for (const call of response.toolCalls ?? []) {
  console.log(`Tool: ${call.name}, Args:`, call.arguments);
}

Hooks

Register lifecycle hooks for observability or request manipulation:

client.addHook({
  onRequest: (req) => console.log(`Sending to ${req.model}`),
  onResponse: (resp) => console.log(`Got ${resp.choices.length} choices`),
  onError: (err) => console.error(`Error: ${err}`),
});

Budget

Set per-model or global cost limits:

const client = new LlmClient({
  apiKey: process.env.OPENAI_API_KEY,
  budget: { globalLimit: 10.0, modelLimits: { "openai/gpt-4o": 5.0 } },
});
console.log(client.budgetUsed); // cumulative spend in USD

Cache

Enable in-memory LRU response caching:

const client = new LlmClient({
  apiKey: process.env.OPENAI_API_KEY,
  cache: { maxEntries: 256, ttlSeconds: 300 },
});

Custom Providers

Register a custom provider at runtime (static method):

LlmClient.registerProvider({
  name: "my-provider",
  baseUrl: "https://api.my-provider.com/v1",
  authHeader: "Authorization",
  authPrefix: "Bearer",
  modelPrefix: "my-provider",
});

Provider Routing

Route requests to any of 142+ providers using a "provider/model" prefix:

openai/gpt-4o
anthropic/claude-opus-4-5
groq/llama3-70b-8192
mistral/mistral-large-latest
together/meta-llama/Llama-3-70b-chat-hf

Set the provider API key as an environment variable (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY). The client picks up keys automatically — no per-call configuration required.

Documentation

• GitHub Repository
• Provider List
• Contributing Guide

License

MIT License — see LICENSE for details.