@node-llm/core

The production-grade LLM engine for Node.js. Provider-agnostic by design.

@node-llm/core provides a single, unified API for interacting with over 540+ models across all major providers. It is built for developers who need stable infrastructure, standard streaming, and automated tool execution without vendor lock-in.

🚀 Key Features

• Unified API: One interface for OpenAI, Anthropic, Gemini, DeepSeek, OpenRouter, and Ollama.
• Automated Tool Loops: Recursive tool execution handled automatically—no manual loops required.
• Streaming + Tools: Seamlessly execute tools and continue the stream with the final response.
• Structured Output: Native Zod support for rigorous schema validation (.withSchema()).
• Multimodal engine: Built-in handling for Vision, Audio (Whisper), and Video (Gemini).
• Security-First: Integrated circuit breakers for timeouts, max tokens, and infinite tool loops.

📋 Supported Providers

| Provider | Supported Features | | :----------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------- | | OpenAI | Chat, Streaming, Tools, Vision, Audio, Images, Transcription, Reasoning | | Anthropic | Chat, Streaming, Tools, Vision, PDF, Structured Output, Extended Thinking (Claude 3.7) | | Gemini | Chat, Streaming, Tools, Vision, Audio, Video, Embeddings | | DeepSeek | Chat (V3), Extended Thinking (R1), Streaming, Tools | | Bedrock | Chat, Streaming, Tools, Image Gen (Titan/SD), Embeddings, Prompt Caching | | OpenRouter | 540+ models, Chat, Streaming, Tools, Vision, Embeddings, Reasoning | | Ollama | Local Inference, Chat, Streaming, Tools, Vision, Embeddings | | Mistral | Chat, Streaming, Tools, Vision, Embeddings, Transcription, Moderation, Reasoning (Magistral) |

⚡ Quick Start

Installation

npm install @node-llm/core

Basic Chat & Streaming

NodeLLM automatically reads your API keys from environment variables (e.g., OPENAI_API_KEY).

import { createLLM } from "@node-llm/core";

const llm = createLLM({ provider: "openai" });

// 1. Standard Request
const res = await llm.chat("gpt-4o").ask("What is the speed of light?");
console.log(res.content);

// 2. Real-time Streaming
for await (const chunk of llm.chat().stream("Tell me a long story")) {
  process.stdout.write(chunk.content);
}

Structured Output (Zod)

Stop parsing markdown. Get typed objects directly.

import { z } from "@node-llm/core";

const PlayerSchema = z.object({
  name: z.string(),
  powerLevel: z.number(),
  abilities: z.array(z.string())
});

const chat = llm.chat("gpt-4o-mini").withSchema(PlayerSchema);
const response = await chat.ask("Generate a random RPG character");

console.log(response.parsed.name); // Fully typed!

🛡️ Security Circuit Breakers

NodeLLM protects your production environment with four built-in safety pillars:

const llm = createLLM({
  requestTimeout: 15000, // 15s DoS Protection
  maxTokens: 4096, // Cost Protection
  maxRetries: 3, // Retry Storm Protection
  maxToolCalls: 5 // Infinite Loop Protection
});

🔌 Middleware System

NodeLLM 1.9.0 introduces a powerful lifecycle hook system for audit, security, and observability.

import { createLLM, PIIMaskMiddleware, UsageLoggerMiddleware } from "@node-llm/core";

const llm = createLLM({
  provider: "openai",
  middlewares: [
    new PIIMaskMiddleware(), // Redact emails/phone numbers automatically
    new UsageLoggerMiddleware() // Log structured token usage & costs
  ]
});

// All chats created from this instance inherit these middlewares
const chat = llm.chat("gpt-4o");

Decisive Tool Safety

Middlewares can control the engine's recovery strategy during tool failures.

const safetyMiddleware = {
  name: "Audit",
  onToolCallError: async (ctx, tool, error) => {
    if (tool.function.name === "delete_user") return "STOP"; // Kill the loop
    return "RETRY"; // Attempt recovery
  }
};

🤖 Agent Class

Define reusable, class-configured agents with a declarative DSL:

import { Agent, Tool, z } from "@node-llm/core";

class LookupOrderTool extends Tool<{ orderId: string }> {
  name = "lookup_order";
  description = "Look up an order by ID";
  schema = z.object({ orderId: z.string() });

  async execute({ orderId }: { orderId: string }) {
    return { status: "shipped", eta: "Tomorrow" };
  }
}

class SupportAgent extends Agent {
  static model = "gpt-4.1";
  static instructions = "You are a helpful support agent.";
  static tools = [LookupOrderTool];
  static temperature = 0.2;
}

// Use anywhere in your app
const agent = new SupportAgent();
const response = await agent.ask("Where is order #123?");
console.log(response.content);

ToolHalt - Early Loop Termination

Stop the agentic loop early when a definitive answer is found:

class FinalAnswerTool extends Tool<{ answer: string }> {
  name = "final_answer";
  description = "Return the final answer to the user";
  schema = z.object({ answer: z.string() });

  async execute({ answer }: { answer: string }) {
    return this.halt(answer); // Stops the loop, returns this result
  }
}

💾 Ecosystem

Looking for persistence? use @node-llm/orm.

• Automatically saves chat history to PostgreSQL/MySQL/SQLite via Prisma.
• Tracks tool execution results and API metrics (latency, cost, tokens).

📚 Full Documentation

Visit nodellm.dev for:

• Deep Dive into Tool Calling
• Multi-modal Vision & Audio Guide
• Custom Provider Plugin System

License

MIT © [NodeLLM Contributors]