llmjs2

A lightweight Node.js library for LLM completions, multi-route routing with guardrails, agent workflows with tools/memory/attachments, local IPC, and a built-in HTTP server.

Install

npm install llmjs2

Build (from source)

npm run build         # tsup -> dist/

Completion

String prompt — returns text only.

import { completion } from 'llmjs2';
const text = await completion('Write a friendly greeting.');

Full options — model format: provider/model_name.

const resp = await completion({
  model: 'openai/gpt-4o-mini',
  messages: [{ role: 'user', content: 'Summarize Node.js event loop.' }],
  apiKey: 'sk-...',       // optional, overrides env
  baseUrl: '...',         // optional, overrides env
  tools: [{ name: 'get_weather', parameters: { location: { type: 'string', description: 'city', required: true } } }],
  timeout: 30000,         // per-request timeout (ms)
});

Provider is auto-selected by first available API key: openai → nvidia → openrouter → zen → bigmodel → ollama.

Router (multi-route LLM routing)

Build from inline config or a JSON/JS file path. Supports three strategies: random, sequential, default.

import { router } from 'llmjs2';

const r = router({
  routing: 'random',
  model_list: [
    { model_name: 'primary', llm_params: { model: 'openai/gpt-4o-mini' } },
    { model_name: 'backup',  llm_params: { model: 'openrouter/openai/gpt-4o-mini' } },
  ],
});

const res = await r.completion({ messages: [{ role: 'user', content: 'Hello' }] });

Vision-aware routing — when messages contain images, only routes with llm_params.vision: true are selected.

Reasoning routing — pass reasoning: true in request options to filter to routes with llm_params.reasoning: true.

Guardrails — pre_call and post_call hooks that transform request/response payloads:

router({
  model_list: [/*...*/],
  guardrails: [
    {
      name: 'redact_email',
      mode: 'pre_call',
      code: "(processId, input) => ({ ...input, messages: input.messages.map(m => ({ ...m, content: m.content.replace(/\\S+@\\S+/g, '[REDACTED]') })) })",
      timeout_ms: 25,
    },
  ],
});

Agent

Multi-turn agent with tool execution, memory, and attachment support.

import { Agent, router, Memory } from 'llmjs2';

const agent = new Agent({
  instruction: 'You are a helpful assistant.',
  route: router({ model_list: [{ model_name: 'r', llm_params: { model: 'openai/gpt-4o-mini' } }] }),
  memory: Memory.inMemory(),
  session: true,      // include conversation history
  relevance: false,   // search relevant past messages
  tools: [
    {
      name: 'echo',
      description: 'Echoes input',
      parameters: { text: { type: 'string', description: 'Text', required: true } },
      execute: async (params) => `ECHO: ${params.text}`,
    },
  ],
});

// String prompt
const out = await agent.generate('Say hello and use the echo tool.');

// Object prompt with memory and attachments
const out2 = await agent.generate({
  userPrompt: 'Analyze this file.',
  memory: { resourceId: 'user_1', threadId: 'session_1', session: true, relevance: true },
  attachments: ['./report.pdf', 'https://example.com/doc.docx'],
  reasoning: true,
});
console.log(out2);  // { content: '...', toolCalls: [...] } when tools used

Agent from YAML config

# agent.yaml
instruction: "You are a helpful assistant."
router:
  routing: random
  model_list:
    - model_name: primary
      llm_params:
        model: openai/gpt-4o-mini
        api_key: ${OPENAI_API_KEY}
memory: inMemory            # or ./memory.json
session: true
tools: ./my_tools.js        # module.exports = AgentTool[]

const agent = await Agent.load('./agent.yaml');
await agent.generate('Hello');

Memory

| API | Description | |-----|-------------| | Memory.inMemory() | In-process storage | | Memory.fileMemory({ path }) | JSON file persistence | | memory.save(resourceId, threadId, role, content) | Store message | | memory.list(resourceId, threadId, limit?) | Retrieve thread history | | memory.search(resourceId, query) | Keyword search across threads | | memory.clear(resourceId, threadId) | Clear thread |

Attachments

agent.generate() accepts an attachments array. Uses any-markdown to process:

• Local file paths (.pdf, .docx, .xlsx, .html, .ipynb, images, .zip, text)
• Remote URLs (fetched, SSRF-protected)
• Buffer objects
• Plain text strings
• YouTube links (transcript extraction)

Guardrails

Hooks attached to the router that run before or after the provider call. Configured as an array in the router config with code as a string (compiled via new Function) or inline function.

• pre_call — receives { model, messages, tools, metadata }, must return same shape.
• post_call — receives the provider response, must return an output.
• Errors: GuardrailCompileError, GuardrailRuntimeError, GuardrailTimeoutError, GuardrailValidationError.

Nearnode IPC

Bidirectional local IPC via Unix domain sockets / Windows named pipes.

import { server, client } from 'llmjs2';

// Server
const srv = server('my-channel');
srv.handle('generate', async (payload) => ({ text: `Echo: ${payload.prompt}` }));
await srv.start();

// Client
const cli = client('my-channel');
const res = await cli.call('generate', { prompt: 'Hello' });
await cli.close();

CLI — HTTP + IPC server

# HTTP server
llmjs2 agent start agent.yaml -P 3099 -H 127.0.0.1

# IPC server only
llmjs2 agent start agent.yaml --ipc my-agent

# Both
llmjs2 agent start agent.yaml -P 3099 -H 127.0.0.1 --ipc my-agent

curl -X POST http://127.0.0.1:3099/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Hello","attachments":["https://example.com/chart.png"]}'

Logging / Verbosity

import { verbose } from 'llmjs2';
verbose('debug');   // 'debug' | 'info' | 'none'

Uses pino with pretty-printing in non-production environments.

Timeout & Retry

• Default timeout: 120s (configurable via LLM_TIMEOUT env or timeout in options).
• Retries: 2 attempts with exponential backoff (1s → 2s) on 5xx, 429, network errors, and timeouts via fetchWithRetry().

Environment Variables

| Variable | Description | |----------|-------------| | OPENAI_API_KEY | OpenAI API key | | OPENAI_BASE_URL | OpenAI base URL (default: https://api.openai.com/v1) | | OPENAI_DEFAULT_MODEL | OpenAI default model | | NVIDIA_API_KEY | Nvidia API key | | NVIDIA_BASE_URL | Nvidia base URL | | NVIDIA_DEFAULT_MODEL | Nvidia default model | | OPEN_ROUTER_API_KEY | OpenRouter API key | | OPEN_ROUTER_BASE_URL | OpenRouter base URL | | OPEN_ROUTER_DEFAULT_MODEL | OpenRouter default model | | ZEN_API_KEY | OpenCodeZen API key | | ZEN_BASE_URL | OpenCodeZen base URL | | ZEN_DEFAULT_MODEL | OpenCodeZen default model | | BIGMODEL_API_KEY | BigModel (Zhipu AI) API key | | BIGMODEL_BASE_URL | BigModel base URL | | BIGMODEL_DEFAULT_MODEL | BigModel default model | | OLLAMA_API_KEY | Ollama API key | | OLLAMA_BASE_URL | Ollama base URL | | OLLAMA_DEFAULT_MODEL | Ollama default model | | LLM_DEFAULT_MODEL | Fallback model (provider/model_name) | | LLM_TIMEOUT | Global timeout in ms (default: 120000) |

API Reference (exports)

| Export | Description | |--------|-------------| | completion(promptOrOptions) | Single-call LLM completion; string → text, object → full response | | router(configOrPath) | Build router from object or JSON/JS file path | | Agent | Multi-turn agent class with tools, memory, attachments | | createAgent(config) | Deprecated — use new Agent(config) | | Agent.load(path) | Load agent from YAML config | | Memory | Namespace with inMemory() / fileMemory() | | inMemory() / fileMemory() | Standalone memory factory functions | | verbose(level) | Set logger verbosity | | listProviders() | Return supported provider prefixes | | listModels(provider, apiKey?) | Fetch model list from provider API | | NearnodeServer / NearnodeClient | IPC server/client classes | | server(name) / client(name) | IPC factory functions | | convertToOpenAITools(tools) | Convert tool definitions to OpenAI schema |

Ollama Response Normalization

Ollama responses are normalized to OpenAI-compatible shape (.choices[0].message, .usage.prompt_tokens, etc.) for uniform downstream handling.

Build & Publish

• Build: npm run build (tsup → dist/)
• Entry points: CJS ./dist/index.js, ESM ./dist/index.mjs, types ./dist/index.d.ts
• CLI: ./dist/cli.js (registered as llmjs2 bin)

Providers

| Prefix | Provider | API Base | |--------|----------|----------| | openai | OpenAI | https://api.openai.com/v1 | | nvidia | Nvidia | https://integrate.api.nvidia.com/v1 | | openrouter | OpenRouter | https://openrouter.ai/api/v1/chat/completions | | zen | OpenCodeZen | https://opencode.ai/zen/v1 | | bigmodel | BigModel (Zhipu AI) | https://open.bigmodel.cn/api/paas/v4/chat/completions | | ollama | Ollama | https://ollama.com |

License

ISC