chattydeer

A Node.js chat completions toolkit

A Node.js LLM chat toolkit built on top of @jsilvanus/embedeer.

Provides Explainer, LLMAdapter, ChatSession, prompt utilities, and higher-level agentic helpers for multi-turn tool-calling. New additions include createChatProvider, runAgentLoop, and createOpenAiChatHandler (OpenAI /v1/chat/completions handler).

Install

npm install @jsilvanus/chattydeer

Usage

Single-turn explanation

import { Explainer } from '@jsilvanus/chattydeer';

const explainer = await Explainer.create('llama-3.2-3b', { deterministic: true }); // deterministic: stable, reproducible outputs (useful for tests)

const result = await explainer.explain({
  task: 'narrate',                     // intent: what you want the explainer to produce (e.g. 'summarize', 'narrate')
  domain: 'evolution',                 // domain helps choose domain-specific phrasing or templates
  context: { filePath: 'src/auth/handler.ts' }, // optional contextual metadata (file path, URL, etc.)
  evidence: [
    // Evidence items: structured blocks the explainer will reason over
    { id: 1, source: 'src/auth/handler.ts', excerpt: '2024-03-15 *** LARGE CHANGE' },
  ],
  maxTokens: 256,                       // limit the response length (tokens)
});

console.log(result.explanation); // "The auth handler underwent a major rewrite..."
await explainer.destroy();

Multi-turn chat with function calling (agentic loop)

import { ChatSession } from '@jsilvanus/chattydeer';

const session = await ChatSession.create('llama-3.2-3b', {
  systemPrompt: 'You are a gitsema guide assistant. Use tools to answer questions.',
  tools: [
    {
      name: 'semantic_search',
      description: 'Search the codebase semantically by natural-language query.',
      parameters: {
        type: 'object',
        properties: { query: { type: 'string' } },
        required: ['query'],
      },
    },
    {
      name: 'recent_commits',
      description: 'Return the N most recent commits touching a file.',
      parameters: {
        type: 'object',
        properties: {
          filePath: { type: 'string' },
          n:        { type: 'number' },
        },
        required: ['filePath'],
      },
    },
  ],
});

// The session runs an agentic loop:
//   LLM requests tool calls → ChatSession executes them → results fed back → repeat
const answer = await session.send('Which files changed most in the last month?', {
  executeTool: async (name, args) => {
    if (name === 'semantic_search')  return mySearch(args.query);
    if (name === 'recent_commits')   return myCommits(args.filePath, args.n);
    throw new Error(`Unknown tool: ${name}`);
  },
  maxIterations: 10, // safeguard against infinite loops
});

console.log(answer);

// Continue the conversation in the same session
const followUp = await session.send('Can you summarise the top file in one sentence?', {
  executeTool: async (name, args) => { /* ... */ },
});

await session.destroy();

Preloading / warming a model

To reduce the latency of the first request you can preload a model into the underlying embedding/generation layer. Two common options:

• Use @jsilvanus/embedeer directly to download and cache a model ahead of time:

import { loadModel } from '@jsilvanus/embedeer';

// downloads and caches the model files so subsequent creation is fast
await loadModel('Xenova/all-MiniLM-L6-v2', { token: process.env.HF_TOKEN });

• Or create and initialize a LLMAdapter/ChatSession once at startup and reuse it:

import { LLMAdapter, ChatSession } from '@jsilvanus/chattydeer';

// warm a generator pipeline (keeps it in memory)
const adapter = await LLMAdapter.create('llama-3.2-3b', { token: process.env.HF_TOKEN });

// reuse adapter for sessions
const session = await ChatSession.create('irrelevant', { adapter });
// later: session.send(...)

Both approaches reduce cold-start latency. Use loadModel() when you only need to ensure model artifacts are present on disk; use LLMAdapter.create() when you want an initialized in-process generator ready to serve requests.

API

Explainer

• Explainer.create(modelName, opts) — create an explainer bound to a model
• explainer.explain(request) — explain using structured evidence blocks; returns { explanation, labels, references, meta }
• explainer.destroy() — release underlying resources

ChatSession

• ChatSession.create(modelName, opts) — create a session; accepts tools, systemPrompt, adapter
• session.send(userMessage, opts) — send a message and run the agentic tool-call loop; returns a final plain-text answer
  • opts.executeTool — async (name, args, callId) => result — called for each tool the LLM requests
  • opts.maxIterations — loop guard (default 10)
  • opts.maxTokens — tokens per LLM call (default 512)
• session.history — read-only snapshot of all ChatMessage objects in the conversation
• session.destroy() — release underlying resources

Additional ChatSession notes:

• session.append(msg) — append a ChatMessage to the session (useful for programmatic injection of tool or system messages).
• Chat history is kept in-memory on the ChatSession instance (the _history array). session.history returns a snapshot copy. There is no built-in durable persistence; to persist history, serialize session.history yourself and replay or rehydrate into a new ChatSession.

LLMAdapter

• LLMAdapter.create(modelName, opts) — low-level text-generation adapter
• adapter.generate(prompt, opts) — returns { text, raw, meta }
• adapter.destroy() — release underlying resources

Utilities

• explainForGitsema(payload, opts) — gitsema-compatible adapter
• renderTemplate(domain, vars) — render a domain-specific prompt template
• estimateTokensFromChars(chars) / trimEvidenceForBudget(prelude, evidence, budget) — prompt utilities

Agentic helpers

• createChatProvider(httpUrl, model, apiKey?, opts?) — factory that returns a ChatCompletionProvider with complete() and stream() methods for any OpenAI-compatible endpoint (OpenAI, Ollama /v1, vLLM, llama.cpp server). opts accepts timeoutMs (default 60000) and maxRetries (default 2, applied to HTTP 429/5xx with exponential backoff). Pass signal on the request to support cancellation.
• runAgentLoop(session, opts) — run an agentic tool-calling loop using a ChatCompletionProvider. Options include provider, tools, executeTool(call: ToolCall): Promise<string>, maxRoundtrips (default 5), maxTokens, temperature, onMessage, and redactContent (a hook applied to every message's content — including tool results — before it leaves the process). Per-tool-call errors become role: 'tool' result messages instead of crashing the loop. If maxRoundtrips is exhausted before a final text answer, the loop returns the best-available answer and history rather than throwing.
• createOpenAiChatHandler(provider, tools?, executeTool?, opts?) — returns an Express RequestHandler implementing a lightweight subset of OpenAI's POST /v1/chat/completions API that maps requests to runAgentLoop. The handler honors the redactContent hook and is suitable for mounting inside an existing HTTP server.
• createAgentSession(opts?) — returns a minimal { history, append(msg) } session object accepted by runAgentLoop and createChatProvider, without requiring an LLM adapter. Accepts systemPrompt and an initial messages array.

OpenAI tool-calling wire format

ChatMessage/ToolCall objects are mapped to OpenAI's chat-completions shape on the wire:

• Tool definitions passed via tools are sent as body.tools: [{ type: 'function', function: { name, description, parameters } }] (the modern OpenAI/Ollama/vLLM tool-calling shape).
• An assistant message with toolCalls is sent as { role: 'assistant', content, tool_calls: [{ id, type: 'function', function: { name, arguments: JSON.stringify(args) } }] }.
• A role: 'tool' result message (with toolCallId set to the corresponding ToolCall.id) is sent as { role: 'tool', tool_call_id, content }.
• Responses are parsed back from message.tool_calls (or the legacy function_call) into ChatMessage.toolCalls, each with a generated id if the server didn't provide one.

This means any OpenAI-compatible endpoint (OpenAI, Ollama, vLLM, llama.cpp server) can be used as the provider for runAgentLoop.

Agent loop example

import { createChatProvider, runAgentLoop, createAgentSession } from '@jsilvanus/chattydeer';

const provider = createChatProvider('http://localhost:11434', 'llama3.1', undefined, { timeoutMs: 30_000 });

const session = createAgentSession({
  systemPrompt: 'You are a helpful assistant with access to tools.',
});
session.append({ role: 'user', content: 'What does the auth module do?' });

const tools = [
  { name: 'semantic_search', description: 'Search the codebase semantically', parameters: { type: 'object', properties: { query: { type: 'string' } }, required: ['query'] } },
];

const result = await runAgentLoop(session, {
  provider,
  tools,
  maxRoundtrips: 5,
  redactContent: (text) => text.replace(/sk-[a-zA-Z0-9]+/g, '[REDACTED]'),
  async executeTool(call) {
    if (call.name === 'semantic_search') return JSON.stringify(await mySearch(call.arguments.query));
    return `Unknown tool: ${call.name}`;
  },
});

console.log(result.answer);
await provider.destroy();

See explainer-contract.md for the full Explainer interface contract.

License

MIT