@cool-ai/beach-llm

Owns the LLM-shaped participant runtime — the respond() discipline, scoped tools, and approval interception. An LLM-backed handler is an EventHandler like any other; this package builds it. The rules here (structured respond() calls, no free text, tool-scoped capability) are what make LLMs composable with deterministic handlers and durable processes.

Home: cool-ai.org · Documentation: cool-ai.org/docs

Install

npm install @cool-ai/beach-llm @anthropic-ai/sdk

What this package provides

• createAnthropicProvider(opts?) — factory that creates an AnthropicProvider without exposing @anthropic-ai/sdk to consumer code. Preferred over constructing AnthropicProvider directly.
• AnthropicProvider — LLMProvider wrapping @anthropic-ai/sdk. Full extended-thinking support.
• VercelAIProvider — LLMProvider wrapping the Vercel AI SDK (ai). 100+ models across OpenAI, Google, Mistral, Meta, Cohere, and others. Does not support Anthropic extended thinking — use AnthropicProvider for that.
• ToolRegistry — declares every tool available in the system. Handler configs select by name.
• createLLMHandler(opts) — factory that returns an EventHandler whose body runs the respond() loop. Register it exactly like a deterministic handler: router.register(id, handler).
• runRespondLoop(options) — drives an LLM through its full tool-use loop until respond() is called. Used inside createLLMHandler; call directly only when you don't need the handler wrapper.
• LLMHandlerConfig — the configuration shape for an LLM-backed handler.
• System-prompt snippets — importable text that teaches the LLM the respond() tool and turn states.
• LLMProvider interface — implement to add other model providers.

LLMHandlerConfig

import type { LLMHandlerConfig } from '@cool-ai/beach-llm';
import { respondToolSnippet, conversationStatesSnippet } from '@cool-ai/beach-llm';

const handlerConfig: LLMHandlerConfig = {
  id: 'baxter',
  model: 'claude-haiku-4-5',
  systemPrompt: [
    respondToolSnippet,         // required — teaches respond() shape
    conversationStatesSnippet,  // required — teaches valid conversationState values
    'You are Baxter, a personal productivity assistant.',
  ].join('\n\n'),
  tools: ['task_list', 'task_create'],  // names from ToolRegistry; respond() is injected automatically
  maxTokens: 4096,      // optional; defaults apply per provider
  temperature: 0,       // optional

  // Domain-data enforcement:
  domainDataSchema: {   // optional — embedded in respond() input_schema; LLM must produce conforming output
    type: 'object',
    properties: {
      tasks: { type: 'array', items: { type: 'object' } },
    },
    required: ['tasks'],
  },
  domainDataMergeStrategy: 'replace',  // optional — 'replace' | 'append' | 'deep-merge'; default 'replace'
};

respond() is injected automatically — do not list it in tools.

Typed state via stateSchema

LLMHandlerConfig is generic over a TState extends Record<string, string> parameter, defaulting to Record<string, string>. Declare stateSchema to type the handler's context.session.state with typed coordinates:

import type { LLMHandlerConfig, StateSchema } from '@cool-ai/beach-llm';

type ConciergeState = {
  mode: 'discover' | 'search' | 'build';
  subMode: 'initial' | 'results_landed' | 'basket_added';
};

const conciergeStateSchema: StateSchema<ConciergeState> = {
  coordinates: {
    mode:    { values: ['discover', 'search', 'build'], default: 'discover' },
    subMode: { values: ['initial', 'results_landed', 'basket_added'], default: 'initial' },
  },
};

const concierge: LLMHandlerConfig<ConciergeState> = {
  id: 'concierge',
  model: 'anthropic/claude-opus-4-7',
  systemPrompt: '...',
  tools: ['search', 'package_results'],
  stateSchema: conciergeStateSchema,
};

The handler reads context.session.state for the concierge coordinates — typed, no runtime undefined check.

The schema must match the same component's declaration in the router's state-machine.json (Beach does not cross-validate the two; the schema is the compile-time view of the same coordinates declared in the runtime config). See the state machines guide for the full picture.

ToolRegistry

import { ToolRegistry } from '@cool-ai/beach-llm';

const registry = new ToolRegistry();

registry.register({
  name: 'task_list',
  description: 'List open tasks for the current user.',
  scope: 'generalist',
  inputSchema: {
    type: 'object',
    properties: {
      limit: { type: 'number', description: 'Maximum number of tasks to return.' },
    },
  },
  handler: async (args, context) => {
    // context.sessionId, context.eventId, context.slotKey, context.signal,
    // context.routeEvent are available.
    return { tasks: await db.tasks.list({ limit: (args as { limit: number }).limit }) };
  },
});

The tool handler returns any JSON-serialisable value. The framework owns the dispatch through the event router (audit, gating, capability scoping); the tool owns the result computation. The result is passed back to the LLM as a tool result and the loop continues until respond() is called.

Registering a name that is already registered throws immediately. Unknown names in LLMHandlerConfig.tools throw at invocation time.

registry.unregister('task_list');  // remove a single tool
registry.clear();                  // remove all tools (useful in tests)

Tool scope and routing — the two-axis design

scope describes ownership; routing describes infrastructure. They are independent.

| Scope | Default routing | Bypass allowed | |---|---|---| | generalist | routed | No — generalist is the trust-gate scope; bypass would defeat the invariant | | specialist | routed | Yes, with an articulated bypassRouting.reason |

A specialist tool is one that operates on a private substrate the consumer team owns; the framework still wraps the call for audit and gating unless the tool elects bypass. A specialist tool that wants framework routing (the default) only needs justification. A specialist tool that elects bypass needs both justification (why specialist) and bypassRouting.reason (why bypass).

// Generalist — the recommended default for any tool that touches shared data.
registry.register({
  name: 'task_list',
  scope: 'generalist',
  description: 'List open tasks',
  inputSchema: { /* ... */ },
  handler: async (args) => db.tasks.list(args),
});

// Specialist with default routing — articulated justification required.
registry.register({
  name: 'imap_fetch',
  scope: 'specialist',
  justification: 'Operates on the researcher\'s private IMAP cache; not part of the user-visible capability surface',
  description: 'Fetch raw IMAP message bytes',
  inputSchema: { /* ... */ },
  handler: async (args) => fetchImapBytes(args),
});

// Specialist electing bypass for inner-loop latency — both reasons required.
registry.register({
  name: 'cache_lookup',
  scope: 'specialist',
  routing: 'bypass',
  justification: 'Operates on a process-local cache substrate not part of the consumer\'s public surface',
  bypassRouting: {
    reason: 'Sub-millisecond latency required for the researcher inner loop; routing overhead exceeds the per-call budget',
  },
  description: 'Look up a value in the in-process cache',
  inputSchema: { /* ... */ },
  handler: async (args) => cache.get((args as { key: string }).key),
});

The framework rejects ill-formed declarations at app startup, not at first invocation:

• scope: 'generalist' with routing: 'bypass' → registration error.
• Specialist without justification, or with a placeholder string ('TODO', 'tbd', 'fix me', etc.) → registration error.
• routing: 'bypass' without bypassRouting.reason → registration error.

Generalist tools may set peerExposed: true to publish on the consumer's Surface Card when that infrastructure ships. Specialist tools cannot set peerExposed: true — specialist scope means private substrate, which is by construction not federation-shaped.

Async tools — ctx.routeEvent

A tool that needs to dispatch async work (research, multi-hop fetches, anything taking seconds-to-minutes) calls ctx.routeEvent and returns an ack. The handler's turn proceeds; the actual result lands later as a routed event triggering a new handler invocation.

registry.register({
  name: 'email_research',
  scope: 'generalist',
  description: 'Search the user\'s email accounts for a topic',
  inputSchema: { /* ... */ },
  handler: async (args, ctx) => {
    const searchId = randomUUID();
    await ctx.routeEvent!({
      source: 'assistant',
      eventType: 'email_research_started',
      data: { ...(args as object), searchId },
    });
    return { searchId };
  },
});

The framework does not auto-attach destinations to the eventual result event — that's the routing config's job. See documentation/changePlans/cr-155-framework-enforced-routing.md for the locked design and documentation/migrations/cr-155-framework-enforced-routing.md for the migration walkthrough.

AnthropicProvider

import { createAnthropicProvider } from '@cool-ai/beach-llm';

const provider = await createAnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY });

createAnthropicProvider wraps the SDK constructor so consumer code never imports @anthropic-ai/sdk directly. Options: apiKey, baseURL, maxRetries, timeout, defaultHeaders, defaultQuery — all optional (the SDK defaults apiKey to process.env.ANTHROPIC_API_KEY).

Use AnthropicProvider (and therefore createAnthropicProvider) for all Anthropic models, including those with extended thinking enabled. It preserves thinking block signatures across multi-turn tool-use loops.

VercelAIProvider

import { generateText, jsonSchema } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';
import { VercelAIProvider } from '@cool-ai/beach-llm';

const provider = new VercelAIProvider(
  createOpenAI()('gpt-4o'),
  { generateText, jsonSchema },
);

import { generateText, jsonSchema } from 'ai';
import { createGoogleGenerativeAI } from '@ai-sdk/google';
import { VercelAIProvider } from '@cool-ai/beach-llm';

const provider = new VercelAIProvider(
  createGoogleGenerativeAI()('gemini-2.0-flash'),
  { generateText, jsonSchema },
);

VercelAIProvider takes the model instance and the two Vercel AI SDK functions it needs (generateText and jsonSchema). Beach does not import ai directly — only the consumer does, meaning ai is not a required install for users of AnthropicProvider.

Install the Vercel AI SDK and the relevant provider package:

npm install ai @ai-sdk/openai       # OpenAI / Azure
npm install ai @ai-sdk/google       # Gemini
npm install ai @ai-sdk/mistral      # Mistral
# etc.

LLMProvider interface

To add other model providers, implement:

interface LLMProvider {
  complete(options: CompletionOptions): Promise<CompletionResult>;
}

CompletionOptions carries the model, system prompt, messages, and tool schemas. CompletionResult carries stop reason, tool calls, text blocks, reasoning blocks, and token usage. Pass your implementation to createLLMHandler() or runRespondLoop().

LLM round-trip observer (CAIB-280)

AnthropicProvider, VercelAIProvider, and @cool-ai/beach-llm-mastra's MastraProvider accept an optional observer at construction. The observer fires onRequest before the SDK call and onResponse after it settles, paired by a generated requestId. Provider errors propagate; onResponse does not fire for failed calls.

import { AnthropicProvider, type LLMObserver } from '@cool-ai/beach-llm';

const observer: LLMObserver = {
  onRequest:  (e) => console.log(`→ ${e.providerName}/${e.model} (${e.requestSummary.messageCount} msgs)`),
  onResponse: (e) => console.log(`← ${e.providerName} ${e.durationMs}ms · ${e.usage.outputTokens} tokens out`),
};

const provider = new AnthropicProvider(sdk, { observer });

@cool-ai/beach-inspect's wireInspect(store) exposes a ready-to-pass llmObserver that writes inspect:llm-request / inspect:llm-response missives — these land under the externalApi detail group in the side panel:

import { wireInspect } from '@cool-ai/beach-inspect';
import { AnthropicProvider } from '@cool-ai/beach-llm';

const { routerOptions, llmObserver } = wireInspect(missiveStore);
const provider = new AnthropicProvider(sdk, { observer: llmObserver! });

requestId correlation lets observers pair onRequest and onResponse for the same call when a single provider instance serves concurrent requests. The constructor-bound shape (rather than per-call observation) keeps the provider's call signature unchanged.

System-prompt snippets

import { respondToolSnippet, conversationStatesSnippet } from '@cool-ai/beach-llm';

respondToolSnippet explains the respond() tool structure. conversationStatesSnippet explains valid conversationState values. Both belong in every LLM handler's system prompt — without them the LLM does not know it must call respond() instead of replying with free text.

createLLMHandler()

Builds the EventHandler for an LLM-backed handler. Register it exactly like a deterministic handler. The handler renders the inbound session:request parts into a message, runs its context-builders, drives the respond() loop, and replies by routing a session:reply event — it never reads session.destinations.

import { createLLMHandler } from '@cool-ai/beach-llm';

const concierge = createLLMHandler({
  config: handlerConfig,
  provider,
  registry,
  contextBuilders: [async ({ sessionId }) => historyStore.load(sessionId)],  // optional — context injected ahead of the inbound parts
  timeoutMs: 60_000,                                                          // optional — wall-clock budget per invocation
  onToolExecution: async (record) => auditLog.write(record),                 // optional — fires after every tool call
});

router.register('concierge', concierge);

The handler reads context.session, drives its own timeout / durable checkpointing / cancellation, and replies by routing a session:reply event. The event-router delivers the reply to each of the session's destinations.

runRespondLoop()

Drives an LLM through its tool loop until it calls respond(). createLLMHandler uses it internally; call it directly only when you don't need the handler wrapper.

import { runRespondLoop } from '@cool-ai/beach-llm';
import type { SpecialistExecutionRecord } from '@cool-ai/beach-llm';

const result = await runRespondLoop({
  config: handlerConfig,
  messages: [{ role: 'user', content: 'Hello' }],
  sessionId: 'my-session',
  registry,
  provider,
  signal: abortController.signal,  // optional — passed to tool handlers
  onTextBlock: (text) => { ... },  // optional — fires for interim text before respond()
  onToolExecution: async (record: SpecialistExecutionRecord) => {
    // Fires after every tool execution — use for audit/replay log entries.
    // record.toolName, record.toolInput, record.toolOutput, record.durationMs,
    // record.handlerId, record.sessionId, record.eventId, record.iteration

    // record.error is set (string) when the tool threw.
    //
    // Routing-audit fields: record.scope ('generalist' | 'specialist'),
    // record.routing ('routed' | 'bypass'), record.bypass (boolean),
    // record.bypassReason (the articulated reason when bypass is true),
    // record.registrationSite (best-effort 'file:line' from the registration
    // site), record.tags (the tool's declared tags).
    await auditLog.write(record);
  },
});

// result.respond   — the RespondCall from the LLM
// result.messages  — full message thread after the tool loop
// result.usage     — { inputTokens, outputTokens }
// result.latencyMs
// result.eventId   — resolved (generated if not supplied)

HITL approval

Tools declare requiresApproval to gate execution on human approval before the handler runs.

registry.register({
  name: 'book_flight',
  // ...
  requiresApproval: true,  // always requires user-level approval
});

For context-dependent requirements, pass an ApprovalPolicy function instead:

import type { ApprovalPolicy } from '@cool-ai/beach-llm';

const bookingPolicy: ApprovalPolicy = async ({ args }, context) => {
  if ((args as { totalValue: number }).totalValue > 500) return 'user';
  return 'auto';
};

registry.register({
  name: 'book_flight',
  // ...
  requiresApproval: bookingPolicy,
});

ApprovalLevel:

• 'auto' — no approval needed; handler executes immediately
• 'user' — requires user approval
• 'admin' — requires admin approval

true is shorthand for always 'user'. Absent or false-y is always 'auto'.

Wire the intercept with withApprovalIntercept(), providing a callback that emits the approval-request part and waits for the user's response:

import { withApprovalIntercept } from '@cool-ai/beach-llm';

const intercepted = withApprovalIntercept(tool, {
  onApprovalRequired: async (request) => {
    // request.level — 'user' | 'admin'
    // request.toolName, request.toolInput, request.approvalId
    // Emit approval-request part, wait for decision, return:
    return { approvalId: request.approvalId, decision: 'approved' };
  },
});

Not in this package

• Event routing and session lifecycle (@cool-ai/beach-core).
• Channel rendering and delivery (the @cool-ai/beach-channel-* adaptors and @cool-ai/beach-a2ui-renderer-*).

Related

• https://cool-ai.org/docs/respond-tool — the respond() tool schema.
• https://cool-ai.org/docs/design-principles — principles 2.3, 2.4 (LLMs never emit free text; tools constrain and prompts guide).