Long Tail

Long Tail turns your PostgreSQL database into a workflow engine where human and AI work share the same durable execution path. Hand off, pull back, escalate, automate — without rewriting systems or losing continuity.

npm install @hotmeshio/long-tail

Use Long Tail for

• Durable execution — Your functions run as workflows and checkpoint to Postgres. If the process crashes, execution resumes from the last completed step.
• Identity everywhere — Workflows know who started them, whose credentials govern their execution, and what permissions are in play. IAM is not bolted on — it's woven into every activity call.
• Human-in-the-loop — When confidence is low, the workflow escalates. RBAC-scoped escalation chains route work to the right reviewer. Approval workflows, content review, document verification — the pattern is the same.
• AI triage — When human-in-the-loop teams can't resolve a request, AI takes over. Its tool calls are checkpointed. And when the fix works, it compiles into a deterministic pipeline for next time.
• MCP tool orchestration — Describe what you need. If you've registered the tools, the Pipeline Designer builds the workflow. Every compiled pipeline deploys as a reusable MCP tool.

A dashboard, REST API, and live event stream ship with the package. Use what you need.

Start

Point at Postgres. Everything else is optional.

import { start } from '@hotmeshio/long-tail';
import * as myWorkflow from './workflows/my-workflow';

const lt = await start({
  database: { host: 'localhost', port: 5432, user: 'postgres', password: 'password', database: 'mydb' },
  workers: [{ taskQueue: 'default', workflow: myWorkflow.reviewContent }],
  auth: { secret: process.env.JWT_SECRET },
});

Dashboard at http://localhost:3000. The boilerplate has a working project with custom MCP servers, MinIO, and example workflows.

Write a Durable Workflow

A workflow receives an envelope and returns a result. Each activity call checkpoints — no work is lost, no step runs twice.

import { Durable } from '@hotmeshio/hotmesh';
import type { LTEnvelope } from '@hotmeshio/long-tail';
import * as activities from './activities';

const { analyzeContent } = Durable.workflow.proxyActivities<typeof activities>({ activities });

export async function reviewContent(envelope: LTEnvelope) {
  const analysis = await analyzeContent(envelope.data.content);

  if (analysis.confidence >= 0.85) {
    return { type: 'return' as const, data: { approved: true, analysis } };
  }

  return {
    type: 'escalation' as const,
    role: 'reviewer',
    message: `Review needed (confidence: ${analysis.confidence})`,
    data: { content: envelope.data.content, analysis },
  };
}

Activities are plain functions with side effects — API calls, LLM invocations, database queries. The proxyActivities call wraps them so the engine can checkpoint each result.

// activities.ts
export async function analyzeContent(content: string) {
  const result = await llm.classify(content);
  return { confidence: result.confidence, flags: result.flags };
}

Compile Workflows

Write durable workflows in TypeScript. Compile to YAML DAGs that run without replay overhead. ltc is the compiler — like tsc for workflows.

export ANTHROPIC_API_KEY=sk-ant-...

npx ltc compile workflows/              # scan and compile all workflow files
npx ltc compile --dry-run               # discover without compiling

The source is the spec. The compiled YAML is the optimized execution. Both live in the repo. See the Compiler Guide.

Certify a Workflow

Any durable workflow can be promoted to certified through the dashboard or API. A certified workflow gains interceptor guarantees: failures escalate instead of throwing, escalation chains route through roles, and every error is either handled or surfaced. It cannot silently fail.

curl -X PUT http://localhost:3000/api/workflows/reviewContent/config \
  -H "Authorization: Bearer $TOKEN" \
  -d '{ "invocable": true, "task_queue": "default", "default_role": "reviewer" }'

De-certifying removes the interceptor. The workflow continues as a standard durable workflow — same code, different guarantees.

Register MCP Servers

Long Tail connects to any MCP server — an npm package, a remote service, or one you write yourself. Registered tools become durable activities and are available to the Pipeline Designer.

Use an existing package — no code, just register:

curl -X POST http://localhost:3000/api/mcp/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "filesystem",
    "transport_type": "stdio",
    "transport_config": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/data"] },
    "tags": ["files", "storage"],
    "auto_connect": true
  }'

Connect a remote server — point at a URL:

curl -X POST http://localhost:3000/api/mcp/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "my-python-server",
    "transport_type": "sse",
    "transport_config": { "url": "http://python-service:8000/mcp" },
    "tags": ["ml", "classification"],
    "compile_hints": "Returns confidence scores. Use threshold 0.85 for auto-approve."
  }'

Write your own and register it in-process:

import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import { registerMcpTool } from '@hotmeshio/long-tail';

export function createImageToolsServer(): McpServer {
  const server = new McpServer({ name: 'image-tools', version: '1.0.0' });

  registerMcpTool(server, 'resize_image', 'Resize an image.', {
    path: z.string().describe('Path to the image'),
    width: z.number().optional().describe('Target width'),
    height: z.number().optional().describe('Target height'),
  }, async (args: any) => ({
    content: [{ type: 'text', text: JSON.stringify(await resize(args)) }],
  }));

  return server;
}

const lt = await start({
  // ...
  mcp: {
    serverFactories: { 'image-tools': createImageToolsServer },
  },
});

All three paths produce the same outcome: tools callable as durable activities. Tags enable discovery. Compile hints guide the compiler when tools are compiled into deterministic pipelines. See the MCP guide for the full registration lifecycle.

Ask It Anything

Once your tools are registered, the Pipeline Designer orchestrates them. Describe what you need in plain language:

"Log into localhost:3000 as superadmin, navigate to every page in the sidebar, and save a screenshot of each."

The system discovers the right MCP servers, calls the tools, chains the results. If it works, the compilation wizard converts the execution into a deterministic pipeline — parameterized inputs, typed schema, no LLM at runtime. It deploys as a new MCP tool that any workflow, agent, or API call can invoke.

The inventory of compiled tools grows over time. The need for LLM reasoning shrinks. Problems that once required a human, then required an AI, eventually require neither.

Full Configuration

const lt = await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [{ taskQueue: 'default', workflow: myWorkflow.reviewContent }],

  // Everything below is optional
  mcp: {
    server: { enabled: true },
    serverFactories: { 'my-tools': createMyToolsServer },
  },
  escalation: { strategy: 'mcp' },
  auth: { secret: process.env.JWT_SECRET },
  telemetry: { honeycomb: { apiKey: process.env.HNY } },
  logging: { pino: { level: 'info' } },
  maintenance: true,
});

Use as a Package (No HTTP Server)

Long Tail can run as an embedded package inside your existing application — no Express server, no socket.io, no extra ports. The same API surface is available as direct function calls.

import { start, createClient } from '@hotmeshio/long-tail';

await start({
  database: { connectionString: process.env.DATABASE_URL },
  server: { enabled: false },
  workers: [{ taskQueue: 'default', workflow: reviewContent.reviewContent }],
});

const lt = createClient({ auth: { userId: 'system' } });

// Same operations as the REST API — no HTTP overhead
const tasks = await lt.tasks.list({ status: 'completed', limit: 10 });
const result = await lt.escalations.claim({ id: 'esc_123', durationMinutes: 30 });

Subscribe to events with callbacks instead of socket.io:

lt.events.on('task.completed', (event) => {
  console.log('done:', event.workflowId);
});

lt.events.on('escalation.*', (event) => {
  notifyTeam(event);
});

Every SDK call returns an LTApiResult — same status codes, same validation, same RBAC. The transport is the only thing that changes. See the SDK guide for the full API reference.

Deployment

Three modes from the same codebase:

// 1. Standalone — dashboard + REST API + workers
await start({ database: { connectionString: process.env.DATABASE_URL } });

// 2. Worker-only — workflow execution, no HTTP server
await start({
  database: { connectionString: process.env.DATABASE_URL },
  server: { enabled: false },
  workers: [{ taskQueue: 'default', workflow: reviewContent.reviewContent }],
});

// 3. Embedded — inside your NestJS/Next.js/Express app, SDK calls only
await start({ database: { connectionString: process.env.DATABASE_URL }, server: { enabled: false } });
const lt = createClient({ auth: { userId: 'service' } });

All modes share PostgreSQL and scale independently. See Cloud Deployment.

Docs

| Guide | What it covers | |-------|---------------| | The Long Tail Story | Why this exists, what accumulates over time, what you own | | Workflows | Activities, interceptor, escalation lifecycle, composition | | IAM | Identity propagation, service accounts, credential exchange | | Dashboard | Navigation, key pages, event feed | | MCP | Server registration, tool calls, human queue | | Compilation | Dynamic → deterministic pipeline wizard | | Compiler | ltc compile — durable TypeScript to YAML DAGs | | CLI | ltc — terminal access to workflows, escalations, knowledge, MCP | | Escalation Strategies | Default, MCP triage, custom handlers | | SDK | Embedded usage, createClient, event subscriptions | | Architecture | Project structure, conventions, discovery | | Cloud | AWS ECS, GCP Cloud Run, Docker | | Data Model | Database schema |

Adapters: Auth · Events · Telemetry · Logging · Maintenance · OAuth

HTTP API: Workflows · Tasks · Escalations · YAML Workflows · Users · Roles · Service Accounts · MCP Servers · MCP Runs · Exports

SDK: Overview · Workflows · Tasks · Escalations · YAML Workflows · MCP · Events

Contributing

git clone https://github.com/hotmeshio/long-tail.git
cd long-tail
docker compose up -d --build

Open http://localhost:3000. Example workflows seed the dashboard.

| User | Password | Role | |------|----------|------| | superadmin | l0ngt@1l | superadmin | | admin | l0ngt@1l | admin | | engineer | l0ngt@1l | engineer | | reviewer | l0ngt@1l | reviewer |

See Contributing.

License

See LICENSE.