Long Tail

The queue durable systems forgot.

Durable platforms queue deterministic work well — retries, timeouts, exactly-once — and give you await condition() to park a workflow until a signal arrives. But the wait itself is invisible: nothing to find, claim, or measure while it sits there.

Long Tail's conditional is that same wait made visible. Waiting mints a row that is searchable, claimable, deadlined, and role-gated on a shared metadata surface. A person or a machine answers it; the workflow resumes exactly where it paused.

npm install @hotmeshio/long-tail

How it works

One workflow, both queues: proxyActivities for the deterministic one, conditional for non-determinism.

import { Durable } from '@hotmeshio/hotmesh';
import { conditional, type LTEnvelope } from '@hotmeshio/long-tail';
import * as activities from './activities';

const { analyzeContent } = Durable.workflow.proxyActivities<typeof activities>({ activities });

export async function reviewContent(envelope: LTEnvelope) {
  // activity calls are checkpointed and crash-safe
  const analysis = await analyzeContent(envelope.data.content);

  if (analysis.confidence >= 0.85) {
    return { data: { approved: true, analysis } };
  }

  // low confidence: pause as a claimable, role-gated escalation
  const { workflowId } = Durable.workflow.workflowInfo();
  const decision = await conditional<{ approved: boolean; notes?: string }>(
    `review-${workflowId}`,
    {
      role: 'reviewer',
      type: 'content-review',
      priority: 2,
      description: `Confidence ${analysis.confidence} — needs a human`,
      metadata: { contentId: envelope.data.contentId },
      envelope: { data: envelope.data, analysis },
      schemaVersion: 1,   // the reviewer form edition this code is written for
    },
  );

  return { data: { approved: decision && decision.approved, analysis } };
}

The reviewer role owns a versioned form schema. The platform renders it as the resolver's UI, validates every submission against it, and schemaVersion pins the edition this code expects — evolve the form and the payload type together, and rows in flight keep the form they were minted for:

{
  "title": "Content Review",
  "required": ["approved"],
  "properties": {
    "approved": { "type": "boolean", "description": "Approve this content?" },
    "notes":    { "type": "string", "format": "textarea", "description": "What you saw" }
  }
}

Two surfaces, one model. A proxyActivity targets a machine: call it, get a result. conditional targets the external world — a reviewer, an operator, a factory cell. It suspends the workflow and writes a single escalation row carrying everything needed to route the work: the role that should act, its type and priority, and any metadata to display or filter on. People work that row through an RBAC-scoped surface — find it, claim it, resolve it — from the dashboard, the API, or MCP, and resolving it resumes the workflow exactly where it paused.

Activities are plain functions:

export async function analyzeContent(content: string) {
  const result = await llm.classify(content);
  return { confidence: result.confidence, flags: result.flags };
}

What the primitive buys

The non-deterministic queue has qualities the deterministic one never needed:

• RBAC is intrinsic. The queue is a role; permission lives in the queue itself. Any member answers — a person in the dashboard, a service account through the API — and the workflow is indifferent to which.
• The form is data. A versioned JSON Schema on the role renders the resolver's UI — fields, validation, conditionals, layout — with zero frontend code. Assign an escalation to one named user with self scope and you have a just-in-time form: one person, one item, one auditable edition.
• Claims are locks. A claim is a TTL window with an extend prompt before it lapses and a resolve guard behind it — the platform rejects a submission against an expired claim atomically, in the same statement that settles the row. Stale work cannot land.
• Cancel is home. Cancellation is a first-class settlement, not an exception. When the pressure no longer applies, the waiting flow sees null and reconciles to a modeled resting state. Handled is broader than resolved — timeout, cancel, and hop-onward are all endings the loop was built to reach.
• One shared surface. Every actor that touches an item writes to the same metadata-keyed row. The contact point is shared by construction, so there is one place to search, one place to claim, one place to audit.
• History accretes. The surface only ever adds: intent at creation, outcome at settlement, every crossing in between. The object's whole history sits there to replay — frames the queue kept for you.
• Objects get lifecycles. Give an object this queue and a while loop and it lives — a digital twin: it advertises when free, asks when it needs service, waits for reality's answer, reconciles when reality gives one. An order, a document only compliance may edit, a machine on a line — all the same shape, an object looping at a role.
• What repeats, stops needing people. Every settled row is a worked example. Recurring patterns become deterministic tools — problems that once required a person, then required AI reasoning, eventually require neither.

Start

Point at Postgres. Everything else is optional.

import { start } from '@hotmeshio/long-tail';

const lt = await start({
  database: { host: 'localhost', port: 5432, user: 'postgres', password: 'password', database: 'mydb' },
  workers: [{ taskQueue: 'default', workflow: reviewContent }],
});

Dashboard at http://localhost:3000. The boilerplate has a working project with workflows, MCP servers, and MinIO.

Register MCP tools

Long Tail connects to any MCP server. Registered tools become durable activities.

Existing package — no code:

curl -X POST http://localhost:3000/api/mcp/servers \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "filesystem",
    "transport_type": "stdio",
    "transport_config": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/data"] },
    "tags": ["files", "storage"],
    "auto_connect": true
  }'

Remote server — point at a URL:

curl -X POST http://localhost:3000/api/mcp/servers \
  -d '{ "name": "my-python-server", "transport_type": "sse", "transport_config": { "url": "http://python-service:8000/mcp" } }'

In-process — write your own:

import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import { registerMcpTool } from '@hotmeshio/long-tail';

export function createImageToolsServer(): McpServer {
  const server = new McpServer({ name: 'image-tools', version: '1.0.0' });

  registerMcpTool(server, 'resize_image', 'Resize an image.', {
    path: z.string().describe('Path to the image'),
    width: z.number().optional(),
    height: z.number().optional(),
  }, async (args: any) => ({
    content: [{ type: 'text', text: JSON.stringify(await resize(args)) }],
  }));

  return server;
}

const lt = await start({
  // ...
  mcp: { serverFactories: { 'image-tools': createImageToolsServer } },
});

All three paths produce the same outcome: tools callable as durable activities. See the MCP guide.

Full configuration

const lt = await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [{ taskQueue: 'default', workflow: reviewContent }],

  // Everything below is optional
  seed: { admin: { externalId: 'admin', password: process.env.ADMIN_PASSWORD } },
  mcp: { server: { enabled: true }, serverFactories: { 'my-tools': createMyToolsServer } },
  auth: { secret: process.env.JWT_SECRET },
  telemetry: { honeycomb: { apiKey: process.env.HNY } },
  logging: { pino: { level: 'info' } },
  maintenance: true,
});

Embed in an existing app

Long Tail runs as an embedded package inside NestJS, Express, or any Node.js application. No extra HTTP server, no extra ports.

import { start, createClient } from '@hotmeshio/long-tail';

const lt = await start({
  database: { connectionString: process.env.DATABASE_URL },
  server: { enabled: false },
  seed: { admin: { externalId: 'system' } },
  workers: [{ taskQueue: 'default', workflow: reviewContent }],
});

const client = createClient({ auth: { userId: lt.adminUserId } });

const tasks = await client.tasks.list({ status: 'completed', limit: 10 });
const result = await client.escalations.claim({ id: 'esc_123', durationMinutes: 30 });

Mount the dashboard at a subpath:

import { LTExpressAdapter } from '@hotmeshio/long-tail';

const adapter = new LTExpressAdapter();
adapter.setBasePath('/admin/longtail');
app.use('/admin/longtail', adapter.getRouter());

Subscribe to events with callbacks:

client.events.on('task.completed', (event) => console.log('done:', event.workflowId));
client.events.on('escalation.*', (event) => notifyTeam(event));

Every SDK call returns an LTApiResult — same status codes, same validation, same RBAC. See the SDK guide.

Deployment

Three modes from the same codebase:

// Standalone — dashboard + API + workers
await start({ database: { connectionString: process.env.DATABASE_URL } });

// Worker-only — no HTTP server
await start({ database: { connectionString: process.env.DATABASE_URL }, server: { enabled: false }, workers: [...] });

// Embedded — inside your app, SDK calls only
await start({ database: { connectionString: process.env.DATABASE_URL }, server: { enabled: false } });
const lt = createClient({ auth: { userId: 'service' } });

All modes share PostgreSQL and scale independently. See Cloud Deployment.

Docs

| Guide | What it covers | |-------|---------------| | The Long Tail Story | Why this exists, what accumulates over time | | Workflows | Activities, interceptor, escalation lifecycle, composition | | IAM | Identity propagation, service accounts, credential exchange | | Dashboard | Navigation, key pages, event feed | | MCP | Server registration, tool calls, human queue | | Compilation | Dynamic to deterministic pipeline wizard | | Compiler | ltc compile — durable TypeScript to YAML DAGs | | CLI | ltc — terminal access to workflows, escalations, knowledge, MCP | | Escalation Strategies | Default, MCP triage, custom handlers | | Schema Enforcement | form_schema as an enforced API contract on every resolve surface | | Code-Owned Configuration | configSource — declared config compared and applied at every boot | | Faceted Routing | Query and atomically claim the queue by facets; dispatcher pattern | | SDK | Embedded usage, createClient, event subscriptions | | Architecture | Project structure, conventions, discovery | | Cloud | AWS ECS, GCP Cloud Run, Docker | | Data Model | Database schema |

Adapters: Auth · Events · Telemetry · Logging · Maintenance · OAuth

HTTP API: Workflows · Tasks · Escalations · YAML Workflows · Users · Roles · Service Accounts · MCP Servers · Pipelines · Exports

SDK: Overview · Workflows · Tasks · Escalations · YAML Workflows · MCP · Events

Contributing

git clone https://github.com/hotmeshio/long-tail.git
cd long-tail
docker compose up -d --build

Open http://localhost:3000. Example workflows seed the dashboard.

| User | Password | Role | |------|----------|------| | superadmin | l0ngt@1l | superadmin | | admin | l0ngt@1l | admin | | engineer | l0ngt@1l | engineer | | reviewer | l0ngt@1l | reviewer |

See Contributing.

License

See LICENSE.