@xeonr/upload-pool-sdk

Self-hosted worker SDK for the upl.im content type pipeline. Implement a handler per content type URN; the SDK handles SSE connection, job acceptance, presigned thumbnail upload, and metadata callbacks.

Install

npm install @xeonr/upload-pool-sdk

Requires Node 20+.

Usage

import { createPool, NonRetryableError } from "@xeonr/upload-pool-sdk";

await createPool({
  token: process.env.UPL_POOL_TOKEN!,         // tpq_... from the pool's creation response
  endpoint: process.env.UPL_API_URL!,
  concurrency: 4,
  handlers: {
    "my-integration:invoice": async (ctx) => {
      const buf = await ctx.downloadBuffer();
      const thumb = await renderInvoiceThumbnail(buf);

      await ctx.uploadMeta({ type: "THUMBNAIL_LIGHT", data: thumb });
      await ctx.setMetadata({ thumbnailGenerationVersion: 3 });
    },
  },
  onError(err, ctx) {
    console.error(`job ${ctx?.jobId} (${ctx?.contentType.urn}) failed:`, err);
  },
}).start();

The SDK advertises the URNs you register handlers for as ?capabilities=... on the SSE connect URL, so the pipeline only dispatches matching jobs to this worker process.

Concepts

Pool token

The opaque tpq_… token authenticates the SSE connection. Mint one via the integration admin UI; the token is shown once at creation. Store it as a deployment secret.

Per-job update token

Embedded in every job envelope as updateToken. It's the existing 24h JWT scoped to the upload (subject = uploadId, audience = internal-uploads-endpoint) and authenticates the four InternalUploadsService callbacks the SDK makes on your behalf — RequestMetaUpload, ConfirmMetaUpload, UpdateUpload, GetProcessingContext. You never see it directly; the SDK uses it internally.

Handler contract

Handlers are keyed by content type URN (e.g. default:image, my-integration:invoice). On dispatch, the SDK:

1. Calls AcceptJob to clear the pipeline's accept-timeout.
2. Invokes your handler with a ctx providing source download + thumbnail upload + metadata callbacks.
3. On clean resolve, calls CompleteJob. On throw, calls ReportError with retry=true.
4. Throw NonRetryableError to signal permanent failure (the pipeline won't requeue).

API

createPool(config: PoolConfig): Pool

See src/types.ts for the full PoolConfig, JobContext, and UploadMetaOpts types.

| ctx method | What it does | |---------------------|---------------------------------------------------------------| | download() | Stream the source file from the presigned URL. | | downloadBuffer() | Buffer the entire source file in memory. | | downloadToFile() | Stream to a local path on disk. | | uploadMeta() | RequestMetaUpload → S3 PUT → ConfirmMetaUpload (atomic). | | setMetadata() | Write upload-row metadata (dimensions, duration, page count). | | setDescription() | Write AI description + tags via UpdateUpload. |

Reference example

See examples/noop-worker.ts — a 35-line worker that echoes source bytes back as both thumbnail variants. Useful for integration testing.

Build

npm run build           # tsc → dist/
npm run build:check     # type-check only
npm run test            # vitest