@thejob/data-processor-client

Typed TypeScript/JavaScript client for the thejob-data-processor HTTP API. External systems use this instead of hand-building fetch requests.

The contract is small. A consumer does two things:

1. Push data in with ingest(records, opts), optionally attaching a webhookConfig so each record's processed result is delivered to your URL.
2. Pull as a backstop with pollProcessed() to catch up on anything a webhook missed.

You never name a pipeline or manage runs, you ingest records under a shared type and receive the processed result. The shared type/source/webhookConfig go once in opts (sent once on the wire, not repeated per record). The client owns the base URL, JSON plumbing, the request envelope, cursor paging, and typed errors.

Zero runtime dependencies. Uses the global fetch (Node 18+, Deno, browsers, edge).

Install

npm install @thejob/data-processor-client

Usage

import { DataProcessorClient, DataProcessorError } from '@thejob/data-processor-client';

const dp = new DataProcessorClient({ baseUrl: 'http://localhost:3000' });

// 1. Push data in. Records share type/source/webhook via opts (sent once). Attach
//    a webhookConfig to have each record's processed result POSTed to your URL
//    (signed when you set a secret, so you can verify it). A repeated dedupKey is
//    reported as a duplicate, not re-ingested.
const report = await dp.ingest(
  [
    { dedupKey: 'job-123', payload: { url: '...' } },
    { dedupKey: 'job-124', payload: { url: '...' } },
  ],
  {
    type: 'job',
    source: 'scraper',
    webhookConfig: { url: 'https://my-app.example/hooks/data-processor', secret: 'whsec_...' },
  },
);
console.log(report.accepted, report.duplicates, report.failed);

// 2. Pull backstop: if your webhook was down (or you attached none), catch up.
//    The cursor is threaded across pages; pass a saved cursor to resume.
for await (const page of dp.pollProcessed({ type: 'job' })) {
  for (const record of page.records) {
    handle(record.payload);
  }
}

// Errors are typed.
try {
  await dp.records.get('does-not-exist');
} catch (e) {
  if (e instanceof DataProcessorError && e.status === 404) { /* not found */ }
}

API surface

| Group | Methods | Purpose | | --------- | -------------------------------------------------------- | ------- | | ingest | ingest, records.list, records.get | push data in (shared type/source/webhook via opts), inspect what you sent | | processed | getProcessed, pollProcessed, getProcessedRecord | pull backstop for results | | health | health | liveness |

Any non-2xx response throws a DataProcessorError carrying .status and the parsed .body.

Configuration

new DataProcessorClient({
  baseUrl: 'https://data-processor.internal',
  headers: { 'x-some-header': 'value' }, // applied to every request
  fetch: customFetch,                    // optional; defaults to global fetch
});