@thejob/data-processor-client

Typed TypeScript/JavaScript client for the thejob-data-processor HTTP API. External systems use this instead of hand-building fetch requests.

The contract is small. A consumer does three things:

1. Push data in with ingest(records, opts), optionally attaching a webhookConfig so each record's processed result is delivered to your URL.
2. Pull as a backstop with pollProcessed() to catch up on anything a webhook missed.
3. Ask for one answer with generateText() / generateJson() / embed(), synchronous, for work whose result is needed before the call returns.

(1)+(2) are the bulk path: batched through the pipeline, cheap per record, results delivered later (up to hours). (3) is the direct path: costs more per call but answers immediately. Pick the bulk path for backfills and the direct path for a maintenance job that writes each chunk before starting the next, a script, or an interactive request.

Either way the provider SDK, credential, model choice and retry policy stay in the service, so a consumer needs no API key and no provider dependency — it brings only its own prompt, schema and domain logic.

You never name a pipeline or manage runs, you ingest records under a shared type and receive the processed result. The shared type/source/webhookConfig go once in opts (sent once on the wire, not repeated per record). The client owns the base URL, JSON plumbing, the request envelope, cursor paging, and typed errors.

Zero runtime dependencies. Uses the global fetch (Node 18+, Deno, browsers, edge).

Install

npm install @thejob/data-processor-client

Usage

import { DataProcessorClient, DataProcessorError } from '@thejob/data-processor-client';

const dp = new DataProcessorClient({ baseUrl: 'http://localhost:3000' });

// 1. Push data in. Records share type/source/webhook via opts (sent once). Attach
//    a webhookConfig to have each record's processed result POSTed to your URL
//    (signed when you set a secret, so you can verify it). A repeated dedupKey is
//    reported as a duplicate, not re-ingested.
const report = await dp.ingest(
  [
    { dedupKey: 'job-123', payload: { url: '...' } },
    { dedupKey: 'job-124', payload: { url: '...' } },
  ],
  {
    type: 'job',
    source: 'scraper',
    webhookConfig: { url: 'https://my-app.example/hooks/data-processor', secret: 'whsec_...' },
  },
);
console.log(report.accepted, report.duplicates, report.failed);

// 2. Pull backstop: if your webhook was down (or you attached none), catch up.
//    The cursor is threaded across pages; pass a saved cursor to resume.
for await (const page of dp.pollProcessed({ type: 'job' })) {
  for (const record of page.records) {
    handle(record.payload);
  }
}

// 3. Ask for one answer, synchronously. No API key here: the model, credential
//    and retry policy live in the service. Use this when you need the result
//    before continuing (a chunked maintenance job, a script), not for backfills.
const text = await dp.generateText('Describe the skill "React" in one sentence.', {
  systemInstruction: 'Write for a job-seeker audience. Be terse.',
});

// Pass a responseSchema for structured output; the reply comes back parsed.
const parsed = await dp.generateJson<{ items: { id: string; description: string }[] }>(
  buildPrompt(chunk),
  {
    responseSchema: {
      type: 'object',
      properties: {
        items: {
          type: 'array',
          items: {
            type: 'object',
            properties: { id: { type: 'string' }, description: { type: 'string' } },
            required: ['id', 'description'],
          },
        },
      },
      required: ['items'],
    },
  },
);

// Embeddings, same synchronous shape and the same "no key here" property.
// Vectors come back in input order; 'query' vs 'document' matters (Voyage embeds
// asymmetrically) and mismatching them degrades similarity silently.
const { vectors } = await dp.embed(['react', 'kubernetes'], { type: 'query' });

// Errors are typed.
try {
  await dp.records.get('does-not-exist');
} catch (e) {
  if (e instanceof DataProcessorError && e.status === 404) { /* not found */ }
}

API surface

| Group | Methods | Purpose | | --------- | -------------------------------------------------------- | ------- | | ingest | ingest, records.list, records.get | push data in (shared type/source/webhook via opts), inspect what you sent | | processed | getProcessed, pollProcessed, getProcessedRecord | pull backstop for results | | generate | generateText, generateJson | one synchronous answer, no API key needed by the caller | | models | models | the offered model roster + the default, for a picker UI | | embed | embed | texts -> vectors; the service owns the key and rate budget | | health | health | liveness |

Any non-2xx response throws a DataProcessorError carrying .status and the parsed .body.

Configuration

new DataProcessorClient({
  baseUrl: 'https://data-processor.internal',
  headers: { 'x-some-header': 'value' }, // applied to every request
  fetch: customFetch,                    // optional; defaults to global fetch
});