@render-lab/tasks-firecrawl

⚠️ Experimental: proof of concept. This package is part of the Render Tasks POC and is published for testing only. It is not fully tested or production ready. Task names, inputs, outputs, and behavior can change or break in any release. Pin exact versions and expect breaking changes.

Durable Firecrawl tasks for Render Workflows.

This pack talks to the Firecrawl v2 HTTP API directly rather than through the official SDK. The SDK retries, polls, and paginates internally, which would hide each attempt from Render Workflows. Direct HTTP keeps every poll, page fetch, and retry as one observable durable run, with the baked-in retry policy as the single source of truth (vendor retries are off).

import {
  scrape,
  map,
  search,
  startCrawl,
  awaitCrawl,
  cancelCrawl,
  startBatchScrape,
  awaitBatchScrape,
  cancelBatchScrape,
  getJobResults,
  startExtract,
  awaitExtract,
} from "@render-lab/tasks-firecrawl";

Tasks

| Task | Input | Output | Retry | | --- | --- | --- | --- | | firecrawl.scrape | ScrapeInput | ScrapeResult | general (3× backoff) | | firecrawl.map | MapInput | MapResult | general (3× backoff) | | firecrawl.search | SearchInput | SearchResult | general (3× backoff) | | firecrawl.startCrawl | StartCrawlInput | FirecrawlJobRef | trigger (0 retries) | | firecrawl.awaitCrawl | AwaitJobInput | FirecrawlJobState | await (poll every 10s up to 1h) | | firecrawl.cancelCrawl | CancelJobInput | CancelJobResult | cancel (3× backoff) | | firecrawl.startBatchScrape | StartBatchScrapeInput | FirecrawlJobRef | trigger (0 retries) | | firecrawl.awaitBatchScrape | AwaitJobInput | FirecrawlJobState | await (poll every 10s up to 1h) | | firecrawl.cancelBatchScrape | CancelJobInput | CancelJobResult | cancel (3× backoff) | | firecrawl.getJobResults | GetJobResultsInput | JobResultsPage | general (3× backoff) | | firecrawl.startExtract | StartExtractInput | ExtractJobRef | trigger (0 retries) | | firecrawl.awaitExtract | AwaitExtractInput | ExtractResult | await (poll every 10s up to 1h) |

The start* tasks get zero retries: a retried start would spawn a second remote job (Firecrawl has no find-or-create). You resume instead via the matching await*/getJobResults task using the returned jobId. The await* tasks are durable waits — SDK 1.0 has no native sleep, so they poll by throwing: still running → throw a progress error the fixed-interval retry re-runs; terminal failure → throw a terminal error so a failure is never retried as ordinary progress; completed → return. cancel* tasks are idempotent — a 404 (already finished/expired) normalizes to { status: "cancelled" }.

Async job example

start returns a jobId; await blocks on it; getJobResults reads bounded pages, storing the opaque nextCursor between calls:

const ref = await startCrawl({ url: "https://example.com/docs", limit: 100 });
await awaitCrawl({ jobId: ref.jobId });

let cursor: string | undefined = undefined;
do {
  const page = await getJobResults({
    jobId: ref.jobId,
    kind: "crawl",
    cursor,
    maxResults: 50,
    maxContentChars: 200_000,
  });
  handle(page.documents);
  cursor = page.nextCursor ?? undefined;
} while (cursor);

Extraction example

const ref = await startExtract({
  urls: ["https://example.com/pricing"],
  prompt: "Extract the pricing tiers as { name, priceUSD }[].",
  schema: { type: "object" },
});
const result = await awaitExtract({ jobId: ref.jobId, maxOutputBytes: 1_000_000 });
console.log(result.data); // JSON-serializable extraction

Webhook adapter

Use @render-lab/tasks-firecrawl/webhooks from a trigger web service to verify Firecrawl webhooks and map verified crawl, batch scrape, and extraction events to Workflow task runs:

import { firecrawlAdapter } from "@render-lab/tasks-firecrawl/webhooks";
import { serveDispatchServer } from "@render-lab/triggers";

serveDispatchServer({
  webhooks: {
    firecrawl: firecrawlAdapter({
      onEvent: ({ family, event, payload }) => {
        if (family === "crawl" && event === "completed") {
          return { task: "research.ingestCrawl", args: [{ jobId: payload.id }] };
        }
        return null;
      },
    }),
  },
});

Firecrawl signs with X-Firecrawl-Signature: sha256=<hex>, an HMAC-SHA256 over the exact raw body, compared in constant time. family/event are derived from the payload type (e.g. crawl.completed, batch_scrape.page, extract.failed); an unrecognized family or event maps to null. The /webhooks subpath does not import the package root, register firecrawl.* tasks, initialize a Firecrawl client, or read credentials at import time. Keep event routing in the trigger service because task names are workflow-specific.

Payload limits

Every document- or extraction-bearing task requires an explicit caller limit so payload budgets are visible in the DTO, not hidden defaults:

• maxContentChars bounds the combined content characters of a returned document (recommended 200_000). Used by scrape, search, and getJobResults.
• maxResults bounds the number of documents in a getJobResults page (recommended 50).
• maxOutputBytes bounds the extraction result's serialized UTF-8 bytes (recommended 1_000_000). Used by awaitExtract.

An oversized result throws before it returns, with corrective guidance (Reduce maxResults or maxContentChars / Reduce the extraction schema or maxOutputBytes). These limits keep every task argument and result under the Render Workflows 4 MB platform cap.

Install

pnpm add @render-lab/tasks-firecrawl @renderinc/sdk

@renderinc/sdk is a peer dependency. If you use the webhook adapter in a trigger web service, also install @render-lab/triggers.

Environment contract

| Variable | Required | Purpose | | --- | --- | --- | | FIRECRAWL_API_KEY | for all firecrawl.* tasks | Firecrawl API key, sent as Authorization: Bearer <key>. Read lazily at the first API call, never at import. | | FIRECRAWL_WEBHOOK_SECRET | for the default webhook verifier | HMAC secret used by @render-lab/tasks-firecrawl/webhooks to verify X-Firecrawl-Signature before dispatch. Read only inside verify(). Belongs on the trigger web service, not the Workflow service. |

Both credentials are read lazily at first use. Constructing the port or the webhook adapter never reads them, so importing this pack for one task never requires the other's secret.

Testing

pnpm -C packages/tasks-firecrawl build
pnpm -C packages/tasks-firecrawl test        # hermetic Tier 1 (no secrets, no network)
pnpm -C packages/tasks-firecrawl typecheck
RUN_LIVE=1 FIRECRAWL_API_KEY=fc_... pnpm -C packages/tasks-firecrawl test:live  # opt-in Tier 2

The raw *Impl functions take an injected FirecrawlDeps, so they unit-test against a fake FirecrawlPort with no network.