@render-lab/tasks-scrape

⚠️ Experimental: proof of concept. This package is part of the Render Tasks POC and is published for testing only. It is not fully tested or production ready. Task names, inputs, outputs, and behavior can change or break in any release. Pin exact versions and expect breaking changes.

Durable web-scrape tasks for Render Workflows.

import { fetchUrl, extractContent, toMarkdown, crawl } from "@render-lab/tasks-scrape";

| Task | Input | Output | | ----------------------- | ---------------------------------------------- | -------------------------------------------------------- | | scrape.fetch | { url, headers? } | { url, status, ok, contentType, body } | | scrape.extract | { html, baseUrl? } | { title, text, links } | | scrape.toMarkdown | { url, headers? } | { url, title, markdown } | | scrape.extractLinks | { url, headers? } | { url, links } | | scrape.extractMetadata| { url, headers? } | { url, title, description, ogTitle?, ogImage?, canonical? } | | scrape.sitemap | { url, headers? } | { url, urls } | | scrape.crawl | { startUrl, maxPages?, sameHost?, headers? } | { pages: [{ url, title }] } |

The two primitives compose: fetch a page, then extract readable text from it. Keeping them separate lets a workflow fan out — fetch many URLs as parallel durable runs — and pass each body to extract on its own instance, which is exactly the dashboard lineage a plain helper can't give you. The higher-level tasks bundle a fetch with a pure transform so common one-shot needs are a single call.

• scrape.fetch follows redirects and returns the raw body. It throws on 429/5xx so the durable retry policy re-runs it, and returns 4xx (e.g. 404) to the caller so a genuine client error doesn't burn retries.
• scrape.extract is pure CPU work (no retry): it strips script/style/tags, decodes entities, collapses whitespace, and pulls out the <title> and de-duplicated absolute links.
• scrape.toMarkdown fetches and converts the page's main content to clean markdown (headings, lists, emphasis, [text](href) links) — RAG/agent input, not a perfect converter.
• scrape.extractLinks fetches and returns the de-duplicated absolute links on the page.
• scrape.extractMetadata fetches and parses <title>, <meta name="description">, OpenGraph og:title/og:image, and <link rel="canonical">.
• scrape.sitemap fetches a sitemap.xml and returns the URLs from its <loc> entries (handles both <urlset> and <sitemapindex>).
• scrape.crawl does a bounded breadth-first walk from startUrl, following links (same host only when sameHost, default true) up to maxPages (default 10), deduping visited URLs — all in one durable run.

Like scrape.fetch, the fetch-backed tasks throw on 429/5xx so the durable retry re-runs, and pass 4xx bodies through. All parsing is done by pure, exported helpers (htmlToMarkdown, extractMetadata, parseSitemap, extractLinks, htmlToText, extractTitle).

Install

pnpm add @render-lab/tasks-scrape @renderinc/sdk

@renderinc/sdk is a peer dependency. There is no vendor dependency — the default port uses global fetch (Node 18+).

Environment contract

None. scrape.fetch takes everything it needs as input. Pass a User-Agent via headers if a site requires one.

A note on extraction quality

scrape.extract is a minimal, dependency-free extractor — regex tag-stripping, not a DOM parser — which is enough to hand readable text to an LLM. It does not run JavaScript. For JS-rendered pages or precise DOM queries, swap in a Firecrawl/Browserbase-backed ScrapePort; the port interface is the seam for exactly that, and it's why this is a capability package rather than tasks-firecrawl.

Extending & testing

Both operations export the wrapped task and its raw *Impl. scrape.fetch depends on a small ScrapePort, so it unit-tests without the network; the extraction transforms (extract, htmlToText, extractLinks, extractTitle) are pure and exported:

import { fetchUrlImpl, type ScrapePort } from "@render-lab/tasks-scrape";

const scrape: ScrapePort = {
  fetch: async () => ({ url, status: 200, ok: true, contentType: "text/html", body: "<p>hi</p>" }),
};
await fetchUrlImpl({ url: "https://example.com" }, { scrape });

Run the tests with pnpm -C packages/tasks-scrape test.