Zenrows Node.js SDK

SDK to access Zenrows API directly from Node.js. Zenrows handles proxies rotation, headless browsers, and CAPTCHAs for you.

Table of Contents

• Installation
• Usage
  • Adaptive Stealth Mode
  • POST Requests
  • Extract
  • Batch
    • Failed runs and extensible values
    • Extract in a batch
  • Crawl
  • Concurrency
    • An important note about Promise.allSettled() on TypeScript
• Examples
• Contributing
  • Running the Crawl e2e test
• License

Installation

Install the SDK with your package manager of choice.

npm install zenrows
yarn add zenrows
pnpm install zenrows
bun install zenrows

Usage

Start using the API by creating your API Key.

The SDK uses the official fetch api for HTTP requests. The client's response will be a Response.

It also uses fetch-retry to automatically retry failed requests (status code 429 and 5XX). Retries are not active by default; you need to specify the number of retries, as shown below. It already includes an exponential back-off retry delay between failed requests.

client.fetch() is the primary method for the main page-scraping product. client.get() still works and is kept as a deprecated alias — new code should call fetch().

const { ZenRows } = require("zenrows");

const apiKey = "YOUR-API-KEY";
const url = "https://www.zenrows.com/";

(async () => {
  const client = new ZenRows(apiKey, { retries: 1 });

  const response = await client.fetch(
    url,
    {
      // Our algorithm allows to automatically extract content from any website
      autoparse: false,

      // CSS Selectors for data extraction (i.e. {"links":"a @href"} to get href attributes from links)
      css_extractor: "",

      // Enable Javascript with a headless browser (5 credits)
      js_render: false,

      // Use residential proxies (10 credits)
      premium_proxy: false,

      // Make your request from a given country. Requires premium_proxy
      proxy_country: "",

      // Wait for a given CSS Selector to load in the DOM. Requires js_render
      wait_for: ".content",

      // Wait a fixed amount of time in milliseconds. Requires js_render
      wait: 2500,

      // Block specific resources from loading, check docs for the full list. Requires js_render
      block_resources: "image,media,font",

      // Change the browser's window width and height. Requires js_render
      window_width: 1920,
      window_height: 1080,

      // Will automatically use either desktop or mobile user agents in the headers
      device: "desktop",

      // Will return the status code returned by the website
      original_status: false,
    },
    {
      headers: {
        Referrer: "https://www.google.com",
        "User-Agent": "MyCustomUserAgent",
      },
    }
  );

  // You can also use response.json() if you're expecting JSON data.
  const data = await response.text();

  console.log(data);

  /* <!doctype html> <html... */

  // With the CSS selector {"links":"a @href"}
  /*
        {
            links: [
                'https://www.zenrows.com',
                'https://www.zenrows.com/blog',
                ...
            ]
        }
    */
})();

You can also pass optional parameters and headers; the list above is a reference. For more info, check out the documentation page.

Sending headers to the target URL will overwrite our defaults. Be careful when doing it and contact us if there is any problem.

Adaptive Stealth Mode

Set mode: "auto" to let Zenrows pick the request configuration for you — it starts with the cheapest viable setup and escalates to js_render/premium_proxy only when the target needs it, billing only for the configuration that succeeds.

const response = await client.get(url, { mode: "auto" });

Compatible with proxy_country, js_instructions, and custom_headers.

POST Requests

The SDK also offers POST requests by calling the client.post method. It can receive a new parameter data that represents the data sent in, for example, a form.

const { ZenRows } = require("zenrows");

const apiKey = "YOUR-API-KEY";
const url = "https://httpbin.org/anything";

(async () => {
  const client = new ZenRows(apiKey, { retries: 1 });

  const response = await client.post(
    url,
    {
      // The same params as in GET requests
    },
    {
      data: new URLSearchParams({
        key1: "value1",
        key2: "value2",
      }).toString(),
    }
  );

  const data = await response.json();

  console.log(data);
  /*
        ...
        form: { key1: 'value1', key2: 'value2' },
        ...
    */
})();

Extract

Extract (beta) runs a page through Zenrows' AI-powered structured extraction instead of returning raw HTML. Use client.extract() — it's the same request as fetch(), with the extract param set for you (defaults to "auto"; pass "native" or "standard" for the other contracts).

const { ZenRows } = require("zenrows");

const apiKey = "YOUR-API-KEY";
const url = "https://www.zenrows.com/";

(async () => {
  const client = new ZenRows(apiKey);

  const response = await client.extract(url); // extract: "auto"
  // const response = await client.extract(url, { extract: "native" });

  const data = await response.json();
  console.log(data);
})();

Batch

The Batch API (beta) runs many URLs asynchronously as a job, and is reachable via client.batch. A job is either open — it keeps accepting tasks via addTasks() until you closeJob() it, useful when you're streaming URLs in over time — or a normal one-shot batch where every task is known upfront.

const { ZenRows } = require("zenrows");

const apiKey = "YOUR-API-KEY";

(async () => {
  const client = new ZenRows(apiKey);

  // One-shot batch: every task known upfront.
  const job = await client.batch.submitJob({
    tasks: [{ url: "https://example.com/1" }, { url: "https://example.com/2" }],
  });

  // Streaming batch: keep the job open, add tasks as they arrive, close when done.
  const streamingJob = await client.batch.submitJob({ status: "open" });
  await client.batch.addTasks(streamingJob.job_id, [{ url: "https://example.com/3" }]);
  // Or close it as part of the last addTasks() call instead of a separate request:
  // await client.batch.addTasks(streamingJob.job_id, [...], { lastBatch: true });
  await client.batch.closeJob(streamingJob.job_id);

  // Poll for progress and page through results once it's done.
  const status = await client.batch.getJob(job.job_id);
  const { results, next_cursor } = await client.batch.getResults(job.job_id);

  // Retry only the tasks that failed, inheriting the rest from the previous run.
  await client.batch.rerun(job.job_id, { status: "failed" });
})();

client.batch also exposes listJobs(), deleteJob(), stopRun(), rerun(), listRuns(), getRun(), deleteRun(), and getTaskContent() (returns the scraped page's raw content as a string, not JSON — the endpoint can return HTML or plain text depending on what the target page served). Scheduling, webhook config, HMAC key rotation, CSV task uploads, and results exports aren't wrapped yet — call the Batch API directly for those.

Failed runs and extensible values

A run the server auto-fails carries failure_reason and a human-readable failure_detail. Starting, rerunning or resuming a job whose API key hit one of its credit caps throws a ZenRowsBatchError with status 402 and code "api_key_cap_reached"; error.problem.detail names the cap and when it resets (UTC).

const run = await client.batch.getRun(jobId, runId);
if (run.data.failure_reason === "api_key_cap_reached") console.log(run.data.failure_detail);

Enum-valued fields on responses (run, job, task and export status, ingest_status, failure_reason) are typed Extensible<...>: the known values plus any string, because the server may add values. The exported named types (RunStatus, JobStatus, TaskStatus, ExportStatus, IngestStatus, RunFailureReason) stay closed, so they still work as request input. Give switches on these fields a default branch; an exhaustive switch that assigns the leftover to never no longer compiles.

Extract in a batch

Set extract in the batch params to run tasks through Extract — structured data instead of raw HTML. It works job-wide or per task, and per-task values win on collision.

const job = await client.batch.submitRegular(
  [
    // Inherits the job-level params below.
    { url: "https://example.com/products", external_id: "p1" },
    // Overrides them for this task only.
    { url: "https://example.com/raw", zenrows_params: {} },
  ],
  undefined,
  { zenrowsParams: { extract: "auto" } },
);

An Extract task's result carries two keys — html (the raw page) and parsed (the structured data). It costs the same as a regular task, so estimateCost prices it correctly. extract_fields is not supported in Batch yet.

The batch client (ZenRowsBatchClient) also works standalone, without a ZenRows instance — matching the Go and Python SDKs' batch clients:

const { ZenRowsBatchClient } = require("zenrows");

const batch = new ZenRowsBatchClient(apiKey, { baseURL: "https://async.api.zenrows.com/v1" }); // baseURL is optional

Crawl

Crawl (new) takes one start URL and returns the URLs behind it, and is reachable via client.crawl. It follows the links on each page up to depth hops; without depth it has no depth limit and follows links until maxPages or maxItems stops it (each defaults to 100), or no link is left to open. A crawl stays on the start URL's registrable domain; subdomains count. A crawl runs asynchronously: create() returns at once, wait() polls until it ends, and results() reads the URLs it kept. With outputFormat: "html" it also fetches each kept page, which you read with content() or all at once with download().

Crawl is still evolving: new features are coming, limits may be tuned, and the changelog announces each change.

const { ZenRows } = require("zenrows");

const apiKey = "YOUR-API-KEY";

(async () => {
  const client = new ZenRows(apiKey);

  const crawl = await client.crawl.create({
    url: "https://www.scrapingcourse.com/ecommerce/",
    depth: 1, // link hops from the start URL; omit for no depth limit
    maxItems: 10, // stop once this many URLs are kept
    maxPages: 20, // stop once this many pages are fetched; bounds the cost
    includePatterns: ["/product/"], // keep only URLs containing one of these
    outputFormat: "html", // omit for URLs only
  });

  const ended = await client.crawl.wait(crawl.crawl_id); // completed, stopped, failed, or running on timeout
  if (ended.status === "failed") throw new Error(ended.error?.detail);

  for await (const result of client.crawl.results(crawl.crawl_id)) {
    console.log(result.url);
    if (result.content_status === "fetched") {
      const html = await client.crawl.content(crawl.crawl_id, result);
    }
  }

  // Or every result, with its page, as parsed NDJSON lines.
  const download = await client.crawl.download(crawl.crawl_id);
  console.log(download.status); // from X-Crawl-Status; "running" means more lines may come later
  for await (const line of download.lines) {
    console.log(line.url, line.content);
  }
})();

client.crawl also exposes get() (status, coverage and one page of results), list() (one page of the account's crawls, newest first, with next_cursor until the last page), and stop() (idempotent; a crawl that already ended answers as it ended). create() takes an idempotencyKey option, so a retried create does not start a second crawl.

results() does not poll. On a running crawl it yields only what the crawl has kept so far, then returns. Call wait() first to read every result. wait() polls for up to timeout seconds (default 600). When the time runs out, it returns the crawl with status running and does not throw; the crawl keeps running.

Errors throw a ZenRowsCrawlError with status, code, detail and retryAfter. code is undefined when the error body has no code. Branch on status and code:

• 400 invalid_request / unknown_parameter / invalid_cursor.
• 403 REQS008: Crawl is not enabled for this account.
• 404 crawl_not_found / content_not_found.
• 409 idempotency_request_in_flight: a request with the same idempotencyKey is still running. Retry after it finishes.
• 422 invalid_parameter / invalid_start_url / domain_not_allowed: detail names the problem. 422 idempotency_key_reused: the key was used with a different body. Use a new key or no key; do not retry as is.
• 429 too_many_crawls: the account has reached its limit of active jobs (3 by default), shared with its Batch jobs. Nothing was created. create() never retries it; retry after error.retryAfter seconds.
• 503 crawl_busy: the stop could not be saved yet; the crawl is still running. Repeating the stop is safe, so stop() retries it, and any other 429, 502, 503 or 504, up to retries times, waiting error.retryAfter seconds (at most 30, or 1 when the header is missing), then throws the last error.

content() takes a content id, a content_url or a result, and throws a TypeError when it cannot read a content id from it. Requests retry network errors and 502/503/504 (and 429 on reads) when it is safe; a create() is retried only with an idempotencyKey, and a stop() as described above.

outputFormat takes "html"; without it a crawl returns URLs only. maxPages and maxItems default to 100 each. A crawl created without depth has no depth field; one created with it echoes it. Response enum fields (status, stop_reason, error.code, content_status) are Extensible<...>: the known values plus any string. A failed crawl carries error.code and error.detail. A download line's content is a string, or an object for a crawl that returns JSON.

The crawl client (ZenRowsCrawlClient) also works standalone. Both forms take the same options:

const { ZenRows, ZenRowsCrawlClient } = require("zenrows");

// All options are optional. timeout is in seconds and applies to each HTTP request, not to reading a download.
const options = { baseURL: "https://api.zenrows.com/v1", retries: 3, timeout: 30 };
const crawl = new ZenRowsCrawlClient(apiKey, options);
const client = new ZenRows(apiKey, { crawl: options }); // client.crawl uses them

Concurrency

To limit the concurrency, it uses fastq, which will simultaneously send a maximum of requests. The concurrency is determined by the plan you are in, so take a look at the pricing and set it accordingly. Take into account that each client instance will have its own limit, meaning that two different scripts will not share it, and 429 (Too Many Requests) errors might arise.

We use Promise.allSettled() in the example below, available from Node 12.9. It will wait for all the promises to finish, and the results are objects with a status marking them as fulfilled or rejected. The main difference with Promise.all() is that it won't fail if any requests fail. It might make your scraping more robust since the whole list of URLs will run, even if some of them fail.

const { ZenRows } = require("zenrows");

const apiKey = "YOUR-API-KEY";

(async () => {
  const client = new ZenRows(apiKey, { concurrency: 5, retries: 1 });

  const urls = [
    "https://www.zenrows.com/",
    // ...
  ];

  const promises = urls.map((url) => client.fetch(url));

  const results = await Promise.allSettled(promises);
  console.log(results);
  /*
    [
        {
            status: 'fulfilled',
            value: {
                status: 200,
                statusText: 'OK',
                data: `<!doctype html> <html lang="en"> <head> ...
            
        ...
    */

  // separate results list into rejected and fulfilled for later processing
  const rejected = results.filter(({ status }) => status === "rejected");
  const fulfilled = results.filter(({ status }) => status === "fulfilled");
})();

An important note about Promise.allSettled() on TypeScript

Promise.allSettled() does not narrow the type of the array elements in the callback function. This means that you will need to cast the type of the array elements to PromiseSettledResult<Response> to access the status and value properties.

const promises = urls.map((url) => client.fetch(url));

const results = await Promise.allSettled(promises);

const fulfilled = results
  .filter(
    (item): item is PromiseFulfilledResult<Response> =>
      item.status === "fulfilled"
  )
  .map((item) => item.value.json());

Examples

Take a look at the examples directory for Javascript and TypeScript files using the SDK. It has its own package.json file and includes zenrows SDK ready to use. Each file makes two requests, the first with CSS selectors and the second with CSS selectors and premium proxies in the US.

cd examples
npm install
node index.js # JS example
npx tsx index.ts # TS example

Contributing

Pull requests are welcome. For significant changes, please open an issue first to discuss what you would like to change.

Before opening a pull request, run pnpm check and pnpm test -- --run. Neither touches the network.

Running the Crawl e2e test

tests/e2e/crawl.e2e.test.ts runs a small real crawl (depth 1, up to 3 items and 5 pages, HTML output) and checks every Crawl method against the live API. It is skipped unless ZENROWS_API_KEY and ZENROWS_E2E_CRAWL_URL are set:

• ZENROWS_API_KEY: a key with Crawl access. Keep it out of your shell history, e.g. read it from a file.
• ZENROWS_CRAWL_BASE_URL (optional): the API base. Default https://api.zenrows.com/v1. Point it at a local or staging deployment to test that instead.
• ZENROWS_E2E_CRAWL_URL: the start URL to crawl, a site you may crawl whose start page links to at least one page.
• ZENROWS_E2E_CRAWL_INCLUDE (optional): an include pattern, e.g. /product/. When set, the crawl keeps only matching URLs and the test asserts every result contains it.

export ZENROWS_API_KEY="$(cat path/to/key-file)"
export ZENROWS_E2E_CRAWL_URL=https://www.scrapingcourse.com/ecommerce/
export ZENROWS_E2E_CRAWL_INCLUDE=/product/
pnpm test:e2e

The run creates one crawl on the account and uses up to a few pages of credits. If the account has reached its limit of active jobs, the test waits and retries (up to 5 minutes).

License

MIT