scrapeland (Node.js)

Zero-dependency rotating proxy client for Node 18+. Authenticate with your API key; every request exits a fresh, working IP.

npm install scrapeland

import { ScrapelandClient } from "scrapeland";

const client = new ScrapelandClient("pb_live_YOURKEY");      // or $SCRAPELAND_API_KEY
const res = await client.get("https://api.ipify.org");
console.log(res.status, res.text());                        // a fresh exit IP

// country targeting + sticky session
const r2 = await client.get("https://example.com", { country: "de", session: "job42" });

Use it with your own agent / fetch

proxyUrl() returns a proxy URL for https-proxy-agent, undici's ProxyAgent, Playwright, Puppeteer, etc.:

import { HttpsProxyAgent } from "https-proxy-agent";
const agent = new HttpsProxyAgent(client.proxyUrl({ country: "us" }));
await fetch("https://api.ipify.org", { agent });

Fetch and extract (data API)

Beyond the raw proxy, the client can call the Scrapeland data extraction API (a different host from the proxy gateway). fetch() returns a rendered page; extract() pulls structured fields with CSS selectors.

import { ScrapelandClient } from "scrapeland";

const client = new ScrapelandClient("pb_live_YOURKEY");      // or $SCRAPELAND_API_KEY

// fetch the page HTML (or pass { format: "text" } for plain text)
const page = await client.fetch("https://example.com", { country: "de" });
console.log(page.status, page.html.slice(0, 200));

// extract structured data; each field is a CSS selector or { css, attr, all }
const { data } = await client.extract("https://example.com", {
  title: "h1",
  links: { css: "a", attr: "href", all: true },
});
console.log(data);                                          // { title: ..., links: [...] }

Browser rendering and response headers

fetch() and extract() accept extra options: render: true renders the page in a real browser, waitFor: "<css selector>" waits for an element before capturing (render mode), and headers: true includes the response headers.

// render the page, wait for a selector, and ask for the response headers
const page = await client.fetch("https://example.com", {
  render: true,
  waitFor: "#app",
  headers: true,
});
console.log(page.status, page.headers);                     // headers included

// the same options apply to extract()
const { data, headers } = await client.extract(
  "https://example.com",
  { title: "h1" },
  { render: true, waitFor: ".loaded", headers: true },
);

device: "mobile", actions, fingerprint and blockResources are live too. Rendering bills 5 request-units instead of the 1 a plain fetch costs (10 with a screenshot, 1 with blockResources), and it downloads everything the page asks for — measured at 1–300× the bytes (median ~17×, image-heavy shops worst), so it is noticeably slower. Unless you need a screenshot or the images themselves, pass blockResources: true: images, fonts and media are skipped, the DOM your selectors read is identical. If every browser slot is busy you get a 503 with Retry-After — retry, it is not a failure and is not billed.

AI extraction, lists, batch, and discovery

Describe fields in plain English instead of selectors, fetch many URLs at once, pull page metadata/links, or ask the deployment what it supports:

// AI extraction — no selectors; add schema to pin the shape, model to route
const { data } = await client.extract("https://books.toscrape.com/", null, {
  prompt: "the first book title and price as a number",
  schema: { title: "string", price: "number" },
  model: "fast",
});
// "list every item" prompts come back under data.items: { items: [ {...}, ... ] }

// Asking a QUESTION rather than listing fields? structured:false returns prose in
// `answer` (and no `data` key), so you don't have to guess the keys a model invented.
const res = await client.extract("https://scrape.land/", null, {
  prompt: "what's this business about?",
  structured: false,
});
console.log(res.answer); // "scrape.land is a web data-extraction API that ..."
// ExtractResult is a union: check `"answer" in res` to narrow. `schema`/`extractType`
// pin a JSON shape, so neither can be combined with structured:false (400).

// many URLs in one call (up to 20); each billed like a single fetch
const { results } = await client.batch(
  ["https://a.example/", "https://b.example/"],
  { format: "text" },
);

// page metadata (title/OpenGraph/JSON-LD) and every link as an absolute URL
const page = await client.fetch("https://example.com", { metadata: true, links: true });
console.log(page.metadata, page.links);

// what can THIS key do? (ai.enabled reflects your plan — AI extraction needs Scale+)
const caps = await client.capabilities();
console.log(caps.ai.enabled, caps.ai.models, caps.render.enabled);

The API base URL defaults to https://scrape.land; override it with the apiBase option or $SCRAPELAND_API_BASE.

Rank links, stream large batches, and handle errors

// rank() — score a page's links by relevance to a goal; pick what to fetch next
// instead of crawling everything.
const r = await client.rank("https://news.example/", "articles about interest rates", { topK: 10 });
for (const link of r.links) console.log(link.score, link.url, "-", link.reason);
// AI transparency: on an AI failure rank() does NOT throw — links come back in
// document order with a ranking_error you can check, so your code never breaks.
if (r.ranking_error) console.warn("ranking degraded:", r.ranking_error);

// batchIter() — any number of URLs, auto-chunked into <=20-URL calls, yielded in
// order as chunks complete (no 20-URL cap to manage).
for await (const res of client.batchIter(allUrls, { fields: { title: "h1" } })) {
  if (res.error) console.warn("failed", res.url, res.error);
  else save(res.url, res.data);
}

// Typed errors — branch on the failure kind instead of parsing strings.
import { RateLimitError, PaymentRequiredError, ScrapelandError } from "scrapeland";
try {
  await client.fetch("https://example.com");
} catch (e) {
  if (e instanceof RateLimitError) await sleep((e.retryAfter ?? 5) * 1000);
  else if (e instanceof PaymentRequiredError) alert("out of quota / credit");
  else if (e instanceof ScrapelandError) console.error(e.status, e.detail, e.path);
  else throw e;
}

API

| Member | Description | |---|---| | new ScrapelandClient(apiKey, { gateway, retries, backoff }) | gateway defaults to $SCRAPELAND_GATEWAY; retries defaults to 2 (transient failures retry with backoff, each rotating IP) | | client.get(url, opts) | opts: country, session, protocol, maxLatencyMs, headers, timeout | | client.request(url, opts) | same, with method / body | | client.proxyUrl(opts) | proxy URL string for any agent-based client | | client.fetch(url, opts) | data API: fetch a page (format html/text/markdown, render, waitFor, screenshot, fullPage, actions, sendHeaders, cookies, method, body, metadata, links, headers, country, session, protocol, maxLatency) | | client.extract(url, fields, opts) | data API: extract by CSS/XPath fields (every plan), or AI via opts.prompt (+ schema, model, extractType, structured) which needs Scale or above; a list answer arrives under data.items, and structured: false answers in prose under answer | | client.batch(urls, opts) | data API: fetch up to 20 URLs in one call; { results: [...] }, each billed independently | | client.capabilities() | what your key can do (render/AI on?, models, formats, limits); ai.enabled is plan-scoped — AI extraction needs Scale or above, ai.min_plan names it | | client.account() | your key's plan, remaining quota, prepaid credit, rate limit, key budget | | proxyUsername(apiKey, opts) | build the API-key-with-params username |

Responses expose status, headers, body (Buffer), text(), json().

The built-in client is validated for HTTPS targets (the common case). For HTTP targets or browser automation, use proxyUrl() with your preferred agent.

Blocked port? Use :443

The gateway answers on gateway.scrape.land:443 as well as :8080, for corporate, university and hotel networks that filter high ports. Identical behaviour and billing — only the port differs, and the scheme stays http:// with your key in the username:

const client = new ScrapelandClient("pb_live_YOURKEY", {
  gateway: "http://gateway.scrape.land:443",       // or $SCRAPELAND_GATEWAY
});

Plan limits

client.account() returns your own numbers; these are the shapes to code against.

| | Free | Starter | Growth | Scale | Business+ | Pay as you go | |---|---|---|---|---|---|---| | Rate limit | 5/s | 50/s | 100/s | 200/s | 1,000/s | none | | Max response | 2 MB | 2 MB | 5 MB | 10 MB | 25 MB | 5 MB | | AI extraction | — | — | — | yes | yes | — |

Over the rate limit you get a 429 with Retry-After. A response larger than your cap is refused whole with a 413 naming the size, the limit and your plan — never truncated, and never billed. opts.prompt / schema / extractType on a plan below Scale return 403; check (await client.capabilities()).ai.enabled first.

Past your included requests, overage draws on prepaid credit and floors at zero — at zero you get a 402, never an invoice after the fact.

Full docs (curl/Python/Scrapy/Playwright): https://scrape.land/docs.