@scrapa/sdk

The official TypeScript client for scrapa.app: read any web page as markdown, HTML, text, links, a screenshot or a PDF, through a plain HTTP request or a real browser, whichever the page turns out to need. Search the web through the same key, and hold a real Chromium your own Playwright or Puppeteer drives.

npm i @scrapa/sdk

import { Scrapa } from "@scrapa/sdk";

const scrapa = new Scrapa(process.env.SCRAPA_API_KEY!);

const page = await scrapa.scrape({ url: "https://example.com" });
console.log(page.markdown);

Every method, every argument and every field is generated from the API's own definitions, so what your editor autocompletes is what the server accepts. Node 20+, Bun, Deno, workers and the browser. It runs anywhere fetch exists.

Reading a page

scrape() takes a URL and gives you the page. engine defaults to auto: a plain HTTP request first, re-read in a real Chromium only when the body turns out to be a JavaScript shell. That is the difference between 1 credit and 5.

const page = await scrapa.scrape({
    url: "https://news.ycombinator.com",
    formats: ["markdown", "links"],
});

page.engine; // "fetch": this one did not need a browser
page.links; // every absolute link, in document order
page.metadata.title;

markdown and text hold the page's main content, without the menus, the share buttons or the related posts. Send onlyMainContent: false for the whole page, or pick the parts yourself with CSS selectors:

const post = await scrapa.scrape({ url: "https://example.com/blog/hello", includeTags: ["article"] });

A link to a PDF or a Word file comes back as text too, and metadata.pages says how many pages the PDF has.

Ask for a screenshot or a PDF and the browser is used whatever engine says. Both come back base64-encoded, and asking for both costs no more than asking for one.

const shot = await scrapa.scrape({ url: "https://stripe.com", formats: ["screenshot"] });
await writeFile("stripe.png", Buffer.from(shot.screenshot!, "base64"));

Reading the same page again within the hour? Send maxAge, the oldest copy you accept in milliseconds (up to 3600000). If we still hold a copy your account read of that URL, from the same country with the same engine, in that time, you get it back for 0 credits (extract is still charged), with cached: true and cachedAt, the time it was read. If not, the page is read now at the usual price. We may drop a copy sooner, to make room or on a deploy, and a batch keeps its own copies, apart from single calls. A copy is never shared with another account.

const page = await scrapa.scrape({ url: "https://example.com/pricing", maxAge: 600_000 }); // 10 minutes
page.cached; // true when it came from a copy, so the page cost nothing

Where calls come from

Every call leaves through a residential address in the United States unless it says otherwise. Name a country, and inside it a state or a city, on any call:

await scrapa.scrape({
    url: "https://www.mercadolivre.com.br",
    proxy: { country: "br", state: "paraná", city: "curitiba" },
});

Or say it once for the whole client. A call's own proxy still wins, field by field:

const scrapa = new Scrapa({
    apiKey: process.env.SCRAPA_API_KEY!,
    defaults: { proxy: { country: "br" } },
});

Write places out in full: são paulo, not sp. A city is not exact: most calls leave from it, a few from elsewhere in its state. A small one can have nobody free right now. That answers PROXY_REFUSED at no charge, and asking for the state or the country alone gets through. search() reads a whole country's results, so it takes the default country and leaves a default city out.

Searching the web

search() runs a real search and hands back results we read ourselves. That is not a detail: every search vendor forbids redistributing their results through another API, so our own fetch is the only kind you are allowed to build on.

const found = await scrapa.search({
    query: "melhores frameworks typescript 2026",
    limit: 10,
    proxy: { country: "br" }, // picks the exit AND the market: Brazilian results, in Portuguese
});

found.results[0].url; // array order IS the ranking: there is no score to sort by, because none is real
found.cached; // the same question inside the window costs nothing and says so

topic: "images" hands back pictures: each hit's url is the page that shows one, and hit.image holds the picture's own address, a thumbnail that always loads and its size. Credit the page, not the file: no hit says who owns a picture. topic: "news" is the one topic that reliably carries publishedAt. includeDomains is asked of the front and enforced on the answer, so what comes back is on your list whatever the front did with the operator. answer: true adds a paragraph written over the hits. To read any result, hand its URL straight to scrape().

A search every front refused costs nothing. A search that ran and found nothing is charged. The index answered, and "nothing published that" is an answer.

Listing a site's pages

map() hands back every URL a site lists, without reading the pages. The list comes from the site's robots.txt, its sitemaps (gzipped ones and indexes too) and the links on the page you name.

const { urls } = await scrapa.map({ url: "https://example.com" });

urls[0]; // { url: "https://example.com", lastmod: "2026-10-01" }

Each URL comes back once, without utm_ parameters, a # part or a trailing slash. Only the host you named is kept unless you pass includeSubdomains: true, and search: "blog" keeps the URLs that contain the word. truncated: true means the site may list more than you got.

The price does not change with what a map finds. Only managed-pool bandwidth past the first megabyte is added. Price one first with quoteMap().

Driving a real browser

A session is one Chromium held for you, with a CDP URL your own tools connect to. We hand back a URL rather than a client, so everything you already know how to do with Playwright or Puppeteer keeps working.

import { chromium } from "playwright";

const session = await scrapa.sessions.create({ timeoutMin: 10 });
const browser = await chromium.connectOverCDP(session.connectUrl);

try {
    const page = await browser.newPage();
    await page.goto("https://example.com");
    await page.getByRole("button", { name: "Accept" }).click();
} finally {
    await browser.close();
    await scrapa.sessions.close({ id: session.id });
}

create() returns only once Chromium is actually up, so connectUrl is a required field and needs no !. It carries the session's token, is shown once and cannot be rebuilt from the id. That is why get() does not repeat it.

Minutes are reserved for the whole window and settled by what you used, so closing early hands the rest back. A browser that never came up costs nothing at all, and if you forget to close one, it is reaped at expiresAt and settled the same way.

Reading a list of pages

A batch is N scrapes on our schedule instead of your connection. Same options, same engines, same meter. What you buy is not holding a socket open for an hour.

const batch = await scrapa.batches.create({
    urls: sitemap.map((entry) => entry.loc),
    formats: ["markdown"],
});

let status = await scrapa.batches.get({ id: batch.id });
while (status.completed < status.total) {
    await new Promise((resolve) => setTimeout(resolve, 5_000));
    status = await scrapa.batches.get({ id: batch.id });
}

for (let from = 0; from < status.total; ) {
    const page = await scrapa.batches.get({ id: batch.id, items: true, from });
    for (const item of page.items ?? []) {
        if (item.status === "done") save(item.url, item.result!.markdown);
    }
    from += page.items?.length ?? status.total;
}

Nothing is charged when you submit and nothing is reserved: each URL then costs what the same call would have cost alone, and cost on the batch is what it has spent so far. Because nothing is held up front, a batch can outlive the credits that started it. Those URLs come back skipped rather than failed, which is not a verdict about the URL, and you can submit exactly those positions again.

completed counts every URL that reached an end, so it is what you compare against total; failed and skipped both live inside it. Results are readable for 24 hours, up to two hundred items per read.

Rather not poll? Pass a webhook and we will POST when it is done. The signing secret comes back once, on the create. Verify a delivery with HMAC-SHA256 over <timestamp>.<raw body> against the v1= value in the scrapa-signature header, and refuse a t= more than five minutes old, or a delivery somebody copied can be sent to you again.

What it costs, per call

Every metered answer carries what it cost and what is left. Methods hand you the page itself, so the meter arrives through a callback, per call, rather than a mutable field two concurrent scrapes would fight over.

await scrapa.scrape(
    { url: "https://example.com" },
    { onUsage: ({ cost, creditsRemaining }) => console.log(cost, creditsRemaining) },
);

// …or price the call without making it:
const { cost, maxCost } = await scrapa.quote({ url: "https://example.com" });

maxCost is the ceiling: with auto it is the browser's price, charged only if the page turns out to need one.

When something says no

A refusal tells you who said no, and that decides what you do next.

import { ScrapaError } from "@scrapa/sdk";

try {
    await scrapa.scrape({ url });
} catch (error) {
    if (!(error instanceof ScrapaError)) throw error;
    switch (error.outcome) {
        case "blocked":
            // time ran out, nobody said no: give the page more time
            if (error.code === "NAV_TIMEOUT") return scrapa.scrape({ url, timeoutMs: 60_000 });
            // the site said no: try another pool, country or engine
            return scrapa.scrape({ url, proxy: { pool: "residential", country: "de" } });
        case "denied": // our policy said no: retrying cannot help, change the URL
            return null;
        case "broken": // the API said it broke, and refunded the call. Worth one retry
            return scrapa.scrape({ url });
    }
    throw error; // your own mistake (a bad field, a missing key, a spent quota), or error.mayHaveRun
}

error.cost is 0 on every one of those, because a page you did not get is a page you do not pay for. The one exception is a call whose own browser.actions script ran in the page first: a blocked or denied answer then keeps the browser price and its bandwidth, because the script may already have read the page. A broken call is always refunded. error.code carries the exact reason, error.retryAfter says when a limit clears, and error.requestId is what support needs to find the call.

One code has no outcome and is not your mistake: INVALID_RESPONSE. The call got a success status, but the body was not an answer from us, most likely because something between you and the API changed it. The call may have run and been charged, so the client does not send it again. error.cost says what it cost when the answer said so, and error.details holds the first 500 characters of what came back.

The client repeats a call only when a repeat can fix it. A 429, 408 or 425 means the call never ran, so any call is sent again, after as long as the answer asks when that is 10 seconds or less. A read, a quote or a cancel is also sent again after no answer or a 5xx. A scrape, a search, a new batch and a new session are not: one that timed out on your side may already have read the page, so you get the error and decide.

error.mayHaveRun is true when the call may have run, and been charged, but no answer from the API says so. That happens with every INVALID_RESPONSE. On any call but a read, it also happens with no answer at all, and with a 5xx page from something between you and the API, such as a gateway that gave up on a slow scrape. Only the API's own answer can say it broke and refunded the call, so such an error has no outcome, and retryable is false. Call again only if a second run is fine.

Bringing your own proxy

await scrapa.scrape({
    url: "https://example.com",
    proxy: { pool: "custom", server: "http://user:[email protected]:8080" },
});

Free on every plan, including the free one, and the bytes are never billed.

Reading the meter

const usage = await scrapa.usage.get();
usage.creditsUsed; // this billing month
usage.creditsIncluded;
usage.creditBalance; // purchased credits, which never expire
usage.resetAt;

Options

new Scrapa({
    apiKey: "sc_…",
    baseUrl: "https://api.scrapa.app/v1", // point at another deployment
    timeout: 65_000, // per attempt, above the API's own ceiling
    maxRetries: 2, // 0 disables
    fetch: myFetch, // a proxy agent, a test double
});

// One call: how long the client waits for it, and a signal that stops it and any retry.
await scrapa.scrape({ url }, { timeoutMs: 30_000, signal });

The second argument's timeoutMs is how long the client waits. A scrape's own timeoutMs, in the first argument, is how long we give the page.

Full reference: docs.scrapa.app.

Licence

MIT