Make your scrapers smarter with every run.

Your crawler teaches it once. It recognizes forever.

Quick start · How it works · Docs · Examples · Changelog

Every scraping run re-fetches thousands of pages you already scraped last time — paying for proxies, browsers, retries, and CAPTCHAs to learn things you already know.

FetchBrain gives your scrapers a shared memory pool. Before each request, a crawler asks "do I already know this request?" — known requests come back in ~50ms without touching the network; unknown ones run normally and are learned. Everything every scraper learns accumulates under your API key, run after run, scraper after scraper — a data asset that compounds instead of costs. Learning is always free.

$ node scraper.js                      # first run — the brain learns
  ● learned   example.com/p/1042        1.9s
  ● learned   example.com/p/1043        2.3s
  ● learned   example.com/p/1044        2.1s

$ node scraper.js                      # every run after that
  ● known     example.com/p/1042        0.05s   ⚡ no fetch
  ● known     example.com/p/1043        0.04s   ⚡ no fetch
  ● known     example.com/p/1044        0.05s   ⚡ no fetch

  🧠 Finished! recalled: 3/3 (100%), learned: 0, duration: 0.3s

One line to adopt

FetchBrain wraps your existing Crawlee crawler — no rewrites, no new framework, no pipeline changes:

+ import { FetchBrain } from "@fetchbrain.com/sdk";
  import { CheerioCrawler } from "crawlee";

- const crawler = new CheerioCrawler({
+ const crawler = FetchBrain.enhance(new CheerioCrawler({
    requestHandler: async ({ $, pushData }) => {
      await pushData({ title: $("h1").text(), price: $(".price").text() });
    },
- });
+ }), { apiKey: process.env.FETCHBRAIN_API_KEY });

  await crawler.run(urls);

That's it. Your handler only runs when there's something new to learn.

Why FetchBrain

| | Without | With FetchBrain | | --- | --- | --- | | Repeat requests | Full fetch, every run | Recalled in ~50ms, zero HTTP | | Proxy & compute spend | Pay per fetch, every run | Pay only for what it knows | | Blocks & CAPTCHAs | Every run risks them | Known requests never touch the site | | When the service is down | — | Your crawl runs normally — guaranteed | | Testing | Mock it yourself | MockFetchBrain ships in the box |

• ⚡ Instant recall — known requests skip the network entirely
• 🌐 Any request, not just pages — HTML, JSON APIs, POSTs with bodies; identity is derived from the full request
• 🎓 Auto-learning — every successful scrape teaches the brain, free
• 🛡️ Never breaks your crawl — circuit breaker degrades gracefully; a FetchBrain outage costs you optimization, never data
• 📦 Built for scale — request batching and deduping for high-concurrency crawlers
• 🔌 Crawlee-native — CheerioCrawler, PlaywrightCrawler, and friends
• 🔒 Private by design — your data is scoped to your key; only your requests, what you choose to learn, and platform identifiers (*_ID env vars) leave your process. Anonymized telemetry is opt-in (opt-in telemetry)
• 🧪 TypeScript-first — full types, ESM + CJS, zero config

Who is it for?

FetchBrain pays off wherever the same expensive request gets made more than once — across a fleet, across users, or across runs. It won't speed up a first-ever fetch of a brand-new URL, and it's not for freshness-critical monitoring where you always need the live value. Everywhere else, it compounds:

Data-service teams & aggregators — fetch an expensive request once, serve it to every client and job. Overlapping crawls draw from one shared pool, so per-run proxy and compute cost falls as coverage grows — and ask() turns everything you've collected into a queryable corpus, with no warehouse to run.

Product teams & AI agents — apps and browsing agents that re-hit popular requests get them back in milliseconds, straight from memory: no fetch, no proxy, no block. The more your users (or agents) converge on the same requests, the more the pool pays off.

Recurring crawls of stable data — catalogs, specs, listings, company and reference data barely change between runs, so recall skips the fetch and each scheduled run costs less. Keep volatile fields on memory: "fresh" so you never trade accuracy for speed.

Solo scraper developers — start free: learning never costs anything, and the requests you've already made come back instantly instead of getting blocked or throttled. One line to add, one brain that grows with everything you scrape.

Quick start

npm install @fetchbrain.com/sdk

import { FetchBrain } from "@fetchbrain.com/sdk";
import { CheerioCrawler } from "crawlee";

const crawler = FetchBrain.enhance(
  new CheerioCrawler({
    requestHandler: async ({ $, pushData }) => {
      // Runs only when the brain needs to learn (new request)
      await pushData({
        title: $("h1").text(),
        price: $(".price").text(),
      });
    },
  }),
  {
    apiKey: process.env.FETCHBRAIN_API_KEY,
    memory: "recent", // how far back the brain recalls (see below)
  },
);

await crawler.run(urls);

How it works

Your crawler ──▶ FetchBrain ──▶ knows this request? ──▶ YES ──▶ data in ~50ms, no fetch
                                      │
                                      └──▶ NO ──▶ your handler runs ──▶ brain learns, free

1. Recall — before each request, the SDK asks if the brain knows it
2. Known — data returns instantly and the fetch is skipped
3. Learning — unknown requests run through your handler as normal, and the result teaches the brain for every future run

Memory depth

How old is too old? You decide, per crawler:

| Depth | Window | Use for | | --- | --- | --- | | fresh | 1 hour | prices, stock, anything volatile | | recent | 24 hours | default — daily crawl cycles | | standard | 7 days | listings, catalogs | | deep | 30 days | slow-changing reference data |

Take control in your handler

context.brain puts the recall decision in your hands:

requestHandler: async ({ $, brain, pushData }) => {
  if (brain?.known) {
    await brain.use(); // push the brain's data, skip scraping
    return;
  }
  await pushData({ title: $("h1").text() }); // scrape + learn
},

| Property | Type | Description | | --- | --- | --- | | known | boolean | whether the brain knows this request | | data | object | the remembered data (if known) | | use() | function | push the brain's data and skip scraping |

Configuration Options

interface FetchBrainConfig {
  // Required
  apiKey: string;

  // Optional
  baseUrl?: string; // API URL (default: production)
  memory?: MemoryDepth; // How far back the brain recalls
  learning?: boolean; // Enable AI learning (default: true)
  alwaysRun?: boolean | string | string[]; // Which handlers to run (default: false)
  timeout?: number; // Request timeout in ms (default: 500)
  debug?: boolean; // Enable debug logging
}

Fine-grained routing control with alwaysRun — run every handler, none, or only specific labels when a request is known:

FetchBrain.enhance(crawler, { alwaysRun: ["listing", "category"] });

Ask your brain anything

This is where the pool pays off twice: ask() searches across everything all your scrapers have ever learned — one question, your whole corpus, no exact URLs, no warehouse:

import { FetchBrainClient } from "@fetchbrain.com/sdk";

const client = new FetchBrainClient({ apiKey: process.env.FETCHBRAIN_API_KEY });
const res = await client.ask("blue widgets under $50", { answer: true });

console.log(res.answer);   // synthesized answer
console.log(res.sources);  // scored sources

Brain Context in Handler

Access brain data directly in your handler via context.brain:

const crawler = FetchBrain.enhance(
  new CheerioCrawler({
    requestHandler: async ({ $, request, brain, pushData }) => {
      // Check if the brain already knows this page
      if (brain?.known) {
        console.log("Brain knows this page");

        // Option 1: Use brain data directly (skip scraping)
        await brain.use();
        return;

        // Option 2: Compare brain data with scraped data
        // const scraped = { title: $('h1').text() };
        // console.log('Brain:', brain.data, 'Scraped:', scraped);
      }

      // Scrape normally if the brain doesn't know
      const data = { title: $("h1").text() };
      await pushData(data);
    },
  }),
  { apiKey: "your-api-key", alwaysRun: true },
);

context.brain Properties

| Property | Type | Description | | --------- | -------- | ---------------------------------- | | known | boolean | Whether the brain knows this URL | | data | object | Brain data (if known) | | use() | function | Push brain data and skip scraping |

Using Dataset.pushData

⚠️ Important: AI learning only happens when you use context.pushData() or the SDK's pushData() wrapper below. Direct calls to Dataset.pushData() will not trigger learning, and the AI won't recognize these URLs in future runs.

If you use Dataset.pushData() instead of context.pushData(), use our wrapper for automatic AI learning:

import { FetchBrain, pushData } from "@fetchbrain.com/sdk";
import { Dataset } from "crawlee";

const crawler = FetchBrain.enhance(
  new CheerioCrawler({
    requestHandler: async ({ $, request }) => {
      const data = { title: $("h1").text() };

      // ✅ Use pushData wrapper for AI learning
      await pushData(data, Dataset);

      // ✅ Or with named dataset
      await pushData(data, Dataset, "products");

      // ❌ This will NOT learn:
      // await Dataset.pushData(data);
    },
  }),
  { apiKey: "your-api-key" },
);

No Crawlee? No problem

import { FetchBrain } from "@fetchbrain.com/sdk";

const brain = new FetchBrain({
  apiKey: "your-api-key",
  memory: "recent",
});

// Check if the brain knows a URL
const result = await brain.recall({ url: "https://example.com/product/123" });
if (result.known) {
  console.log("Brain knows:", result.data);
} else {
  // Fetch and teach
  const data = await scrapeYourWay("https://example.com/product/123");
  await brain.learn({ url: "https://example.com/product/123", data }); // free
}

It will never break your crawl

The circuit breaker is not an afterthought — it's the core design contract:

• Healthy → full optimization
• Slow (>500ms) → time out, continue without recall
• Down → circuit opens, your scraper runs standalone
• Recovered → circuit closes, optimization resumes

A FetchBrain problem can cost you optimization, never data. Every SDK call degrades gracefully; none of them throw into your crawl.

Telemetry (opt-in)

Telemetry is off by default. You can opt in to send anonymized operational diagnostics that help improve FetchBrain:

FetchBrain.enhance(crawler, {
  apiKey: process.env.FETCHBRAIN_API_KEY,
  telemetry: { enabled: true },
});

What's sent (all anonymized): domain (e.g. walmart.com), a SHA-256 hash of the full URL, a generalized path pattern, timing and status codes, retry counts, proxy country/type and success, coarse session aggregates (age, error rate, cookie count), block indicators, and crawler type. Never: raw URLs, request/response bodies, cookies or their values, proxy IPs, credentials, or any PII.

Share selectively with the sub-flags — e.g. telemetry: { enabled: true, shareProxyInfo: false }. Collection is buffered, best-effort, and never blocks or fails your crawl.

Testing

MockFetchBrain ships with the SDK — seed it, run your tests, no network:

import { MockFetchBrain } from "@fetchbrain.com/sdk/mock";

const mock = new MockFetchBrain({
  initialKnowledge: new Map([
    ["https://example.com/product", { title: "Known Product" }],
  ]),
});

// Use in tests
const result = await mock.recall({ url: "https://example.com/product" });
expect(result.known).toBe(true);

// Ask a natural-language question against seeded knowledge
const answer = await mock.ask("what is the price?");
expect(answer.sources.length).toBeGreaterThan(0);
console.log(answer.sources);

There's also a local mock server (npm run mock-server) for running real crawlers against http://localhost:3456 without an API key.

Examples

| Example | Shows | | --- | --- | | basic-cheerio | CheerioCrawler + FetchBrain in ~40 lines | | manual-recall | recall/learn without Crawlee | | with-mock | unit testing with MockFetchBrain |

• basic-cheerio - CheerioCrawler with FetchBrain
• manual-recall - Direct API usage without Crawlee
• with-mock - Unit testing with MockFetchBrain

FAQ

Does my scraped data train anyone else's brain? No. Everything your scrapers teach is scoped to your API key. Your data answers only your recalls.

Do my scrapers share memory? Yes — that's the point. Everything learned under your API key accumulates in one brain, and ask() searches across all of it. Recall is namespaced per scraper identity, so one scraper's responses never pollute another's results.

What leaves my process? By default: your requests, the data you choose to learn, and platform identifiers (env vars ending in _ID, plus NODE_ENV/REGION/CI). Never your general environment, credentials, or anything credential-shaped. If you opt into telemetry, anonymized access signal too — see what.

What does learning cost? Nothing. Learning is always free — you pay only for known queries.

What if the data changed since the brain learned it? That's what memory depth is for: pick fresh (1h) for volatile data, or set refreshOnRebuild to re-learn whenever you ship a new scraper build.

Contributing

Issues and PRs welcome — see open issues. If FetchBrain saves your crawler time or money, a ⭐ helps other scraper developers find it.

License

MIT © FetchBrain

Need help? Open an issue · Read the docs