Contents

• What this is
• Install
• Quick start
• Three ways to read a page
• Step by step
• Scheduling
• Data sinks
• Network capture
• Anti-block
• Browser automation
• Zero-schema extraction
• Command line
• Core API
• Schema syntax
• Modules
• Configuration
• Validation rules
• Plugins
• Property naming
• Errors
• Troubleshooting
• Testing
• API reference
• Version history
• Architecture
• Behavior notes
• License

What this is

sengkrep sits between your scraper and the network. Requests leave through Node's built-in http, https and http2 modules, HTML is parsed with cheerio, and the library adds the parts a scraper needs once it runs for longer than a few minutes.

• retries with backoff that honor Retry-After
• a circuit breaker per host
• response caching, conditional requests and single-flight request sharing
• selector health monitoring, so a layout change shows up before your data goes empty
• diff detection between runs against the same URL
• rate limiting and adaptive throttling per host
• proxy rotation, cookie jar, session pool, CSRF handling, token refresh
• bot-wall detection with per-session browser identities and proxy session pinning
• browser automation over the DevTools Protocol: navigate, click, type, evaluate, screenshot, wait
• zero-schema auto() extraction from JSON-LD, microdata, tables and repeating containers
• resumable crawling, a distributed queue, and three storage backends
• a scheduler with persistent job state, so a cadence survives a restart
• data sinks with upsert by key, to Postgres, MySQL, ClickHouse, S3, a file or memory
• network capture, so you can find the API behind a page without opening DevTools
• JSONPath on JSON responses, without a dependency
• TypeScript definitions for the full public surface

The only runtime dependency is cheerio. Everything else is Node built-ins, including the DevTools Protocol client and its WebSocket.

Every exported interface, class, type and function is listed in the API reference, which is generated from index.d.ts and checked in CI.

Install

npm install sengkrep

Node 22.5 or newer, from both CommonJS and ESM:

import sengkrep from 'sengkrep';          // ESM: also named exports
const sengkrep = require('sengkrep');     // CJS: unchanged

The ESM entry exports every name from the CJS one, so import sengkrep, { create, jsonPath, Browser } from 'sengkrep' works. The 22.5 floor is what node:sqlite needs, which makes the sqlite storage backend available on every supported Node instead of only on some.

The package was published as sengkrep-ryna up to 3.4.0. That name is deprecated and receives no updates.

Before debugging anything, run sengkrep doctor. It reports the Node version, which optional drivers are installed, and whether the directories a scraper writes to are usable, so a missing dependency shows up as a WARN line instead of a confusing error later. Details in Command line.

Quick start

const sengkrep = require('sengkrep');

const data = await sengkrep.extract('https://example.com/product/1', {
  title: 'h1',
  price: { selector: '.price', transform: (v) => parseFloat(v.replace(/[^0-9.]/g, '')) },
  stock: { selector: '.stock', default: 'unknown' },
});

console.log(data.title, data.price);
console.log(data._sengkrep.health, data._sengkrep.diff);

With your own configured instance:

const scraper = sengkrep.create({
  retry: { max: 3 },
  circuitBreaker: { threshold: 5, cooldown: 60000 },
  cache: { ttl: 300, storage: 'memory' },
  rateLimit: { requestsPerSecond: 2 },
});

const page = await scraper.fetch('https://example.com');
const rows = await scraper.batch(['https://example.com/a', 'https://example.com/b'], { title: 'h1' });

await scraper.close();

Three ways to read a page

A page can hand you its data in three different shapes, and they need three different approaches. Pick the cheapest one that works.

| | 1. Plain HTTP | 2. DevTools Protocol | 3. Playwright | |---|---|---|---| | What it needs | nothing | a Chrome you already run | npm i playwright yourself | | Bundled with sengkrep | yes | yes | no, by design | | Sees server-rendered HTML | yes | yes | yes | | Sees HTML built by JavaScript | no | yes | yes | | Sees JSON the page fetches | yes, if you call the API directly | yes, through network capture | yes, through network capture | | Speeds | fastest | middle | slowest | | Memory | lowest | middle | highest |

Most sites can be handled with option 1, either by scraping the HTML or by calling the JSON API the page itself uses. Reach for option 2 when the HTML only exists after JavaScript runs. Option 3 is for when Playwright is already in your project and you would rather reuse it than run Chrome yourself.

1. Plain HTTP

const data = await sengkrep.extract('https://example.com/items', { title: 'h2.title' });

If the page calls a JSON API, capture that call once (see Network capture) and then hit the endpoint directly. This is the fastest and most stable option, because the response is data rather than markup.

2. DevTools Protocol, no extra install

sengkrep.renderers.cdp() returns a renderer backed by the DevTools Protocol. Start Chrome with the debugging port open, then let sengkrep ask it for the rendered HTML.

chrome --headless=new --remote-debugging-port=9222 https://app.example.com

const scraper = sengkrep.create({
  renderer: sengkrep.renderers.cdp({
    host: 'http://127.0.0.1:9222',
    waitForSelector: '#app',
  }),
  render: true,
});

const data = await scraper.extract('https://app.example.com/dashboard', { title: 'h1' });

| Option | Default | Notes | |---|---|---| | host | http://127.0.0.1:9222 | DevTools endpoint | | idleMs | 400 | Quiet time after load before reading the DOM | | waitForSelector | null | Poll until the selector exists, then continue | | waitForSelectorTimeout | 10000 | How long to keep polling before failing | | expression | document.documentElement.outerHTML | What to evaluate, returned as a string | | includeMeta | false | Return { html, url, title, readyState, consoleMsgs } instead of a string | | bodies, timeout, debuggerUrl, connect | same as CdpCapture | Browsers that need custom connection handling can pass connect and debuggerUrl |

3. Playwright, when it is already installed

Playwright is never bundled, and nothing breaks if it is missing. PlaywrightCapture requires the module at call time and fails with a clear message when it is absent. It drives capture on its own, and it becomes a renderer through a two-line adapter, since a renderer is any function that returns an HTML string or an object with an html property.

const capture = new sengkrep.PlaywrightCapture({ headless: true });

const scraper = sengkrep.create({
  renderer: async (url) => {
    const { html } = await capture.capture(url, { html: true });
    return html;
  },
  render: true,
});

capture() launches its own browser, waits for networkidle by default, returns the recorded entries alongside the HTML, and closes the browser again. Set waitUntil or timeout in the options when a site needs something different.

sengkrep capture playwright https://app.example.com --out session.har
sengkrep capture playwright https://app.example.com --headless=false

Step by step

The sections below go from a single request to a long-running job. Each one works on its own.

1. Get the raw response

const res = await scraper.fetch('https://example.com');
console.log(res.status, res.headers['content-type'], res.body.length);

The return value is { status, headers, url, body, binary, streamed, filePath, fromCache, notModified }. Nothing is parsed. Use it when you want the bytes, a status code, or a response you plan to handle yourself.

2. Extract with a schema

const data = await scraper.extract('https://books.toscrape.com', {
  title: { selector: 'h1', required: true },
  price: { selector: '.price_color', transform: (v) => parseFloat(v.replace(/[^0-9.]/g, '')) },
  tags: { selector: '.tag', multiple: true },
});

The response type is detected from the content. HTML uses CSS selectors, JSON uses path expressions, and RSS, Atom and CSV are parsed for you.

3. Run many URLs

const results = await scraper.batch(urls, { title: 'h1' }, { concurrency: 5 });

for await (const { url, data, error } of scraper.stream(urls, { title: 'h1' })) {
  if (!error) await save(data);
}

batch() returns [{ url, data, error }] in the order you passed the URLs. stream() is the same work as an async generator, so you can save each result while the rest are still running.

4. Follow pagination

const pages = await scraper.paginate(
  'https://example.com/items',
  { nextSelector: 'auto', itemsSelector: '.product', maxPages: 10, delayBetweenPages: 1200 },
  { title: 'h2', price: '.price' },
);

nextSelector: 'auto' looks for rel="next", common link text, common class names and numeric URL increments. Set stopOnDuplicate: false if a site legitimately repeats content across pages.

5. Crawl a site

const job = scraper.crawl({
  seed: 'https://example.com',
  schema: { title: 'h1' },
  follow: /\/article\//,
  maxUrls: 5000,
  concurrency: 3,
  stateFile: './crawl-state.json',
});

job.on('url:done', ({ url }) => console.log(url));
await job.start();

With stateFile the queue lives on disk, so a killed process resumes with job.resume() instead of starting over. respectRobotsTxt: true skips disallowed paths.

6. Export what you collected

const csv = await scraper.export(urls, { title: 'h1', price: '.price' }, {
  format: 'csv',
  path: './out.csv',
  pagination: { nextSelector: 'auto', itemsSelector: '.product', maxPages: 5 },
});

Formats are csv, json, ndjson and markdown. For result sets too large for memory, StreamWriter writes CSV or JSONL to disk as it goes, and Storage, MemoryStorage and SqliteStorage provide a keyed backend behind one createStorage() factory.

7. Turn on the reliability you need

const scraper = sengkrep.create({
  retry: { max: 3, respectRetryAfter: true },
  rateLimit: { requestsPerSecond: 2, concurrency: 4 },
  circuitBreaker: { threshold: 5, cooldown: 60000 },
  cache: { ttl: 300, staleWhileRevalidate: true, staleTtl: 120 },
  singleFlight: true,
  adaptive: true,
  dedupContent: true,
  health: { alertThreshold: 0.5, onAlert: (report) => console.warn(report) },
});

singleFlight: true merges identical in-flight requests, so fifty parallel calls for the same URL become one request and forty-nine waiters on its result. cache.staleWhileRevalidate serves a stale entry immediately and refreshes it in the background; await scraper.flush() waits for those background refreshes to finish. A cached response carries fromCache: true, and a stale one also carries stale: true.

8. Schedule the work

.start() runs a crawl once. When a job needs to run on a cadence and survive a restart, hand it to the scheduler.

const scheduler = scraper.scheduler;

scheduler.add({ id: 'catalog', schedule: '0 3 * * *' }, async () => {
  const rows = await scraper.batch(catalogUrls, { title: 'h1', price: '.price' });
  await save(rows);
});

scheduler.on('run:error', ({ job, error }) => console.error(job.id, error.message));
await scheduler.start();

The schedule and the last run are on disk, so after a restart start() picks the jobs back up. Details in Scheduling.

9. Send the rows somewhere

const sink = sengkrep.createSink({ type: 'postgres', table: 'items', key: 'id' });

await scraper.batch(urls, { id: '.sku', name: 'h1', price: '.price' }, { sink });
await sink.close();

Rows are written as they arrive, a row that carries an existing key replaces the stored one, and one failing batch is retried before it is reported. That is what the next section covers.

10. Close the scraper

scraper.close();

A scraper holds keep-alive sockets and, when observability.enabled is set, a metrics server. Without close() a finished script hangs. Scripts generated by capture.toScript() already call it.

Scheduling

A scraper that runs once is a script. A scraper that runs on a cadence needs to know what is due, what already ran, and what to do about a run that was missed while the process was down. Scheduler and JobStore handle that, and both sit on top of the same storage backends as the cache.

const scraper = sengkrep.create({
  logLevel: 'info',
  scheduler: {
    backend: 'file',
    storageDir: '.sengkrep-jobs',
    concurrency: 2,
    catchUp: false,
    jobs: [
      { id: 'catalog', schedule: '*/15 * * * *', handler: (job) => syncCatalog(job.data) },
      { id: 'report', schedule: { every: '6h' }, handler: () => buildReport() },
      { id: 'cleanup', schedule: { at: '2026-10-01T00:00:00Z' }, handler: () => dropTemp() },
    ],
  },
});

await scraper.scheduler.start();

The same jobs can be registered after the fact, which is how you keep a handler that needs the scraper itself.

const scheduler = scraper.scheduler ?? sengkrep.create({ scheduler: true }).scheduler;

scheduler.add({ id: 'catalog', schedule: '*/15 * * * *', data: { origin: 'https://example.com' } }, async (job, own) => {
  const rows = await scraper.batch(urlsFor(job.data.origin), { title: 'h1' });
  return rows.length;
});

Schedules

| Form | Example | Meaning | |---|---|---| | Cron | '0 3 * * *' | Five fields: minute, hour, day of month, month, day of week. UTC | | Interval | { every: '30s' } | ms, s, m, h or d. Also { everyMs: 30000 } or { ms: 30000 } | | One shot | { at: '2026-10-01T00:00:00Z' } | Runs once, then disables itself |

Cron fields accept *, a value, a range, a list, a step, or a name for months and weekdays: */5, 9-17, 0,30, 9-17/2, jan-mar, mon-fri. When both day fields are restricted, a day matches either one, which is what cron does. An expression that cannot match anything, like */0 * * * * or 0 25 * * *, throws when the job is added rather than failing silently later.

An interval keeps its alignment to the last run and skips missed slots instead of firing a burst to catch up.

Restarts and missed runs

Every change is written to the job store, so a restart does not lose the schedule. What happens to a run that was due while the process was down depends on catchUp.

| catchUp | Behaviour at start() | |---|---| | false (default) | The missed run is skipped, missed goes up by one, and the next slot is computed from now | | true | The job runs once as soon as the scheduler ticks, then returns to its normal cadence |

A missed run never turns into several runs. If the process was down for a week with a daily job, catchUp: true runs it once.

Running jobs

A job whose next slot arrives while it is still running is skipped, and stats().skipped counts it. That is the whole overlap policy: one instance of a job at a time. concurrency limits how many different jobs run in parallel.

| Call | Result | |---|---| | scheduler.add(job, handler) | Register or update a job, returns the stored record | | scheduler.list() | Every record, ordered by the next run | | scheduler.get(id) | One record, or null | | scheduler.runNow(id) | Run a job immediately, ignoring its schedule | | scheduler.pause(id) / resume(id) | Disable a job, or enable it with a fresh next run | | scheduler.remove(id) | Drop the record and its handler | | scheduler.tick() | Process everything that is due, returns how many jobs started | | scheduler.start() / stop() | Run the timer, or clear it. stop() waits for running jobs unless { wait: false } | | scheduler.stats() | ticks, runs, errors, skipped, jobs, enabled, running, started |

tick() is public on purpose: a test or a custom trigger can drive the schedule without waiting on a timer.

Events are run ({ job, result }), run:error ({ job, error }), tick ({ at, started }), start and stop.

The same scheduler is reachable from a terminal with sengkrep jobs and sengkrep run, with the job store on disk so a cadence survives a restart. See Command line.

What a job record holds

{
  id: 'catalog',
  name: 'catalog',
  schedule: '*/15 * * * *',
  enabled: true,
  data: null,
  nextRunAt: 1789000000000,
  lastRunAt: '2026-09-13T03:15:00.042Z',
  lastStatus: 'ok',
  lastError: null,
  lastDurationMs: 1840,
  runs: 96,
  failures: 0,
  missed: 1,
  createdAt: '2026-09-01T03:14:58.900Z',
  updatedAt: '2026-09-13T03:15:01.882Z',
}

nextRunAt is epoch milliseconds and always points at the future once a run has finished, so a job cannot strand itself in the past. A handler that throws does not stop the scheduler: the record keeps lastStatus: 'error', lastError and a failures count, and the next slot still runs.

JobStore can be used on its own, and any object with get, set, delete and list can be injected as its storage.

const store = new sengkrep.JobStore({ backend: 'sqlite', file: 'jobs.db' });
store.save({ id: 'nightly', schedule: '0 3 * * *', enabled: true, nextRunAt: Date.now() });
console.log(store.list(), store.all());

The cron helpers are exported too, which makes it possible to check what an expression means before trusting it.

sengkrep.cron.nextRunTime('*/5 * * * *', { now: Date.now() });
sengkrep.cron.scheduleLabel({ every: '6h' }); // 'every 6h'

Nothing in the scheduler is timezone aware. Schedules are evaluated in UTC, and a Date or ISO string in at carries its own offset.

Data sinks

Extraction produces rows. A sink moves them somewhere, in batches, with a key that decides whether a row is an insert or an update.

const sink = sengkrep.createSink({
  type: 'postgres',
  table: 'items',
  key: 'sku',
  batchSize: 500,
  connection: { connectionString: process.env.DATABASE_URL },
});

await scraper.batch(urls, { sku: '.sku', name: 'h1', price: '.price' }, { sink });
await sink.close();

INSERT INTO "items" ("sku", "name", "price") VALUES ($1, $2, $3), ($4, $5, $6)
ON CONFLICT ("sku") DO UPDATE SET "name" = EXCLUDED."name", "price" = EXCLUDED."price"

That statement is what the Postgres sink builds, and the same shape is available without a database.

| Sink | Driver | What key does | |---|---|---| | MemorySink | none | a row with an existing key replaces the stored one | | FileSink (JSONL) | none | the file is loaded, the row is replaced, the file is rewritten | | FileSink (CSV) | none | the same, and the header is written once | | PostgresSink | pg | ON CONFLICT (...) DO UPDATE | | MySQLSink | mysql2/promise | ON DUPLICATE KEY UPDATE | | ClickHouseSink | @clickhouse/client | deduplicates inside the batch; the table needs ReplacingMergeTree for real upsert | | S3Sink | @aws-sdk/client-s3 | one object per key, so a rerun overwrites that object |

No driver is a dependency of this package. Each one is required the first time it is needed, and a missing one fails with the exact npm install command in the message. You can also inject a client and skip the driver entirely, which is how the test suite exercises every statement without a database.

Options

| Option | Default | Meaning | |---|---|---| | key | none | Field name, or an array of them for a composite key. Without a key every row is appended | | replace | true | When false, rows are appended and the key is ignored | | batchSize | 500 | Buffered rows before a flush is triggered | | flushInterval | none | Milliseconds between automatic flushes for rows that arrive slowly | | retries | 2 | Extra attempts for a failed batch, with exponential backoff | | retryDelayMs | 200 | Base delay for that backoff | | transform | none | Runs on each row before the key is read | | onError | none | Called when a timer-driven flush fails | | columns | discovered | Explicit column list for the SQL, file and S3 sinks |

Behaviour

A row whose key is missing stops the batch with a clear error rather than landing somewhere unexpected. Two rows with the same key in one buffer collapse to the last one, and stats().duplicates counts that. A batch that keeps failing after retries throws, and the thrown error is the last real error, not a wrapper.

Buffered rows are held until batchSize is reached, a flushInterval fires, or you call flush(). close() flushes what is left, stops the timer and marks the sink closed, after which write() throws.

const sink = sengkrep.createSink({ type: 'memory', key: 'id' });

await sink.write([{ id: 1, name: 'a' }, { id: 1, name: 'b' }]);
await sink.flush();

sink.rows;      // [{ id: 1, name: 'b' }]
sink.stats();   // { written: 1, batches: 1, duplicates: 1, retried: 0, buffered: 0, closed: false, ... }

Passing a sink to the scraper

batch, stream, export and crawl all take a sink. Either form works, and the difference matters.

| What you pass | Ownership | |---|---| | A sink instance, for example new sengkrep.MemorySink({ key: 'id' }) | Yours. The scraper writes and flushes it, and leaves it open | | A descriptor, for example { type: 'jsonl', path: 'out.jsonl', key: 'id' } | The scraper creates it, flushes it and closes it when the run ends |

await scraper.batch(urls, schema, { sink: { type: 'jsonl', path: 'items.jsonl', key: 'sku' } });

for await (const { data, error } of scraper.stream(urls, schema, { sink: { type: 's3', bucket: 'data', prefix: 'items', key: 'sku' } })) {
  if (error) continue;
  console.log(data.sku);
}

A stream that is abandoned early still flushes, because the flush sits in a finally. Crawl writes each page as it is visited, so a long crawl does not have to hold every result in memory.

File sinks

FileSink infers the format from the extension. With replace on it loads the file once, keeps the keys in memory and rewrites the file on each flush, which is convenient up to a few hundred thousand rows and the wrong tool past that. Set replace: false to append instead, and give columns explicitly if later rows do not carry the same fields.

const sink = new sengkrep.FileSink('items.csv', { key: 'sku' });
await sink.write({ sku: 'A1', name: 'Roti' });
await sink.close();

fs.readFileSync('items.csv', 'utf8'); // 'sku,name\nA1,Roti\n'

A CSV file that cannot be parsed stops the batch, and the file is left exactly as it was.

Database and object storage

new sengkrep.PostgresSink({ table: 'items', key: 'sku', connection: { connectionString } });
new sengkrep.MySQLSink({ table: 'items', key: 'sku', connection: { uri } });
new sengkrep.ClickHouseSink({ table: 'items', connection: { url, username, password } });
new sengkrep.S3Sink({ bucket: 'data', prefix: 'items', key: 'sku', format: 'json' });

Any client you already have can be injected, and injected clients are never closed by the sink.

const sink = new sengkrep.PostgresSink({ table: 'items', key: 'sku', client: existingPool });
const mysql = new sengkrep.MySQLSink({ table: 'items', key: 'sku', client: pool });
const ch = new sengkrep.ClickHouseSink({ table: 'items', client: chClient });
const s3 = new sengkrep.S3Sink({ bucket: 'data', key: 'sku', put: async ({ key, body, contentType }) => upload(key, body, contentType) });

ClickHouse has no upsert, so the sink deduplicates within a batch and inserts with FORMAT JSONEachRow. Use a ReplacingMergeTree(key) table and query with FINAL if you need one row per key at read time. S3 has no upsert either, and does not need one: with a key set, each row becomes its own object named from that key, so writing the same key again replaces the same object.

Network capture

Finding the API behind a JavaScript-heavy page usually means opening DevTools and copying requests by hand. The capture layer does that step for you.

Four sources feed one analyzer.

| Source | Extra dependencies | What it sees | |---|---|---| | CdpCapture | none, the DevTools Protocol client and its WebSocket are built in | Full requests, responses, headers, bodies, WebSocket frames | | HarImporter | none | Whatever a HAR file contains | | CaptureProxy | none | Full bodies for plain HTTP, plus CONNECT metadata for HTTPS tunnels | | PlaywrightCapture | your own Playwright install, never bundled | Full requests and responses |

From a running browser

chrome --headless=new --remote-debugging-port=9222 https://app.example.com

const capture = await sengkrep.NetworkCapture.fromCdp({
  url: 'https://app.example.com/dashboard',
  host: 'http://127.0.0.1:9222',
});

for (const endpoint of capture.endpoints()) {
  console.log(endpoint.method, endpoint.template, endpoint.statuses, endpoint.params);
}

CdpCapture creates its own target, enables Page and Network, waits for Page.loadEventFired, reads response bodies with Network.getResponseBody, and closes the target when it is done.

Through a local proxy

const { capture, proxy } = await sengkrep.NetworkCapture.fromProxy({ port: 8899 });
console.log(`route your client through ${proxy.address.url}`);

const scraper = sengkrep.create({ proxies: [proxy.address.url] });
await scraper.fetch('http://example.com/catalog');

await new Promise((resolve) => setTimeout(resolve, 2000));
capture.pull(proxy);
await proxy.stop();
console.log(capture.summary());

For https:// targets a proxy without TLS interception only reports CONNECT metadata: host, port, bytes and duration. That is how TLS works, not a gap in the library. Use CdpCapture or PlaywrightCapture when the bodies matter.

From a HAR file

const capture = sengkrep.NetworkCapture.fromHar('./session.har');
console.log(capture.endpoints({ bodies: true }));

Reading the results

| Call | Returns | |---|---| | capture.entries | Every normalized request and response | | capture.api() | Only XHR, fetch and JSON traffic, as another capture | | capture.filter({ method, status, host, url, api, failed }) | A filtered capture | | capture.endpoints({ all }) | Grouped endpoints with template, count, statuses, params and an inferred JSON schema. Static requests are excluded unless all: true | | capture.summary() | Counts by resource type, status and source | | capture.toFetchCode(entry) | A fetch() snippet for one request | | capture.toCurl(entry) | The same request as a curl command | | capture.toSchema(endpoint) | An extraction schema derived from the captured JSON body | | capture.toScript(endpoint) | A runnable scraper script for that endpoint | | capture.frames(id) | WebSocket frames recorded for one socket | | capture.saveHar('out.har') | HAR 1.2 export | | capture.json() | The whole analysis as a JSON-serializable object |

Numeric, UUID and opaque segments collapse into :id, so /v1/items/17 and /v1/items/8321 land in the same row:

const [endpoint] = capture.endpoints();
// endpoint.template === 'https://api.example.com/v1/items/:id'
// endpoint.schema   === { type: 'object', properties: { id: { type: 'integer' }, ... } }

toHAR(), toCurl() and toFetchCode() redact Authorization, Cookie, Set-Cookie, proxy credentials and API key headers by default. Pass { redact: false } to keep the original values.

From capture to a scraper

Open the page once, then let the capture write the code you would otherwise type by hand. The schema step needs a JSON response, because JSON is what the library can turn into paths.

const capture = await sengkrep.captureUrl('https://app.example.com/dashboard');
const [endpoint] = capture.endpoints();

const schema = capture.toSchema(endpoint);
// { title: 'items[].title', total: 'total' }

const script = capture.toScript(endpoint);
// a complete .js file using extract(), Retry and RateLimiter

Nested objects become dotted paths, arrays become wildcards, and collisions take a longer key.

| Captured JSON | Schema | |---|---| | { items: [{ id }] } | { id: 'items[].id' } | | { data: { user: { name } } } | { name: 'data.user.name' } | | { id, data: { id } } | { id: 'id', data_id: 'data.id' } | | { tags: ['a'] } | { tags: 'tags[]' } |

toScript() writes a file that ends with scraper.close(), so it exits on its own. Options: { require, logLevel, schemaObject, params, headers, body, redact, maxDepth }. Headers and bodies are redacted by default, the same way toFetchCode() does it.

sengkrep capture har session.har --schema
sengkrep capture har session.har --script-out scraper.js

WebSocket frames

Network.webSocketFrameSent and Network.webSocketFrameReceived are recorded per socket. Read them back with capture.frames(id), and cap them with maxFramesPerSocket. When the cap is hit, framesTruncated is set on the entry.

Cookies

The same DevTools connection can hand its cookies to a CookieJar, and a cookies.txt file works too. This saves a login round trip when the session already exists in the browser.

const scraper = sengkrep.create({ cookies: true });

await sengkrep.importCookies(scraper.cookieJar, { from: 'cdp' });
await sengkrep.importCookies(scraper.cookieJar, { from: 'file', path: 'cookies.txt', domains: ['app.example.com'] });

| from | Reads | |---|---| | file | Netscape cookies.txt, including #HttpOnly_ lines. Needs path | | text | The same format from a string. Needs text | | json | Cookie JSON, either a bare array or { cookies: [...] } | | cdp | Network.getAllCookies from the browser. browser is an accepted alias | | cookies | An array of cookie objects you already have |

In the Netscape format the second column marks whether a cookie applies to subdomains. It is not the HttpOnly flag; that comes from the #HttpOnly_ prefix on the line.

Capture from the command line

sengkrep capture har session.har --out api.har --json
sengkrep capture browser https://app.example.com --cdp http://127.0.0.1:9222
sengkrep capture proxy --port 8899 --seconds 60 --out session.har
sengkrep capture playwright https://app.example.com --out session.har
sengkrep capture cookies cookies.txt --domain app.example.com --out jar.json

| Flag | Meaning | |---|---| | --out <path> | Write a HAR file with every captured request | | --json | Print the full analysis as JSON | | --all | Include static assets as well as API calls | | --code | Print a fetch() snippet for the first endpoint | | --schema | Print an extraction schema for the first JSON endpoint | | --script | Print a runnable scraper script for the first JSON endpoint | | --script-out <path> | Write that script to a file | | --cdp <host> | DevTools endpoint, default http://127.0.0.1:9222 | | --port <n> | Local capture proxy port, default 8899 | | --seconds <n> | How long to keep the proxy open, default 30 | | --headless=false | Show the browser when using Playwright |

When --schema, --script or --script-out is used, the text report is not printed, so stdout stays clean for a pipe.

Anti-block

A scraper that gets a response is not the same as a scraper that gets the data. Bot walls (Cloudflare, DataDome, PerimeterX, Akamai, Imperva, Kasada, AWS WAF, Sucuri) answer with a challenge page, a CAPTCHA, or a 403 that looks like any other error. sengkrep names what happened, then lets you decide what to do about it.

Recognise the wall

const client = sengkrep.create({ blocks: true });

const result = await client.extract(url, schema);
console.log(result._sengkrep.block);
// { blocked: true, vendor: 'cloudflare', kind: 'challenge', confidence: 'high', ... }

blocks has three modes:

| Mode | What happens on a block | |---|---| | report (default) | The verdict is attached to result._sengkrep.block and the response is returned | | retry | A BlockError is thrown so Retry tries again with the next identity and proxy, up to retry.max | | throw | A BlockError is thrown on the first block |

Use the mode that matches the job: report to measure how often you are blocked, retry for a normal scrape, throw when you would rather fail fast than burn requests.

A blocked 403 is still returned in report mode so the metadata exists, but any other 4xx stays an error. That is the one behaviour blocks changes about status handling.

Check without scraping

const verdict = await client.probe(url);
console.log(verdict.blocked, verdict.verdict.vendor, verdict.verdict.signals);

probe() sends one request and returns { url, finalUrl, status, blocked, verdict, headers, fromCache } without throwing on a non-2xx. It works with blocks off too, since it creates a detector when one is not configured. The CLI wraps it and exits 2 when a wall is found, so a shell can branch on the result.

Detector on its own

const detector = new sengkrep.BlockDetector();
const verdict = detector.detect({ status: 403, headers: response.headers, body });

A verdict carries blocked, confidence (none to high), vendor, vendorName, kind (challenge, captcha, rate-limit, denied), retryable, status, the signals that matched, and a timestamp. Vendor signatures are data, so a site-specific wall can be added without touching the library:

detector.addSignature({
  id: 'acme',
  name: 'Acme WAF',
  kind: 'denied',
  headers: [{ name: 'x-acme-block' }],
  body: [{ label: 'acme', pattern: /acme waf/i }],
});

minConfidence defaults to low. Raise it to high when a single weak signal should not count as a block.

Identities

An identity is one coherent browser: user agent, client hints, locale, timezone, viewport, memory and CPU count. A pool keeps one identity per session, so a site sees the same visitor across requests, and rotates when it is blocked.

const client = sengkrep.create({
  blocks: { mode: 'retry' },
  identity: { size: 12, rotation: 'sticky' },
});

| Option | Meaning | |---|---| | size | How many identities to generate when none are given | | identities | Your own list, each an Identity or a plain spec | | rotation | sticky by session (default), round-robin, or random | | rotateOnBlock | Choose a new identity after a detected block (default true) | | identitySession | What to key session stickiness on. Defaults to the hostname |

An identity also reaches the renderer, so a CDP or Playwright session can present the same locale and viewport as its HTTP requests.

Proxy sessions

When a proxy URL carries the {session} placeholder, the session id is substituted for you, so a rotating proxy can still keep one exit IP per identity:

const client = sengkrep.create({
  proxies: ['http://user-{session}:[email protected]:8080'],
  proxyStrategy: 'sticky',
  identity: { size: 8 },
});

ProxyRotator.next(host, { session }) returns http://user-abc123:[email protected]:8080 for session abc123, the same URL for the same session, and a different one for a different session. Failures are counted against the template rather than each resolved URL, so a banned gateway is still retired.

Watching block events

Every detected block fires the onBlock webhook with the URL, the verdict, the identity in use and the attempt number. The payload is the same shape as the other events:

const client = sengkrep.create({
  blocks: true,
  webhook: { onBlock: 'https://hooks.example/sengkrep' },
});

Browser automation

Browser drives a real Chrome over the DevTools Protocol, through the same connection code as the capture layer, with no Playwright and no Puppeteer:

const browser = await sengkrep.Browser.connect();

await browser.goto('https://app.example.com');
await browser.waitForSelector('.dashboard');
await browser.click('button.load-more');
await browser.type('#search', 'kopi');

const html = await browser.html();
const title = await browser.title();
const price = await browser.evaluate(() => document.querySelector('.price')?.textContent);
const shot = await browser.screenshot({ fullPage: true });

browser.close();

| Call | What it does | |---|---| | goto(url, options?) | Navigate and wait for the load event, then optionally waitForSelector | | evaluate(fn or expression) | Run JavaScript in the page and return the value | | html() / text() | document.documentElement.outerHTML, or the page's inner text | | url() / title() | Current location and document title | | waitForSelector(selector, options?) | Poll until the element exists; SELECTOR_TIMEOUT when it never does | | click(selector, options?) | Scroll into view, then move, press and release at the element's center | | type(selector, text, options?) | Focus the field, send real key events, fire input and change | | screenshot(options?) | PNG or JPEG of the viewport or the full page, as a Buffer | | pdf(options?) | Print the page to PDF, as a Buffer | | scroll(options?) | window.scrollTo, with an optional settle delay | | close() | Close the target and the connection |

Options: viewport sets the window size and mobile flag through Emulation.setDeviceMetricsOverride, userAgent overrides the browser's own, and timeout bounds every wait. click and type wait for their selector first, so a slow render does not turn into a silent no-op.

Because it is the same CDP client the capture layer uses, a Browser session and a NetworkCapture can look at the same browser. The existing renderers.cdp() is this class reduced to one goto plus outerHTML, so render: true keeps working unchanged.

Zero-schema extraction

Writing a schema needs a page in front of you. auto() reads what the page volunteers instead, which is enough for a first pass, a diff baseline, or a quick answer:

const result = await sengkrep.auto('https://shop.example.com/product/123');

result.title;         // <title> or og:title
result.description;   // meta description or og:description
result.item;          // JSON-LD or microdata entity, e.g. { name, offers: { price } }
result.items;         // repeating containers: cards, list rows, articles
result.tables;        // every <table> as rows of header-keyed objects
result.sources;       // where each part came from: ['json-ld', 'tables', 'repeating:.product-card']
result.text;          // the page text, capped at maxText (default 2000)

Detection order: structured data first, because JSON-LD and microdata are the most reliable thing on the page; then tables; then repeating containers, found by grouping elements that share a class and appear three or more times; then data-* attributes. Every source is labeled in sources, so you know what to trust and what to verify.

auto() is also a good step before writing a schema: run it, look at item and items[0], and the selectors you need are usually the class names on those objects. It accepts render: true, the same request options as extract(), and skipRepeating, dataAttributes: false, text: false and maxText to trim the work.

Command line

The package installs a sengkrep command. sengkrep help prints the same list, and every command is a thin wrapper over the library, so anything the CLI does can be done from code.

fetch and discover

sengkrep fetch https://example.com
sengkrep discover https://example.com

fetch prints the response body. discover prints the sitemap and robots.txt URLs it found, one per line.

scrape

sengkrep scrape https://books.toscrape.com --schema '{"title":"h1"}' --format csv --output books.csv

| Flag | Meaning | |---|---| | --schema <json> | Required. The extraction schema, as a JSON string | | --format <fmt> | csv, json, ndjson or markdown. Default json | | --output <path> | Write to a file instead of stdout | | --pages <n> | Paginate up to n pages | | --next <selector> | Next-page link selector, or auto | | --items <selector> | Selector for the repeated item containers | | --proxy <url> | Route requests through a proxy | | --delay <ms> | Base delay between requests | | --sink <json> | Sink descriptor, for example {"type":"file","path":"rows.jsonl","key":"id"} |

--sink takes the same descriptor createSink() accepts, so a scrape can land in a file, Postgres, MySQL, ClickHouse or S3 without a script around it:

sengkrep scrape https://books.toscrape.com \
  --schema '{"sku":".sku","price":".price"}' \
  --sink '{"type":"postgres","table":"books","key":"sku"}'

The sink is created, flushed and closed for you. A malformed descriptor is rejected before a request is sent. Flags accept both --flag value and --flag=value, so the --headless=false written in Network capture behaves the way it reads.

probe

sengkrep probe https://example.com
sengkrep probe https://example.com --json
sengkrep probe https://example.com --method POST --proxy http://gw.example:8080

Sends one request and prints the verdict: status, whether a bot wall was recognized, which vendor, what kind of wall, how confident the match is, and the signals behind it. Exit codes: 0 when the response looks normal, 2 when a bot wall is detected, 1 on a request error, so a shell can branch on the result.

sengkrep probe "$(cat urls.txt)" >/dev/null || echo blocked

jobs and run

A jobs file exports an array of jobs, or an object with jobs, concurrency and catchUp. Handlers are ordinary functions and can require('sengkrep'), so a job file is a complete scraper with a schedule attached.

// jobs.js
module.exports = {
  concurrency: 2,
  jobs: [
    { id: 'catalog', schedule: '*/15 * * * *', handler: async () => sync() },
    { id: 'nightly', schedule: { every: '6h' }, handler: async () => report() },
    { id: 'launch',  schedule: { at: '2026-10-01T00:00:00Z' }, handler: async () => drop() },
  ],
};

sengkrep jobs jobs.js                 # what is registered, and when each one runs next
sengkrep jobs jobs.js --json          # the stored records
sengkrep run jobs.js --due            # run the jobs whose slot has arrived, then exit
sengkrep run jobs.js --once           # run every enabled job once, then exit
sengkrep run jobs.js --job catalog    # run one job now
sengkrep run jobs.js --watch          # keep the scheduler running until Ctrl+C

| Flag | Meaning | |---|---| | --dir <path> | Job store directory. Default .sengkrep-jobs | | --job <id> | Run one job and exit | | --once | Run every enabled job once and exit | | --due | Run only the jobs whose slot has arrived, then exit | | --watch | Keep running until SIGINT. A handler that throws is reported and does not stop the loop | | --catch-up | Also run a slot missed while the process was down | | --concurrency <n> | How many jobs may run at once | | --json | Print records as JSON |

The command exits non-zero when a handler throws, so a container or a cron wrapper notices. sengkrep jobs shows the next run as a relative time plus the absolute timestamp, and the same record fields the library keeps: lastStatus, runs, failures and missed.

doctor

sengkrep doctor
sengkrep doctor --host internal.example.com --cdp http://127.0.0.1:9222
sengkrep doctor --json

doctor checks what decides which features work, before you spend time on a request that was never going to succeed:

• the Node version against the >=22.5.0 floor
• whether node:sqlite is available, which gates the sqlite storage backend
• whether this Node build can decompress zstd, which decides if Accept-Encoding advertises it
• whether the temp directory and the working directory are writable, which streaming and cache state need
• whether each optional driver is installed: playwright, pg, mysql2, @clickhouse/client, @aws-sdk/client-s3
• DNS resolution for --host, and a DevTools endpoint when --cdp is given

Each line is PASS, WARN or FAIL. A WARN means an optional feature is off; a FAIL means something required is broken, and the command exits 1 so a setup script or CI can act on it. --json prints the report as { ok, node, platform, checks, failures, warnings }. All of it is also callable as sengkrep.doctor(options), which returns that same object.

const report = await sengkrep.doctor({ skipNetwork: true });
for (const check of report.checks) console.log(check.status, check.name, check.detail);

Core API

These methods exist on the default export, which holds one shared instance, and on any instance from sengkrep.create(options).

fetch(url, options?)

Returns the raw response: { status, headers, url, body, binary, streamed, filePath, fromCache, notModified }. No extraction, no schema.

const res = await scraper.fetch('https://example.com');
console.log(res.status, res.body.length);

load(html)

Loads an HTML string into cheerio for manual inspection. No network call.

const $ = scraper.load('<h1>Hi</h1>');
console.log($('h1').text());

extract<T>(url, schema, options?)

Fetches a URL, runs the schema, then records health, diff and validation metadata on data._sengkrep. The response type is detected from the content: HTML, JSON, RSS/Atom, or CSV.

Options include strict (throw on validation failure), render (use the configured renderer for this call), and request (per-call request config such as headers, timeout or rejectUnauthorized).

batch<T>(urls, schema, options?)

Runs extract() over a list with a concurrency limit. Returns [{ url, data, error }] in the order the URLs were given.

const results = await scraper.batch(urls, { title: 'h1' }, { concurrency: 5 });

stream<T>(urls, schema, options?)

An async generator over the same work, so results can be handled while the rest are still running.

for await (const { url, data, error } of scraper.stream(urls, schema, { concurrency: 5 })) {
  if (!error) await save(data);
}

paginate<T>(startUrl, config, schema, options?)

Follows a next-page link and collects items from every page.

| Option | Default | Meaning | |---|---|---| | nextSelector | required | CSS selector for the next link, or auto for the built-in detector | | itemsSelector | none | Selector for repeated item containers. Without it each page is one object | | maxPages | 10 | Page limit | | delayBetweenPages | 1200 | Delay between pages, jittered | | stopOnDuplicate | true | Stop when a page extracts the same content as the previous one |

crawl(options)

Breadth-first crawler with optional disk-backed state, so a crashed process can resume where it stopped.

| Option | Default | Meaning | |---|---|---| | seed | required | Starting URL or list of URLs | | schema | required | Schema applied to every page | | follow | none | RegExp or predicate that decides which links enter the queue | | maxUrls | 1000 | Queue limit | | concurrency | 3 | Parallel requests | | stateFile | none | JSON file for progress. Omit for an in-memory run | | respectRobotsTxt | false | Skip paths disallowed by robots.txt |

Returns a CrawlJob with .start(), .resume(), .pause(), .results(), .stats() and .on(event, fn) for url:done, url:error, progress, start and done.

const job = scraper.crawl({
  seed: 'https://example.com',
  schema: { title: 'h1' },
  follow: /\/article\//,
  maxUrls: 5000,
  stateFile: './crawl-state.json',
});

job.on('url:done', ({ url }) => console.log(url));
await job.start();

export(input, schema?, options?)

Runs the pipeline and serializes the result as csv, json, ndjson or markdown. Accepts a URL, a list of URLs, or data that is already extracted.

const csv = await scraper.export(urls, { title: 'h1', price: '.price' }, {
  format: 'csv',
  path: './out.csv',
  pagination: { nextSelector: 'auto', itemsSelector: '.product_pod', maxPages: 5 },
});

discover(origin, options?)

Reads robots.txt and sitemap.xml for an origin and returns the URLs they list.

isAllowed(url, userAgent?) and getCrawlDelay(origin, userAgent?)

Check robots.txt for a single URL before requesting it. getCrawlDelay() returns the Crawl-delay value in seconds, or null.

login(url, formData?, options?)

Submits a login form through FormHandler and keeps the resulting cookies in the jar. Returns true when the response does not look like a failed login.

submitForm(url, formSelector, overrides?)

Reads a form from the page, fills the fields with the overrides, and submits it.

inferSchema(url, options?)

Reads a page and suggests selectors for common fields, plus any repeating container it finds.

const suggestion = await scraper.inferSchema('https://books.toscrape.com');
console.log(suggestion.type, suggestion.container, suggestion.schema);

flush() and close()

flush() waits for background cache revalidations and resolves to how many were pending. close() releases keep-alive sockets, temp files and the metrics server.

Health, diff, validation and rate-limit metadata are attached to extraction results under _sengkrep, which is non-enumerable, so it stays out of JSON.stringify(data).

Schema syntax

A schema is an object of field name to selector. Short form:

const schema = { title: 'h1', price: '.price_color' };

Long form:

const schema = {
  title: { selector: ['h1.title', 'h1'], required: true },
  image: { selector: 'img.cover', attr: 'src' },
  tags: { selector: '.tag', multiple: true },
  price: { selector: '.price', transform: (v) => parseFloat(v.replace(/[^0-9.]/g, '')) },
  published: { selector: 'time', attr: 'datetime', pattern: /^\d{4}-\d{2}-\d{2}$/ },
  stock: { selector: '.stock', default: 'unknown' },
};

A selector can be a string or an array, and an array is tried in order until one matches. type: 'html' returns inner HTML instead of trimmed text. required: true raises ExtractionError when the field comes back empty.

For JSON responses the schema follows the same shape with path instead of selector, and a path written as a JSONPath expression is evaluated by the built-in JSONPath engine:

const schema = {
  id: { path: 'data.items[0].id', required: true },
  names: 'data.items[*].name',
  total: { path: 'meta.total', transform: (v) => Number(v) },
  cheap:  '$.data.items[?(@.price < 10)].title',
  deep:   '$..author.name',
  window: '$.store.book[0:3].title',
};

The engine supports child access, [*] wildcards, indexes including negatives, slices [0:3], unions [0,2], recursive descent ..key, and filters [?(@.price > 15)] with ==, !=, >, <, >=, <= and =~. It is exported as sengkrep.jsonPath(data, expression) for use on its own.

Modules

Every class below is exported from the package root and covered by index.d.ts.

Reliability:

| Class | Purpose | |---|---| | Retry | Backoff with jitter, status allowlist, Retry-After support, a retryable BLOCKED code, and an optional total budgetMs | | BlockDetector | Classifies a response as a Cloudflare, DataDome, PerimeterX, Akamai, Imperva, Kasada, AWS WAF, Sucuri or CAPTCHA wall, with a confidence and the signals that matched | | SingleFlight | Merges identical in-flight requests into one and shares the result or the error with every waiter | | CircuitBreaker | Opens after repeated failures on a key and closes again after a cooldown | | HealthMonitor | Tracks field fill rates and selector matches over a rolling window, then alerts | | DiffDetector | Compares structured results across runs and reports changes by severity | | SchemaValidator | Per-field rules: required, type, pattern, minLength, maxItems, custom | | Incremental | Sends If-None-Match and If-Modified-Since, and reports notModified | | Cache | TTL cache with an LRU cap, in memory or on disk, with optional stale-while-revalidate | | CrawlQueue | Disk-backed URL queue with deduplication and resume |

Identity and access:

| Class | Purpose | |---|---| | Fingerprint | One browser profile drives the User-Agent, client hints, Sec-Fetch-* and language together, or one pooled identity does | | Identity / IdentityPool | Coherent browser identities (UA, client hints, locale, timezone, viewport) kept per session and rotated on block | | CookieJar | Cookie storage per registrable host, including IPv4 and IPv6 hosts | | RateLimiter | Serialized per-host spacing at a fixed rate | | ProxyRotator | Round robin, random or sticky proxy selection with failure tracking and {session} substitution for per-identity exits | | SessionPool | Reusable sessions with cookies and user agents, round robin or least used | | AuthManager | Bearer or JWT auth with a refresh hook on 401 | | CsrfHandler | Reads CSRF tokens from meta tags, hidden inputs and cookies, then replays them | | SecurityGuard | Blocks private address ranges, allowlisted and blocklisted domains, and configured ports |

Transport and performance:

| Class | Purpose | |---|---| | Transport | One interface over HTTP/1.1 and HTTP/2, with automatic downgrade when the server refuses h2 | | Fetcher | HTTP/1.1 with manual redirects, streaming decompression and truncated-response detection | | Http2Fetcher | The same contract over HTTP/2, including cancellation and byte accounting | | DnsCache | Caches DNS lookups for a TTL, which SecurityGuard also uses to pin addresses | | AdaptiveThrottle | AIMD concurrency and delay per host, fed by 429, 503 and timeouts | | ContentDedup | SimHash and Hamming distance to drop near-duplicate pages | | StreamWriter | Writes large result sets to disk as CSV or JSONL instead of holding them in memory |

Extraction and data:

| Class | Purpose | |---|---| | Extractor | CSS extraction with fallback chains and transforms | | JsonExtractor | Path expressions with wildcards for JSON responses | | SchemaInference | Suggests selectors for common fields and finds repeating containers | | FormHandler | Reads, fills and submits forms |

Operations and storage:

| Class | Purpose | |---|---| | Observability | Counters per domain with a Prometheus text endpoint | | Webhook | Event delivery with exponential backoff and optional HMAC-SHA256 signing | | HarRecorder | Records requests through the interceptors and writes a HAR file | | DistributedQueue | Multi-worker queue with leases, priorities, per-worker streaks and a dead-letter list | | Scheduler | Runs stored jobs on cron, interval or one-shot schedules, with catch-up and skip policies | | JobStore | Persistent job records behind the same file, memory and sqlite backends as the cache | | doctor() | Environment report: Node, node:sqlite, zstd, writable directories, optional drivers, DNS and a DevTools endpoint | | Sink | Batching, key-based upsert, retries and stats shared by every sink | | MemorySink, FileSink | In-memory rows, or JSONL and CSV files with an append mode | | PostgresSink, MySQLSink, ClickHouseSink, S3Sink | Driver-backed sinks behind createSink() | | Storage / MemoryStorage / SqliteStorage | Backends behind one createStorage() factory | | PluginSystem | beforeRequest, afterExtract and onError hooks, with timestamp, logToFile and fieldMapper built in | | ProgressBar | Terminal progress for long runs | | WordPress, GraphQLClient | WP REST helper and GraphQL client | | UrlDeduplicator | Normalized-URL deduplication, also used by crawl() |

Capture:

| Class | Purpose | |---|---| | NetworkCapture | Merges capture sources and produces endpoints, schemas and code snippets | | CdpCapture | DevTools Protocol capture with no extra dependencies | | CdpRenderer | Renders a page through the DevTools Protocol and returns its HTML | | CaptureProxy | Local HTTP proxy that records what passes through it | | PlaywrightCapture | Attaches to Playwright pages when Playwright is installed | | HarImporter | Parses and writes HAR 1.2 |

Configuration

const scraper = sengkrep.create({
  logLevel: 'info',
  timeout: 30000,
  retry: { max: 3, respectRetryAfter: true },
  rateLimit: { requestsPerSecond: 2, concurrency: 4 },
  cache: { ttl: 300, storage: 'memory', staleWhileRevalidate: true, staleTtl: 120 },
  singleFlight: true,
  circuitBreaker: { threshold: 5, cooldown: 60000 },
  proxies: ['http://user:pass@proxy1:8080', 'http://proxy2:8080'],
  http2: true,
  adaptive: true,
  dedupContent: true,
});

Options and defaults:

| Option | Default | Notes | |---|---|---| | logLevel | 'info' | error, warn, info or debug | | logPretty | true | Human-readable log lines | | baseURL | null | Prefix for relative URLs | | timeout | 30000 | Per-request timeout in ms | | connectTimeout | null | Overrides the connect phase only | | totalTimeout | null | Hard deadline for a whole request | | maxRedirects | 5 | Manual redirect following | | redirectPolicy | { validateEachHop: true, forwardSensitiveHeaders: false, maxCrossHostHops: 3 } | Security guard runs on every hop, credentials are dropped when the origin changes, and cross-host hops are capped | | keepAlive | true | Reuse sockets | | delay | none | Fixed-ish delay in ms. The actual wait is jittered between 0.8x and 1.2x | | delayMin, delayMax | 500, 2500 | Range for the jittered delay, used when delay is not set | | maxMemoryBuffer | 10 MB | Above this, responses stream to disk | | responseType | 'auto' | auto, html, json, rss or csv | | cookies | true | Enable the cookie jar | | http2 | false | Use HTTP/2 for HTTPS origins, with fallback | | fingerprint | see below | { userAgent, rotateUAOnEachRequest, randomizeHeaderOrder, randomizeTiming } | | retry | { max: 3 } | retryOn, retryOnNetwork, retryOnTimeout, respectRetryAfter, maxRetryAfter, budgetMs, jitter | | singleFlight | false | true or { maxKeys }. Identical in-flight requests share one response | | scheduler | disabled | true or { backend, storageDir, file, table, concurrency, catchUp, pollInterval, now, jobs, store } | | health | enabled | { alertThreshold: 0.5, windowSize: 10, onAlert } or false | | diff | enabled | { storageDir: '.sengkrep', sensitivity: 'structural', onDiff, maxHistory: 500, backend } or false | | cache | false | { ttl, storage: 'memory' \| 'disk', storageDir, maxItems, backend, staleWhileRevalidate, staleTtl } | | circuitBreaker | false | { threshold, cooldown, halfOpenMaxAttempts, onOpen, onClose } | | incremental | false | true or { storageDir, backend }. Default directory is .sengkrep-incremental | | rateLimit | disabled | { requestsPerSecond, concurrency } | | proxies | [] | URL list for ProxyRotator | | proxyStrategy | 'round-robin' | round-robin, random or sticky | | proxyMaxFailures | 3 | Failures before a proxy is skipped | | dns | false | true or { ttl } | | sessionPool | { size: 1 } | strategy is round-robin or least-used | | security | disabled ports only | { blockPrivateIPs, allowDomains, blockDomains, blockedPorts } | | auth | none | { type: 'bearer', token, refresh, refreshOn } | | csrf | { auto: true } | Automatic token replay | | observability | { enabled: false } | { enabled, port } for the metrics endpoint | | webhook | none | { onStart, onComplete, onError, onBlock, onProgress, retries, secret } | | har | false | Record every request into a HAR file | | robotsTtl, tempFileTtl | 3600000 | Cache lifetime for robots.txt, cleanup age for streamed temp files | | adaptive | false | true or { minConcurrency, maxConcurrency, backoffFactor, baseDelay } | | dedupContent | false | true or { threshold, shingleSize, maxEntries } | | backend | 'file' | Storage backend for cache, diff and incremental: file, memory or sqlite. Each of those modules also takes storageDir, file and table | | renderer, render | none | renderer(url, options) returns HTML, or sengkrep.renderers.cdp() for the built-in one; render: true applies it to every extract() | | compliance | false | { userAgent, respectXRobotsTag, maskFields, auditLog, purpose } | | validate | {} | Per-field validation rules | | blocks | false | true or { mode: 'report' \| 'retry' \| 'throw', minConfidence, statuses, signatures }. Detects bot walls on every response | | identity | false | true or { size, identities, rotation, rotateOnBlock }. Keeps one coherent browser identity per session | | identitySession | hostname | What to key identity stickiness on | | identities | none | A direct list of Identity objects or plain specs, as an alternative to identity |

Validation rules

sengkrep.create({
  validate: {
    price: { required: true, type: 'number', pattern: /^\d+(\.\d{2})?$/ },
    email: { type: 'email' },
    tags: { minItems: 1 },
    name: { minLength: 2, maxLength: 200, notEmpty: true },
    score: { custom: (value) => (value >= 0 && value <= 100) || 'score out of range' },
  },
});

type accepts string, number, boolean, url, email or date. Results appear at data._sengkrep.validation, and extract(url, schema, { strict: true }) throws ValidationError instead.

Plugins

Hooks run at three points: beforeRequest, afterExtract and onError. Register one with scraper.plugins.use(plugin) or scraper.plugins.hook(name, fn).

const scraper = sengkrep.create();

scraper.plugins.use(sengkrep.plugins.timestamp('scrapedAt'));
scraper.plugins.use(sengkrep.plugins.fieldMapper({ priceText: 'price' }));

scraper.plugins.hook('afterExtract', ({ data, meta }) => {
  return { data: { ...data, source: 'example.com' }, meta };
});

The built-in factories are timestamp(fieldName), logToFile(filePath) and fieldMapper(mapping). A hook returning undefined leaves the payload untouched.

Property naming

Some sub-clients are reachable under two names. Both names point to the same instance.

| Short | Long | |---|---| | scraper.auth | scraper.authManager | | scraper.security | scraper.securityGuard | | scraper.csrf | scraper.csrfHandler | | scraper.health | scraper.healthMonitor | | scraper.diff | scraper.diffDetector | | scraper.plugins | scraper.pluginSystem | | scraper.cache | scraper.cacheManager | | scraper.scheduler | only exists when the scheduler is configured, otherwise null |

Always present, whatever the configuration: sessionPool, wordpress, graphql, formHandler, deduplicator, proxyRotator, rateLimiter, fingerprint, observability, singleFlight, interceptors, cookieJar.

Present only when enabled, otherwise null: cache, circuitBreaker, incremental, diff, health.

Errors

const { errors } = require('sengkrep');

| Class | code | Meaning | |---|---|---| | FetchError | HTTP_ERROR | Status 400 or above. err.status holds the code | | FetchError | NETWORK_ERROR | The connection failed | | FetchError | UNSUPPORTED_ENCODING | The server used a Content-Encoding this Node build cannot decompress | | FetchError | TRUNCATED_RESPONSE | The connection dropped before the response ended. Retried automatically | | TimeoutError | TIMEOUT | The request passed its timeout | | CanceledError | CANCELED | An `Abo