@browserless.io/anti-bot-classifier

A pure, dependency-free classifier that decides whether a page load was blocked or challenged by an anti-bot system (Cloudflare, DataDome, Imperva, PerimeterX, DDoS-Guard, or a generic block page).

The package is usable both offline (batch-scoring recorded page snapshots) and inside a runtime service (classifying live sessions), so that "block rate" means the same thing in both places. There is no I/O, no browser or DOM globals, and no non-determinism (no Date, no Math.random) anywhere in the classification path — the same Observation always produces the same Classification.

The Observation → Classification contract

classify(observation, ruleset) takes an Observation — a privacy-scrubbed snapshot of a page — and a Ruleset, and returns a Classification.

Observation channels

All fields are optional; supply whatever you were able to capture.

| Field | Type | Channel | Notes | | ---------- | ------------------------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | host | string | — | Host only, no query string. Not itself matched by a channel. | | status | number | status | Main-document HTTP status code. | | headers | Record<string, string> | header | Response headers; keys are lowercased before matching. | | cookies | string[] | cookie | Set-cookie names only — values are never included. | | finalUrl | string | document-url | Query string already stripped by the caller. | | title | string | title | page.title(). | | text | string | text | Top-frame rendered innerText. | | markup | string | dom / markup | Top-frame served outerHTML. | | frames | FrameObservation[] | { url?, text?, markup? } | Same-origin child iframes (no title — FrameObservation only carries url/text/markup), so interstitials rendered inside an iframe (e.g. Imperva's iframe#main-iframe) are still reachable. |

Callers are responsible for the privacy scrub before constructing an Observation: strip query strings, drop cookie values (names only), and truncate markup/frame markup to whatever the caller is comfortable retaining.

Classification fields

| Field | Type | Notes | | -------------- | ----------------- | ------------------------------------------------------------------------------------- | | status | BlockStatus | Tri-state: 'blocked' \| 'clear' \| 'inconclusive'. | | blocked | boolean | Convenience for status === 'blocked'. | | inconclusive | boolean | Convenience for status === 'inconclusive'. | | vendor | string? | The winning vendor (e.g. 'cloudflare'), only set when blocked. | | kind | Kind? | 'challenge' \| 'block' \| 'captcha' \| 'rate-limit' \| 'network-error'. | | confidence | number | 0..1, the winning vendor's noisy-OR score. | | tier | Channel? | The strongest matching channel for the winner. | | matched | MatchedSignal[] | Every signal that matched, { vendor, channel, marker, weight, note? }. | | reason | string | Human-readable summary, e.g. "Blocked (cloudflare/challenge): header cf-mitigated". |

inconclusive covers cases where the classifier can't tell — a browser-level network error (net::ERR_CONNECTION_RESET-style strings), no content at all was captured, or the page was sampled but came back empty. Treat inconclusive rows as "exclude from scoring", not as clear.

Usage

classify

import { classify, DEFAULT_RULESET } from '@browserless.io/anti-bot-classifier';

const result = classify(
  {
    host: 'example.com',
    status: 403,
    headers: { 'cf-mitigated': 'challenge' },
  },
  DEFAULT_RULESET,
);

// result.status === 'blocked'
// result.vendor === 'cloudflare'
// result.kind === 'challenge'

detectBlock (DOM-only convenience shim)

For call sites that only have title / text / markup (no headers, cookies, status, or final URL) — this is a drop-in replacement for the legacy detectBlock, classifying against DEFAULT_RULESET under the hood:

import { detectBlock } from '@browserless.io/anti-bot-classifier';

const result = detectBlock(
  document.title,
  document.body.innerText,
  document.documentElement.outerHTML,
);

// result: { status, blocked, inconclusive, reason }

Overriding or extending the ruleset

classify takes the ruleset as an explicit argument — there's no hidden global state. To customize detection, either build a Ruleset from scratch, or spread DEFAULT_RULESET and add or replace signals:

import {
  classify,
  DEFAULT_RULESET,
  type Observation,
  type Ruleset,
} from '@browserless.io/anti-bot-classifier';

const myRuleset: Ruleset = {
  ...DEFAULT_RULESET,
  signals: [
    ...DEFAULT_RULESET.signals,
    {
      vendor: 'acme-waf',
      channel: 'header',
      key: 'x-acme-block',
      marker: 'B',
      kind: 'block',
      weight: 0.9,
      note: 'x-acme-block header present',
    },
  ],
};

const observation: Observation = {
  status: 200,
  headers: { 'x-acme-block': '1' },
};

classify(observation, myRuleset);

A Ruleset is { signals, structural?, networkErrorSignals, threshold, version }. signals are substring/exact-value matches against a single channel; structural entries are JS predicates ((o: Observation) => boolean) for patterns that need to look across multiple fields at once (e.g. "empty top frame plus an Imperva iframe"). Both kinds carry a marker: 'B' (block-only — scores) or 'P' (present-always — context only, reported in matched but never moves the score).

The confidence model

Only 'B'-marked signals contribute to scoring; 'P' signals are fingerprint context that show up in matched but never move confidence. Matched 'B' signals are grouped by vendor and combined per-vendor via noisy-OR — score = 1 − Π(1 − weightᵢ) — so multiple weak signals from the same vendor can still add up to a confident verdict without any single signal needing to be definitive on its own. The vendor with the highest noisy-OR score wins; the page is classified blocked when that winning score is ≥ ruleset.threshold, and clear (or inconclusive, see above) otherwise. The weights on every signal in DEFAULT_RULESET, and its threshold (currently 0.5), are placeholders reproducing the legacy detector's boolean behavior — they get fit against real block/clear outcomes from a labeled corpus in a future release, not in this package's current form.

Privacy

Classification is metadata-only: vendor, kind, status/blocked/ inconclusive, confidence, tier, and reason (plus matched, which itself only carries vendor/channel/marker/weight/note). It never echoes back page content, cookie values, or query strings — those live only in the Observation you pass in, which the classifier reads but never returns any part of.

Development

npm test            # vitest
npm run build       # tsc -> dist/