Valiron Crawler Protection

Install the integration package from npm: npm install @valiron/crawler-protection (Node.js 22+).

Any HTTP backend: deploy the standalone HTTP gateway before your origin, with shared route/rate/origin-concurrency admission. PHP, Python, Ruby, Java, Go and other HTTP backends do not need to install a Node SDK. Use the private work-admission protocol around individual database operations; a standard-library Python client is included. Node/Nuxt in-process adapters remain available. SQLite and MariaDB are tested; other SQL/NoSQL drivers require integration and confirmed-settlement validation. The gateway alone does not automatically instrument queries or connection pools. Gateway deployment requires control of routing and origin access; hosted platforms that prevent those controls need a platform integration.

Integration for Valiron Crawler Protection. Protects origin work in customer infrastructure. No blockchain or crawler signup is required. SDK 0.3.0+ requires a Free/Pro/Enterprise operator API key at startup and refreshes a short-lived account lease in the background.

Customer-facing behavior and integration contracts are documented in the controlled crawler access guide. Install it with npm install @valiron/crawler-protection; it is separate from @valiron/sdk.

Implemented:

• Strict client IP parsing and an authenticated CloudFront origin-header adapter; never trusts the leftmost forwarded value.
• Official IP verification for OpenAI search/user/training crawlers, Anthropic search/user/training crawlers, Google-hosted agents and Gemini Notebook; each class remains subject to configured budgets.
• Explicit crawlerAccessDefault and unverifiedAccess controls for opt-in default-deny access, independent of authenticated shopper lanes. See AI service access.
• Broader directory discovery with reviewed metadata approvals, distinct third-party CIDR evidence, and conservative fallback; see directory setup.
• Bing forward-confirmed reverse DNS with deduplicated, concurrency-limited cold lookups and a bounded cache.
• Atomic Redis token buckets across global, route, IP, provider and crawler-class budgets.
• Shared DB work slots, local concurrency limits, short-lived guard-owned request contexts and per-request operation bounds.
• Node middleware and a structural Nuxt/Nitro server middleware adapter. Unknown callers retain baseline limits.
• Noncacheable 429/503 responses, policy/evidence expiry and conservative dependency-failure handling.

This is an origin admission library, not a complete bot detector. A matching user-agent and source network establish provenance, not human identity or proven crawler intent. Meta classes can use verified AWS WAF evidence over a protected ingress path; no Meta CIDR/ASN verifier is assumed. Header spoofing without a network match does not receive verified allowances.

Returning-browser recognition (optional, observe-only)

The package also includes returning-browser recognition, folded in from the former standalone @valiron/device-identity package so an existing Crawler Protection install gains it without a second SDK. Two signal classes stay strictly separate:

1. Deterministic - a first-party, server-signed visitor cookie (HttpOnly; Secure; SameSite=Lax). A valid cookie is the only exact match.
2. Probabilistic - coarsened device signals (browser family/major, languages, timezone, platform, screen, hardware hints), compared per component with weighted scoring. A strong match produces a similar link, never an exact one.

Recognition is evidence, not a verdict: no risk score, no block decision, no enforcement hook. similar means "this browser resembles one we have seen" - shared devices, corporate fleets and fresh installs all produce false links, so a similar link is never sufficient grounds to deny or limit service.

With SDK 0.5.0 or later, mount the middleware to expose the endpoints, then call the browser collector to begin recognition:

import { activateCrawlerProtection, DeviceIdentity, RedisIdentityStore, createNodeDeviceIdentityMiddleware } from '@valiron/crawler-protection';

const activation = await activateCrawlerProtection({ apiKey: process.env.VALIRON_API_KEY!, tenant: process.env.VALIRON_TENANT! });
const identity = new DeviceIdentity({
  activation,
  salt: process.env.DEVICE_IDENTITY_SALT!,        // per-deployment, 16+ chars, from a secret manager
  cookie: {
    name: '__Host-vid',
    secret: process.env.DEVICE_IDENTITY_COOKIE_SECRET!,
    maxAgeMs: 180 * 24 * 3600 * 1000,
  },
  store: new RedisIdentityStore(redis),           // or omit for the in-memory store
  retentionMs: 30 * 24 * 3600 * 1000,             // enforced bounds: 1h..90d
});

app.use(createNodeDeviceIdentityMiddleware({
  identity,
  onRecognition: (recognition) => analytics.record(recognition),  // server-side only
}));

Load the collector script and call ValironDevice.identify({ endpoint: "/api/valiron/device-identity/identify", respectPrivacySignals: true }) only when your deployment permits collection. Mounting the middleware alone does not run the browser collector. Recognition is not automatically sent to the hosted crawler dashboard.

The middleware serves the browser collector at GET /api/valiron/device-identity/collector.js and recognition at POST /api/valiron/device-identity/identify (override with basePath). The browser only ever receives { state } - visitor ids, matched ids and evidence stay server-side, because echoing them would let a caller mimicking another browser's signals harvest that browser's identity. The collector never reads the cookie and never uses localStorage; pass respectPrivacySignals: true to skip collection when the browser sends DNT/GPC.

Privacy properties enforced in code:

• Raw signal values, raw user agents and IP addresses are never stored. Profiles hold per-component salted SHA-256 hashes only. These hashes are pseudonymous, not anonymous: anyone holding the deployment salt can test candidate values. Protect the salt like a signing key, rotate saltVersion on schedule, and treat stored profiles as personal data for retention and deletion purposes.
• Every stored profile carries a TTL; retention is bounded to 1 hour - 90 days (default 30).
• The cookie carries only a random id and an HMAC - no device data.
• identity.forget(visitorId) deletes a profile and its index entries (data-subject deletion hook).

Every default here - 30-day retention, the 8/11.5 similarity threshold, the 180-day example cookie lifetime, optional privacy-signal handling - is a placeholder so the module runs, not an approved production policy. Before production rollout, the deployment must explicitly decide: consent flow per jurisdiction (a first-party identification cookie is not strictly necessary for most sites), DNT/GPC behavior, retention, similarity-threshold tuning against real traffic, what may consume similar links (recommend observe-only analytics first; the link graph is attacker-influenceable, though links and matched ids never leave the server), and cookie lifetime/rotation. SDK 0.5.0 requires account-bound activation for this module. Each recognize() call consumes one bot-quota admission before identity storage. Missing/revoked/expired activation and unavailable quota checks stop recognition; its middleware returns 429 for quota exhaustion and 503 for unavailable admission. Exclude the recognition endpoint from outer request admission if it would otherwise admit that same HTTP request twice. Previous public releases cannot be retroactively restricted.

Account activation and monthly usage

Sign up at Valiron and create an operator API key in Dashboard → API Keys. Keep it server-side as VALIRON_API_KEY. Use your dashboard-exported operator_<account ID> tenant. activateCrawlerProtection() obtains a five-minute lease and refreshes every minute; pass its handle to the runtime/guard. Missing, invalid or revoked credentials cannot activate the supported SDK. Startup/background refresh use HTTPS. Each protected HTTP request obtains quota admission through the hosted usage endpoint; a database operation within that request is not counted again. Outages never extend a lease. After revocation/expiry, new protected work stops, while confirmed releases and recovery remain available. Call activation.stop() on shutdown. The public SDK is modifiable; these checks are not tamper-proof and do not retroactively restrict version 0.1.0. Deploy the account activation endpoint before upgrading clients.

Install and validate inside this workspace

pnpm --filter @valiron/crawler-protection build
pnpm test:crawlers
pnpm test:crawlers:redis
pnpm --filter @valiron/crawler-protection test:python
pnpm test:crawlers:service

The Redis and service tests start temporary Redis processes and loopback Python/SQLite origins with generated fixture credentials. redis-server and Python 3 must be installed. External verification-feed egress is disabled in service lifecycle fixtures; no customer URLs or production databases are used. The Python unit suite uses local HTTP fixtures only for deliberate authentication/error/redirect cases. The package is distributed through npm as @valiron/crawler-protection.

Website availability: Free, Pro and Enterprise, within each plan’s usage allowance. Dashboard → Crawler Protection offers authenticated configuration validation/export and integration guidance. Optional createCrawlerTelemetry reporting adds hosted rolling origin analytics, shared capacity, instance status and dashboard monitoring alerts. It does not deploy customer infrastructure. Optional 0.7.0+ managed controls support dashboard publishing/rollback within installed ceilings, bounded opt-in decision logs and separately configured email/webhook alerts. See the public integration guide for telemetry setup, retention and measurement limits.

Release CI coverage

| Command/job | What it checks | | --- | --- | | pnpm test:crawlers | Source verification, signed context, policy/configuration, cache and work-control unit tests. | | pnpm test:crawlers:redis | Real shared Redis/HTTP behavior, Python/SQLite work, forged/expired/foreign-tenant capabilities, malformed/oversized private API input, Redis outage, fixed-origin proxy boundaries and concurrency/recovery. | | pnpm --filter @valiron/crawler-protection test:python | Client authentication denial, blocked work, synchronous settlement, failed-response closure, release failure, redirect denial and environment-proxy isolation. | | pnpm test:crawlers:service | The actual service/admin commands: inactive startup, explicit initialize/resume, hard crash/restart hold retention, cross-worker release and SIGTERM draining actual SQLite work. | | Gateway CI on Node 22 and 24 | Runs the complete SDK unit suite (activation, source verification, directory, policies, admission, adapters and telemetry), plus Python, Redis/HTTP and service suites with isolated fixtures and captures failure logs. | | Gateway release-container CI | Builds the shipped image and repeats service lifecycle acceptance in the nonroot Linux container (CRAWLER_TEST_IMAGE selects that mode). | | Nuxt/Node/MariaDB CI | Real framework/database workload, cancellation, cache and recovery scenarios from the isolated full-stack fixture. | | Website CI | Pro/Free account gating, configuration validation/export, pricing/checkout selection and desktop/mobile setup journeys. |

Both gateway matrix results and the Nuxt/MariaDB job feed the aggregate CI required result. Package-content checks include the gateway commands, private protocol, container files and Python client, and reject fixture/env/source leakage. Container-mode lifecycle tests require Linux Docker host networking and a locally built image; leave CRAWLER_TEST_IMAGE unset for local process tests. Full-stack fixtures are maintained in the source repository and are not part of the npm package. Passing fixtures does not establish customer capacity, real crawler/Meta acceptance or production deployment.

Generic runtime and deployment configuration

The recommended managed integration is compileCrawlerConfiguration() / createCrawlerRuntime(). Schema version 1 validates immutable site policy, explicit route/method inventory, staged rollout, verified crawler rules and public cache settings. Unknown or overlapping paths reject; GET includes HEAD. No customer-specific route logic is built into the engine.

const activation = await activateCrawlerProtection({ apiKey: process.env.VALIRON_API_KEY!, tenant: siteConfiguration.tenant });
const runtime = createCrawlerRuntime({ activation,
  configuration: siteConfiguration, redis, clientIp: trustedClientIp,
  customer: validateShoppingGrant,
});
app.use(runtime.nodeMiddleware);
// Nuxt: defineEventHandler(runtime.nuxtMiddleware)
// Handlers: runtime.withRenderingAdmission(context, renderAndFetch)
// Queries: runtime.withDatabaseAdmission(context, () => queryProduct(productId))

These imports are package exports; the configuration/Redis/ingress/customer dependencies are application supplied. Use runtime.guard for page admission and signed internal forwarding. The runtime's store uses an effective policy that binds the whole configuration fingerprint, not just rate numbers. Use runtime.compiled.policy with private management tools. Never initialize admission from worker startup.

runtime.publicResponse() authenticates ingress even on cache hits, resolves inventory, and caches only configured public groups through an admitted loader. Cache hits remain public content, outside origin crawler access policy; deny cached access separately at the edge where required. runtime.report.snapshot() provides bounded process-local request/work/cache aggregates without raw caller data. These are origin observations, not CDN totals, inferred humans or measured SQL cost savings.

Observation shadows candidate rate/access decisions while hard safety/customer rates, SSR/DB caps, fanout, policy/ingress checks and outages remain enforced. Selective enforcement uses explicit groups. Changing routing, cache, rollout or access rules changes the managed policy identity and requires pause/drain/activation; stale configurations reject.

Read-only onboarding uses compileCrawlerConfiguration() for the effective policy/fingerprint and planCrawlerRollout() for measured rollout blockers. createEcommerceConfiguration() prepares reusable ecommerce route inventory from operator-supplied paths and budgets. Validation does not activate a deployment. See the integration guide. The source checkout also includes development fixtures and CLI examples; they are not included in the integration archive.

Construct one guard per process

The following uses illustrative low fixture values. Production limits, context lifetime and evidence TTL must come from the actual application inventory and load measurements.

import Redis from 'ioredis';
import {
  activateCrawlerProtection, BingVerifier, CrawlerGuard, CrawlerRegistry, RedisAdmission,
  refreshCrawlerFeeds, createCloudFrontClientIpResolver,
} from '@valiron/crawler-protection';

const redis = new Redis(process.env.CRAWLER_REDIS_URL!, {
  enableOfflineQueue: false,
  maxRetriesPerRequest: 0,
  retryStrategy: null,
  commandTimeout: 200,
  connectTimeout: 1000,
});
// Await readiness before accepting requests. Reconnection/recovery is an
// operator action: do not automatically resume against an empty failover store.
redis.on('error', reportDependencyError);

const registry = new CrawlerRegistry();
const rate = { capacity: 10, refillPerSecond: 1 };
const activation = await activateCrawlerProtection({ apiKey: process.env.VALIRON_API_KEY!, tenant: process.env.VALIRON_TENANT! });
const guard = new CrawlerGuard({ activation,
  registry,
  bing: new BingVerifier(),
  store: new RedisAdmission(redis, 'customer-tenant-id', 250),
  policy: {
    version: 'production-1', expiresAt: Date.now() + 60 * 60 * 1000,
    globalRate: rate, perIpRate: rate,
    unknownRate: { capacity: 3, refillPerSecond: 0.2 },
    providers: {
      openai: { rate, classes: { 'oai-searchbot': rate } },
      bing: { rate, classes: { bingbot: rate } },
    },
    routes: {
      product: { cost: 1, rate },
      listing: { cost: 3, rate },
    },
    globalWorkLimit: 4, unknownWorkLimit: 1,
    providerWorkLimit: 2, localWorkLimit: 4,
    maxOperationsPerRequest: 4, maxContextAgeMs: 10000,
  },
  onDecision: decision => boundedTelemetryQueue.tryPush(decision),
});

// Background startup/scheduled refresh; never fetch a feed in a page handler.
// Run at an interval shorter than its TTL, report refresh failure and retain
// last-known-good data only until expiry. Each process refreshes independently
// centrally distributed snapshots are not implemented yet.
const feedStatus = await refreshCrawlerFeeds(registry, { ttlMs: 60 * 60 * 1000, concurrency: 3 });
// Monitor 'unavailable' entries; expired evidence stays conservative.
// refreshOpenAiFeed remains available for existing OpenAI-only integrations.

const clientIp = createCloudFrontClientIpResolver({
  originSecret: process.env.CRAWLER_ORIGIN_SECRET!,
  trustedProxyCidrs: configuredCloudFrontOriginCidrs,
  additionalProxyHops: 0,
});

reportDependencyError, boundedTelemetryQueue and configuredCloudFrontOriginCidrs are application-supplied dependencies, not package exports. Keep secrets out of logs. Configure CloudFront to overwrite X-Valiron-Origin-Auth with the shared high-entropy secret, restrict origin ingress and forward user-agent without including arbitrary user-agent strings in the public cache key. Validate every proxy hop for the actual topology.

Use only explicit inventoried routes. Normalize route/query semantics in application code; do not remove filters, currency, locale or pagination that change the representation. Every route's configured cost must match the middleware mapping. Both GET and HEAD need protection when they execute the same expensive code. Do not accidentally exempt writes, alternate API paths or background refreshes: apply existing authorization and workload caps to those independently.

Nuxt 4 middleware

// server/middleware/crawler.ts
import { createNuxtCrawlerMiddleware } from '@valiron/crawler-protection';
import { guard, clientIp } from '../utils/crawlerGuard';

const middleware = createNuxtCrawlerMiddleware({
  guard, clientIp,
  route: request => {
    const path = new URL(request.url!, 'https://internal.invalid').pathname;
    // Replace with your website's actual route inventory.
    if (path.startsWith('/products/')) return { group: 'product', cost: 1 };
    if (path === '/rings') return { group: 'listing', cost: 3 };
    return undefined;
  },
});
export default defineEventHandler(middleware);

The Nuxt adapter ends denied Node responses before returning. It stores an admitted context on event.context.valironCrawler. The structural adapter and a separate Nuxt/Nitro Vue SSR route fixture are tested locally; application-specific response-cache ordering still requires validation. It is server middleware, not Vue navigation middleware.

Next.js (App Router, bundled consumers)

The package is safe to import from code that a bundler processes. The device-identity browser collector is embedded in the built package as a string at publish time - deviceIdentityCollectorScript() performs no filesystem reads - so it works unchanged in Next.js dev (Turbopack) and production builds, with no serverExternalPackages entry, asset copying, or __dirname assumptions. A regression test asserts the embedded script is byte-identical to public/collector.js in this repository, so the shipped asset and the embedded copy cannot drift apart.

Mount the device-identity endpoints from Node-runtime route handlers (the middleware uses Node APIs; do not use the Edge runtime):

// app/api/valiron/device-identity/collector.js/route.ts
import { deviceIdentityCollectorScript } from '@valiron/crawler-protection';

export const runtime = 'nodejs';

export function GET() {
  return new Response(deviceIdentityCollectorScript(), {
    headers: { 'Content-Type': 'application/javascript; charset=utf-8', 'Cache-Control': 'no-store' },
  });
}

The crawler guard itself runs wherever a Node runtime is available; the same bundling note applies - no special configuration is required.

Node API and actual database work

app.use(createNodeCrawlerMiddleware({ guard, clientIp, route: mapProtectedApi }));

app.get('/api/products/:id', async (req, res, next) => {
  try {
    const product = await guard.withDatabaseAdmission(req.valironCrawler!, async () => {
      // The promise must settle only after the query stops and its connection
      // is released. A timeout response alone does not terminate DB work.
      return queryProductWithBoundedPoolAndServerTimeout(req.params.id);
    });
    res.json(product);
  } catch (error) {
    if (error instanceof AdmissionError) return sendDenial(res, error.decision);
    next(error);
  }
});

Import the middleware, error and denial helper from this package. Protect each expensive operation, including parallel fanout. Missing or fabricated contexts cannot acquire work through CrawlerGuard; contexts expire and can start only the configured number of operations. Keep database pools bounded across all workers, jobs and Nuxt processes; this package cannot protect unwrapped database calls.

Managed mode supports signed Nuxt-to-Node propagation. Create InternalAdmission with a server-held keyring (minimum 32-byte secrets), issuer and audience. Nuxt calls guard.forward(context, channel, { method, path, body }); Node calls guard.acceptInternal(token, channel, target) on a dedicated authenticated internal route, then withDatabaseAdmission. Bind the exact method, path/query and raw request bytes; use TLS or a protected private network. Do not run the public adapter over this internal route: it intentionally strips internal headers. Missing/invalid internal tokens must reject before the handler. Public direct API calls still perform normal admission. Tokens never enter browser responses or cache entries.

The page budget is consumed once; each local or downstream DB operation atomically consumes the shared parent fanout budget. A child token authorizes one DB operation, not an entire unrestricted handler. Replayed child tokens cannot execute another operation. Context policy fingerprints, issuer/audience, signature and expiry must match. Managed mode is required for cross-service replay protection.

Call guard.withRenderingAdmission(context, renderAndFetch) around actual SSR. Its independent shared render slots use the same configured ceilings and customer reservations; it does not consume a DB fanout unit. Wait for all fetches/render work to settle before release.

Slot recovery and outages

Work-slot hashes deliberately have no automatic expiry. A disconnected browser, a crashed Node process or a response timeout does not prove a database query stopped. Release follows actual operation settlement. A release error preserves the query result, emits work_release_failed, and leaves a conservative slot held.

Use ManagedRedisAdmission for deployment, with the exact immutable policy on every process. New namespaces require explicit drained initialization and resume; workers never initialize themselves. Every rate/work/child mutation atomically checks the active policy fingerprint and pause flag. inspect() lists held request IDs, owner host/process, age timestamp and keys. pause(), confirmed-drain reconcile(exactIds, evidence), drained CAS activate(previousFingerprint), and resume() provide recovery. No public admin endpoint is added. See deployment and recovery instructions. Legacy RedisAdmission remains for compatibility and has no managed pause/policy gate.

observeDatabase(actualSample, thresholds) opens the shared catalog circuit on measured pool waits, query latency or failures in a bounded server-time window. It does not infer health from HTTP status. catalogCircuit('probe', evidence) permits one catalog DB operation at a time; bypass regeneration caches while probing. An operator returns to healthy after observing fresh successful probes. Checkout keeps its reserved capacity during catalog distress. A global pause still denies every expensive lane.

Redis command timeout/offline errors cause 503 before expensive callbacks. Use no offline queue or automatic command retries. If Redis loses recent state or fails over to an empty store, keep admission stopped until all prior work is drained/reconciled. The package cannot itself establish that a newly reachable Redis contains the authoritative pre-failure state. Retain independent process/pool caps.

The guard sees origin requests only. CloudFront cache hits are neither classified nor charged here; serving a cached public page remains an edge decision. Configure CloudFront error handling to avoid caching denials. Keep personalized responses out of public caches and retain WAF controls.

Customer lanes and cache

GuardPolicy.customers configures shopperReserve, checkoutReserve, shopperRate and checkoutRate. Together reservations must be smaller than globalWorkLimit, and localWorkLimit must cover the shared total. A trusted customer(request, routeGroup) callback must validate a bounded, server-stored shopping/cart/session grant and return { lane, principal }. Caller flags, UA or arbitrary cookies never grant priority. Checkout priority additionally requires routes[group].customerLane = 'checkout'; other uses downgrade to shopper. Authentication alone does not establish a genuine shopper. Unknown anonymous traffic remains conservative; monitor false throttles rather than calling it human.

Use physically separate bounded catalog/checkout DB pools. Catalog work cannot occupy the reserved slots; the sum of every process pool and job pool still must fit the DB safe budget. Reservations are ceilings and reserved headroom, not a latency guarantee.

CatalogCache canonicalizes explicit public GET/HEAD URL/query keys, preserves meaningful variants, bypasses cookies/auth/unknown query parameters and refuses denials, Set-Cookie, Vary or private/no-store responses. The application must ensure the representation depends only on those keys; separate namespaces for hosts/tenants/locale variants as appropriate. A shared non-expiring regeneration hold deduplicates a key, and a global regeneration limit bounds distinct cold misses. Fresh HTML bypasses origin work; stale HTML can refresh only through the supplied admitted loader and explicit background-lifetime hook. A busy cold miss returns uncached 503. No generic route cache is installed automatically.

Validation and remaining boundaries

Run unit/Redis tests, then the isolated stack fixture. It runs Nuxt 4.1.3/Nitro Vue SSR, a separate Node service, MariaDB 10.11.8 and Redis. It tests HTML metadata/image/redirect mechanics, signed child replay/fanout, work rejection, checkout isolation, real SQL timeouts/disconnects/process death, cache misses and managed recovery. Fixture latency is local evidence, not a customer capacity promise.

Real Meta fetch/ad diagnostics, customer-specific AWS label configuration, production policy/telemetry delivery and customer budgets remain external rollout work. The temporary website lab is still the earlier public-document experiment; this stack milestone is not automatically enabled on valiron.co or another site.

Directory data attribution: Verified Bots Directory by Mayank Gupta (https://verifiedbots.dev), CC-BY-4.0. Directory membership does not grant access. Bare IPs become host ranges; invalid/broad range sets disable only the affected entry. Feed-backed approval hashes survive address rotation; static ranges and changed identity metadata require review. Existing approval files need regenerated hashes after this update. Run node gateway/directory.cjs to review candidates; mount selected approval hashes using CRAWLER_DIRECTORY_APPROVALS_FILE, then configure explicit site budgets.

Monthly bot quotas require Crawler SDK 0.3.0 or later. Upgrade existing 0.2.x deployments to use Free accounts and hosted per-request quota admission. Earlier public SDK versions are not retroactively metered.

Free includes 10,000 protected HTTP requests per UTC calendar month; Pro ($149/month) includes 100,000. The bot allowance is independent from agent access and shared across an account's API keys, sites and replicas. Local access/rate denials run before hosted quota admission and do not consume the allowance. Once hosted admission succeeds, the request is counted even if a later work-capacity check fails. Quota exhaustion returns HTTP 429; an unavailable usage check fails closed with 503. This adds a hosted round trip before protected origin work. Pro can enable additional crawler calls at 0.001 each in the subscription billing currency after explicit price acceptance; Free remains capped at 10,000. Unconfigured or inactive usage billing retains the 100,000 Pro cap. Enterprise: email [email protected].

Security compatibility and admission recovery

Directory approval hashes include the resolved CIDR set. Review and approve again when that evidence changes; a generic browser user-agent token cannot override an official crawler claim that fails source verification. Local access/rate denials run before hosted monthly quota admission, so rejected requests do not consume the account allowance.

Uncertain or timed-out permit acquisitions carry a deadline and cancellation tombstone so late execution cannot leave an unstarted reservation. This does not cancel work that already started: retain its hold until settlement is confirmed, or pause admission and reconcile the exact hold with confirmed-drain evidence. An HTTP timeout alone never proves database work has stopped.

Web Bot Auth verification (draft protocol, opt-in)

WebBotAuthVerifier verifies Web Bot Auth signatures (draft-ietf-webbotauth-httpsig-protocol, an active IETF draft) as a third evidence path for crawlers and agents that cannot publish IP ranges. A signer proves control of an Ed25519 key pinned for its Signature-Agent HTTPS URL; a valid signature produces evidence with method: 'web_bot_auth' and the operator-configured provider/class, and the usual provider budgets still apply. Wire it through the guard's fallbackEvidence hook, which is consulted only when trusted/IP/DNS evidence did not verify - an existing verified identity always stands:

const wba = new WebBotAuthVerifier({ agents: {
  'https://signer.example.com/.well-known/http-message-signatures-directory': {
    provider: 'example-ai', crawlerClass: 'examplebot',
    keys: [/* pinned Ed25519 JWKs from the signer's directory */],
    promote: true, // opt-in per signer; default only reports the authenticated signer
  },
} });
new CrawlerGuard({ /* ... */ fallbackEvidence: (request) => wba.fallbackEvidence(request) });

The standalone gateway and createCrawlerRuntime() do not automatically enable this verifier; wire it explicitly when constructing CrawlerGuard.

Trust boundaries, deliberately stricter than the draft minimum:

• Pin-only key sources. The verifier performs no network fetch, so a request cannot steer this process into contacting an attacker-chosen directory URL. Dynamic directory resolution is omitted until it can ship with connect-time destination validation and bounded egress.
• A valid signature is proof of key control, not good intent or authorization. Unpinned signer URLs stay unverified, and pinned-but-not-promoted signers keep unverified budgets while telemetry records the authenticated signerUrl. Promotion is an explicit per-signer opt-in.
• Freshness is not replay protection. The draft's minimum coverage binds only @authority, so a captured signature can be replayed against this origin until expires (bounded to the draft's recommended 24-hour maximum, configurable down). Verified signers remain subject to every budget, which bounds replay damage; profile: 'strict' additionally requires @method and @path coverage. The strict profile is a Valiron-specific strengthening, not a standard compliance claim.
• The protocol is a draft. Verification is signature-only and tolerates non-Web-Bot-Auth signatures on the same request; expect wire-format changes before RFC publication.

Request usage

Dashboard → Usage shows account-wide monthly crawler requests, remaining allowance, additional calls and estimated request-overage charges. It refreshes every 15 seconds while visible. Free includes 10,000; Pro includes 100,000 and separately opted-in additional requests at 0.001 each in the accepted subscription currency. Middleware has a separate allowance and opt-in. Traffic charts come from optional runtime telemetry; billing counts come from hosted admission.

Upgrade to 0.6.0

Install @valiron/crawler-protection@^0.6.0 for official Anthropic and Google agent/Notebook verification and explicit default-deny controls. Existing configurations retain their budget defaults. Review and regenerate directory approvals before deploying; static addresses remain pinned, while reviewed feed-backed recipes can refresh addresses. Migrate any overlapping directory-based Claude crawler and supported Google agent/Notebook rules to the built-in IDs. See the included CHANGELOG and selected AI service guide for the complete configuration and anonymous-visitor boundaries.

Dashboard controls

Managed controls require Crawler Protection SDK/gateway 0.7.0+, the management backend and its database migration. Existing static runtimes and aggregate telemetry do not automatically enable this feature. The dashboard does not install software, deploy infrastructure or increase database capacity.

Connect a gateway

Keep the existing validated installation configuration, Redis and operator API key. Add these server-side variables and restart your gateway:

VALIRON_API_KEY=<your-server-side-operator-key>
CRAWLER_SITE_ID=storefront
CRAWLER_MANAGED_CONTROLS=true

Use the same site ID for controls and telemetry. Continue CRAWLER_TELEMETRY_URL reporting for aggregate analytics and traffic alerts. On first successful control report, the installation registers with its current crawler access and mode, preserving a partial enforceGroups rollout. Selecting a new mode in Crawler controls applies that mode to all configured route groups. Decision logging and outbound alerts start disabled. All instances sharing a site must use the same base configuration. Managed registration accepts at most 64 KiB of compact configuration JSON, 16 sites per account and 64 runtime instances per site in 24 hours.

Connect Node middleware

import { createCrawlerControlClient, createCrawlerRuntime } from '@valiron/crawler-protection';

const controls = createCrawlerControlClient({
  configuration, // your validated deployment configuration
  apiKey: process.env.VALIRON_API_KEY!,
  site: process.env.CRAWLER_SITE_ID!,
});
await controls.refresh();
const stopControls = controls.start();
const runtime = createCrawlerRuntime({
  configuration, redis, activation, clientIp, registry,
  controlClient: controls,
});
// Mount runtime.nodeMiddleware/runtime.nuxtMiddleware before protected work.
// Keep your existing database/render admission wrappers and telemetry.
// On shutdown: stopControls(); telemetry.stop(); activation.stop();

The control client uses its fixed hosted endpoint by default. An optional endpoint must be trusted HTTPS (HTTP is accepted only for loopback tests), end in /operator/crawlers/control/runtime, and contain no credentials, query or fragment. Do not accept this value from website visitors. status() reports the locally held revision and lease availability. refresh() records failures in status; current() throws when no valid policy is available.

Choose access and budgets

Open Crawler Protection → Crawler controls, select a registered site, edit and Publish changes:

• Choose Allow within limits or Block for each installed crawler class and route group. Search, user fetches and training are separate identities. Allowing a class still requires its configured verifier and available budget; it does not grant every browser identifying itself as that service access.
• Set the fallback for other verified crawlers. Unverified traffic has a separate control: it includes ordinary anonymous visitors, so blocking it can block people. Independently authenticated shopper/checkout grants retain their own lanes.
• Reduce global, source, route or per-IP credits within the installed ceilings. Credits follow configured route costs; they are not necessarily one credit per request. Unknown traffic remains no more privileged than reduced provider budgets. Work-pool/concurrency ceilings, customer reservations, source verifiers and routing remain installation settings.
• Observe reports proposed access/rate denials while preserving hard safety and workload limits. Enforce applies access/rate decisions to catalog traffic. Review measured headroom before switching.

Customer publishing requires the owner's authenticated dashboard session. Authorized platform admins can also make customer-scoped changes through the admin workspace, with a required reason and atomic audit record; customer history labels those revisions Valiron support. Runtime operator API keys cannot publish, change alert destinations or read owner logs. Settings and the last 20 revisions are durable and account/site scoped. Concurrent edits use revision checks; refresh and review a conflicting revision rather than overwriting it. Rollback publishes the selected settings as a new increasing revision.

Know whether a revision is active

Runtimes poll every 30 seconds and report their previously applied revision before receiving an update. The dashboard distinguishes the desired revision from acknowledgment by instances with reports within the five-minute lease window. An acknowledgment is a runtime report, not independent deployment attestation. Instances may temporarily use different revisions during propagation; the installed safety ceilings remain the upper bound.

Control fetching runs in the background; it adds no hosted network call to the per-request control decision. It does not remove the existing hosted monthly quota admission call. Transient update failures retain the current policy only to its original expiry, at most five minutes. Authentication/configuration rejection invalidates the local policy immediately. Without a valid managed policy, new catalog admissions fail closed; authenticated customer grants remain independent. Already running work must still settle and release normally.

Changing routes, source candidates, work ceilings or the base configuration requires a planned installation update. Stop the old deployment, validate a replacement in Setup, use Version history → Replace the installed configuration, and deploy that exact file. Adoption resets access to the new installation defaults in observation mode, disables decision logging, clears logs and replaces version history. Old configuration fingerprints are rejected. This can interrupt traffic; use a maintenance window. A runtime API key cannot silently overwrite the site's installed configuration.

Inspect recent decisions

Decision logs is separately enabled in Crawler controls. It retains up to 1,000 decisions per site for 24 hours, with source/class, route group, action, reason, verification state, runtime time, server receipt time and generated event ID. No raw IPs, URLs, user agents, credentials or caller-supplied request identifiers are collected by this log buffer. These coarse dimensions can still be business-sensitive; access is account scoped.

Each process buffers up to 1,000 events, uploads at most 100 per control report, and drops new events when full. Retry event IDs are deduplicated in storage. This is a bounded recent sample, not a complete audit archive. The dashboard shows runtime drop indications; aggregate Traffic & analytics remains the source of reported totals. Filter by source, route or decision and export the matching rows. Disabling logging clears retained events and buffers once runtimes receive the update. Runtime clocks can differ; server receipt time controls retention.

Configure notifications

Alerts can send email to the account email address and/or a public HTTPS webhook, only after the owner saves enabled destinations. Email requires server-side RESEND_API_KEY and optional CRAWLER_ALERT_EMAIL_FROM; the UI shows when email is unavailable. Webhooks must use public HTTPS on port 443 without URL credentials. DNS is resolved and checked again for each delivery, pinned to the checked public address; redirects are not followed.

Available triggers are runtime reporting absent for two minutes and a configured threshold of limited/blocked requests in the current fixed five-minute reporting window. The latter requires aggregate telemetry using the same site ID; it is not computed from sampled decision logs. Checks run about once a minute in bounded worker batches, so notifications can be delayed. A per-destination/type cooldown is configurable from 15 minutes to 24 hours. Delivery attempts and status are visible; failures retry up to three times. Deliveries are retained for seven days. These notifications supplement existing infrastructure alarms and do not change policy or reclaim workload holds.

Webhook bodies contain id, type (crawler.stale or crawler.blocked), site, occurredAt and a fixed message. Delivery is not signed; treat it as a notification and confirm incidents in the authenticated dashboard before taking privileged actions. Receivers should deduplicate by id: failures after delivery can produce retries. Turning destinations off or changing settings cancels queued deliveries on their next check; it cannot retract a delivery already in flight.

Management endpoints

All responses are private and noncacheable. API keys authenticate only POST /operator/crawlers/control/runtime, which registers/updates a site, receives a five-minute policy lease and uploads bounded decisions. A configuration fingerprint mismatch returns 409; excessively frequent reports or cardinality limits return 429.

Owner dashboard sessions authenticate:

• GET /operator/crawlers/control: registered sites, installed ceilings, desired settings and runtime acknowledgments.
• POST /operator/crawlers/control/publish?site=...: { expectedRevision, settings } or { expectedRevision, rollbackRevision }.
• POST /operator/crawlers/control/adopt?site=...: { expectedRevision, configuration } for a planned installation replacement.
• GET /operator/crawlers/control/history?site=...: last 20 revisions.
• GET /operator/crawlers/control/logs?site=...: bounded recent decisions.
• POST /operator/crawlers/control/alerts?site=...: { expectedVersion, settings } to configure destinations/triggers.
• GET /operator/crawlers/control/deliveries?site=...: recent delivery statuses.

These controls do not enable paid request overages or change plan allowances. Database utilization, SQL query counts and money saved are not inferred from blocked requests.