bot-defense

Self-hosted bot & AI-scraper defense for Node.js. Detects, deceives, and hardens against automated traffic. No external AI APIs, no ML dependencies — heuristic scoring only. Runs entirely in-process; no request data leaves your network.

Use it two ways: as drop-in Express middleware in your own app, or as the standalone server / .exe with a live dashboard.

Use as middleware

import express from 'express';
import { tarpit } from 'bot-defense';

const app = express();
app.use(express.json());

const protection = tarpit();      // sensible defaults; override any subset
app.use(protection);              // one conventional Express middleware

app.get('/', (req, res) => {
  res.json({ ok: true, botScore: req.botScore }); // 0-100 on every request
});

// Ed25519 signature required on sensitive routes:
app.post('/api/payout', protection.defense.requireSignature(), (req, res) => res.json({ ok: true }));

// Optional hard-block for the worst offenders, after your routes:
app.use(protection.defense.block);

app.listen(3000);

For the advanced/composable API, createDefense(config, { store }) remains available and returns a handle with protect(app), middleware (the pipeline array), installHoneypots(app), block, requireSignature(), validateLogin(ctx), status(req), reset(), close(), plus the raw components. Pass { store } to swap the in-memory backend for Redis. A full runnable example is in examples/express-app.js:

npm run example      # protected demo app on http://localhost:8791

Standalone server & dashboard

npm install
npm start            # server + live dashboard on http://127.0.0.1:8790
npm run demo:client  # legitimate client: signing, replay rejection, login
npm run demo:bot     # simulated scraper: watch it get scored, baited, banned
npm run demo:evasive # browser-mimicking evader (shows detection's limits)

The standalone server is a thin consumer of the same library (createDefense), so there is no behavioral divergence between the two.

Verify it's working

After wiring the middleware into your app, prove the defenses are live with the built-in self-check. It fires a labelled battery of requests and prints a pass/fail scorecard (exit code 0 only if nothing failed, so it works as a CI or post-deploy health check):

npm run verify                                   # against the standalone server
node src/verify.js --url http://localhost:3000 \
     --honeypot /.env --signed /api/payout \
     --data /api/quote --field price --status /whoami --reset /reset

✓ PASS  Browser traffic passes clean      browser scored 0 (< 55 suspect)
✓ PASS  Automation UA is flagged          python-requests scored 60 (>= 55)
✓ PASS  Honeypot is armed                 /.env served bait (200)
✓ PASS  Signed routes reject unsigned     /api/transfer → 401 signature_required
✓ PASS  Data is poisoned for bots         price: real=129.99 → served 124.01
✓ PASS  Rate limiting engages             61/90 concurrent requests throttled

Checks are behaviour-based, so they work even when your app exposes no internals; pass --status for score-level detail. This is also the fastest way for a prospective user to confirm the product does what it claims.

Architecture

Every request flows through the pipeline in this order:

request
  │
  ├─► honeypots        (installed first: bait is served un-obfuscated,
  │                     hits pin the IP at score 100 and raise an alert)
  ├─► detection        fingerprint + behavior → botScore 0-100 on req
  ├─► rate limiter     adaptive token bucket, exponential-backoff bans, 429
  ├─► tarpit           progressive delays for suspects (score ≥ 55)
  ├─► obfuscation      50-200ms jitter, JSON field shuffle,
  │                     data poisoning for score ≥ 70
  ├─► routes           /api/*  (signing required on sensitive paths)
  └─► hard block       403 for score ≥ 85

Detection (src/detection/)

• fingerprint.js — per-request static signals: automation User-Agents, forged-browser tells (Chromium UA without sec-ch-ua, missing sec-fetch-*), missing baseline headers, header wire-order anomalies (via req.rawHeaders), cloud/datacenter source IPs (embedded CIDR list), and JA3 TLS hashes when a fronting proxy forwards x-ja3.
• behavior.js — per-IP sliding window: inter-request-interval coefficient of variation (metronomic = bot), linear-schedule R² fit (catches jittered loops), request-rate Z-score against the live population baseline, endpoint distribution entropy (scanning), 4xx error ratio (probing).
• scorer.js — configurable weighted sum → 0-100, with a max-signal floor so one near-certain signal can't be averaged away, plus score pinning for honeypot hits and ban violations.

Deception (src/deception/)

• honeypots.js — fake admin/internal endpoints from config. Serves realistic bait (fake .env, API keys, admin creds) generated fresh per boot. Every generated secret is a canary token: if it's ever presented to the real /api/login, the source IP is pinned at 100 and an alert fires.
• obfuscate.js — recursive JSON field-order shuffling on all responses, 50-200ms response jitter, and data poisoning (numeric drift ±3-8%, timestamp skew) for high-score requests only.
• tarpit.js — per-IP escalating delays (250ms × 2^level, capped 15s, ±20% noise) for suspects; levels decay after 30s of quiet so false-positive humans recover fast. Plus dedicated random delay inside auth flows.

Hardening (src/hardening/)

• rateLimiter.js — token bucket whose capacity shrinks linearly with bot score (score 100 ⇒ 15% capacity). Attack mode engages when ≥30% of active IPs are suspects and squeezes everyone toward the floor. Violations earn exponentially growing bans (30s → 24h cap).
• signing.js — Ed25519 request signing for sensitive paths. Canonical message keyId\nts\nnonce\nMETHOD\npath\nsha256(body); ±30s timestamp window; single-use nonces; server keypair auto-rotates hourly with a grace window; server signs its own responses so clients can verify origin.
• distributedAuth.js — login validation as independent validators (credentials/scrypt, canary check, reputation, velocity) run in random order, with random inter-step delays, and all steps always run even after a failure, so total latency reveals nothing. Same-cost scrypt for unknown users kills user-enumeration timing.

Ops

• Config: config/defense.config.json — thresholds, weights, honeypot paths, rate limits, signing windows, webhook URL.
• Storage: in-memory with TTL (src/storage/memoryStore.js); Redis adapter skeleton in redisStore.js for distributed setups (npm install redis and swap it in).
• Monitoring: JSON console logs; optional webhook alerts (honeypot hits, attack mode, repeat violators, canary credential use) with per-kind rate limiting. GET /defense/status shows attack mode, tracked IPs, recent honeypot hits, and your own current score with reasons.

Building a standalone .exe

npm run build:exe

Produces dist/bot-defense.exe with an embedded Node runtime — it runs on a machine with no Node installed. The pipeline (build/build.mjs):

1. esbuild bundles the ESM app + all of Express into one CommonJS file.
2. @yao-pkg/pkg wraps that bundle with a Node 20 runtime.

config/ and public/ are copied next to the exe, not bundled, on purpose: express.static needs real files on disk, and you can edit thresholds, honeypot paths, and the dashboard without rebuilding. The server detects packaged mode and loads both from the exe's own folder.

Ship the whole dist/ folder (bot-defense.exe + config/ + public/), then double-click the exe. Dashboard at http://localhost:8790.

Performance is identical to node src/server.js — same V8 engine, just a self-contained runtime. (Electron was considered and rejected: it would bundle a second Chromium browser (~150 MB) to show a window, when the app already serves its dashboard to your real browser over HTTP. No speed benefit, large size/RAM cost.)

Production notes

• Remove or authenticate /defense/* — the status endpoint tells a visitor their own score, which is a debugging gift and an attacker oracle. Gate key registration behind a real enrollment flow.
• trust proxy: set server.trustProxy correctly behind a load balancer or every client shares one IP (and one bucket).
• JA3: Node can't see the TLS ClientHello. Terminate TLS at nginx/HAProxy/Cloudflare with a JA3 module and forward x-ja3 — the fingerprint scorer picks it up automatically. Strip x-ja3 from client-supplied headers at the proxy.
• Cloud CIDR list is a representative sample; refresh from the providers' published ranges (AWS ip-ranges.json, GCP cloud.json, Azure Service Tags) on a schedule.
• Data poisoning is served only above poisonThreshold — tune it high enough that no human ever sees poisoned data, and never enable it on endpoints where a false positive could cause real-world harm (pricing you contractually owe accuracy on, health data, etc.).
• Memory: all trackers are TTL-swept, but they are per-process. Behind multiple instances, sticky-hash by IP at the balancer or move state to Redis.