crawler-tricker

Serve a static site only when a User-Agent rule hits.

Default behavior in createUserAgentSiteRouter is hit-only:

• requireSingleMatch: true
• allowDefaultTarget: false

So if there is no match (or multiple matches), nothing is served.

Install

npm install crawler-tricker

Quick start

const http = require("http");
const path = require("path");
const { createUserAgentSiteRouter } = require("crawler-tricker");

const router = createUserAgentSiteRouter({
  useDefaultCrawlerRules: true,
  crawlerRootDir: path.join(__dirname, "sites/bot")
});

http.createServer((req, res) => {
  router.handle(req, res, () => {
    res.statusCode = 404;
    res.end("Not Found");
  });
}).listen(3000);

Default crawler list

You can enable built-in crawler matching with:

• useDefaultCrawlerRules: true
• crawlerRootDir (required for static router)

Built-ins exported:

• DEFAULT_CRAWLER_KEYWORDS
• createCrawlerRegex(keywords)

Framework integration

Express:

app.use(router.express());

Connect-compatible:

app.use(router.connect());

Next custom server:

router.handle(req, res, () => nextHandler(req, res));

Matcher-only (no file serving):

const { createUserAgentMatcher } = require("crawler-tricker");

API

createUserAgentSiteRouter(config)

• rules (optional if useDefaultCrawlerRules is true)
• rule.test: RegExp | string | (userAgent, req) => boolean
• rule.rootDir: required per custom rule
• useDefaultCrawlerRules (default: false)
• crawlerRootDir (required when useDefaultCrawlerRules is true)
• crawlerRuleName (default: "crawler")
• crawlerKeywords (default: DEFAULT_CRAWLER_KEYWORDS)
• headerName (default: "user-agent")
• indexFile (default: "index.html")
• spaFallback (default: true)
• addResponseHeader (default: true)
• addSecurityHeaders (default: true, sets X-Content-Type-Options: nosniff)
• requireSingleMatch (default: true)
• allowDefaultTarget (default: false)
• defaultRootDir (used only if allowDefaultTarget is true)

Methods:

• handle(req, res, next?)
• middleware(req, res, next)
• express()
• connect()
• createHandler(next?)
• match(reqLike)
• selectByUserAgent(userAgent, reqLike?)

createUserAgentMatcher(config)

• rules (optional if useDefaultCrawlerRules is true)
• useDefaultCrawlerRules (default: false)
• crawlerRuleName (default: "crawler")
• crawlerKeywords (default: DEFAULT_CRAWLER_KEYWORDS)
• defaultTarget (optional)
• headerName (default: "user-agent")
• requireSingleMatch (default: false)

Methods:

• match(reqLike)
• selectByUserAgent(userAgent, reqLike?)

Production checks

npm run check
npm test