@obscrd/robots

AI crawler blocking — generate robots.txt and serve it via middleware for Express, Fastify, or Node.js.

Install

npm install @obscrd/robots

Quick Start

import { generateRobotsTxt } from '@obscrd/robots'

const robotsTxt = generateRobotsTxt()
// Blocks all 20 known AI crawlers while allowing Googlebot and Bingbot

API

generateRobotsTxt(options?)

Generate a robots.txt string that blocks AI crawlers.

const txt = generateRobotsTxt({
  blockAll: true,        // default — block all known AI crawlers
  allow: ['ClaudeBot'],  // allow specific crawlers
  sitemap: 'https://example.com/sitemap.xml',
})

| Option | Type | Default | Description | |--------|------|---------|-------------| | blockAll | boolean | true | Block all known AI crawlers | | block | string[] | — | Specific crawlers to block (overrides blockAll) | | allow | string[] | — | Specific crawlers to allow | | customRules | string | — | Additional rules to append | | sitemap | string | — | Sitemap URL |

createMiddleware(options?)

Create middleware that serves robots.txt at /robots.txt. Works with Express, Fastify, and Node's http module.

Express:

import express from 'express'
import { createMiddleware } from '@obscrd/robots'

const app = express()
app.use(createMiddleware())

Node.js http:

import { createServer } from 'node:http'
import { createMiddleware } from '@obscrd/robots'

const middleware = createMiddleware({ sitemap: 'https://example.com/sitemap.xml' })

createServer((req, res) => {
  middleware(req, res, () => {
    res.writeHead(200)
    res.end('Hello')
  })
}).listen(3000)

AI_CRAWLERS

Exported list of 20 known AI crawler definitions.

import { AI_CRAWLERS } from '@obscrd/robots'

console.log(AI_CRAWLERS.length) // 20
// Each: { name, userAgent, operator, purpose }
// purpose: 'training' | 'search' | 'assistant'

Includes GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, Meta-ExternalAgent, and more.

License

MIT