CloudFlare Bot Directory

A comprehensive list of 500+ verified bots and web crawlers from CloudFlare Radar, available as a JSON dataset for bot detection, user agent analysis, and web scraping identification.

Why

Identifying legitimate bots from malicious scrapers is essential for web security and analytics. This package provides the official CloudFlare Radar verified bots directory, helping you:

• Detect verified bots – Identify legitimate crawlers like Googlebot, Bingbot, and more
• Filter analytics – Exclude known bots from your traffic reports
• Allow-list crawlers – Permit verified bots while blocking suspicious traffic
• User agent lookup – Match user agent strings against known bot patterns

Data Structure

Each bot entry includes:

| Field | Description | | ------------------- | ---------------------------------------------- | | slug | URL-friendly unique identifier | | name | Human-readable bot name | | kind | Bot kind classification | | operator | Company or organization operating the bot | | operatorUrl | URL to operator's documentation | | category | Bot category (Search Engine, Monitoring, etc.) | | description | What the bot does | | followsRobotsTxt | Whether the bot respects robots.txt | | userAgentPatterns | User agent pattern(s) for matching | | userAgents | Known user agent string(s) |

Install

npm install cloudflare-bot-directory

Usage

const bots = require('cloudflare-bot-directory')

// Get all bot slugs
console.log(bots.map(bot => bot.slug))
// ['2checkout', '360monitoring', 'googlebot', 'bingbot', ...]

// Find a specific bot
const googlebot = bots.find(bot => bot.slug === 'googlebot')

// Filter by category
const searchBots = bots.filter(bot => bot.category === 'Search Engine Crawler')

// Check if a user agent is a known bot
const isKnownBot = (userAgent) =>
  bots.some(bot => userAgent.includes(bot.name))

Related

• top-user-agents – An always up-to-date list of the top 100 HTTP user-agents most used over the Internet.

License

cloudflare-bot-directory © Kiko Beats, released under the MIT License. Authored and maintained by Kiko Beats with help from contributors.

kikobeats.com · GitHub Kiko Beats · X @kikobeats