CloudFlare Bot Directory

A list of 500+ verified bots and web crawlers sourced from CloudFlare Radar, available as a JSON dataset for bot detection, user agent analysis, and web scraping identification.

[!NOTE] Not affiliated with or endorsed by Cloudflare, Inc. The dataset is licensed CC BY-NC 4.0 and limited to non-commercial use — see License.

Why

Identifying legitimate bots from malicious scrapers is essential for web security and analytics. This package repackages the CloudFlare Radar verified bots directory as data, helping you:

• Detect verified bots – Identify legitimate crawlers like Googlebot, Bingbot, and more
• Filter analytics – Exclude known bots from your traffic reports
• Allow-list crawlers – Permit verified bots while blocking suspicious traffic
• User agent lookup – Match user agent strings against known bot patterns

Data Structure

Each bot entry includes:

| Field | Description | | ------------------- | ---------------------------------------------- | | slug | URL-friendly unique identifier | | name | Human-readable bot name | | kind | Bot kind classification | | operator | Company or organization operating the bot | | operatorUrl | URL to operator's documentation | | category | Bot category (Search Engine, Monitoring, etc.) | | description | What the bot does | | followsRobotsTxt | Whether the bot respects robots.txt | | userAgentPatterns | User agent pattern(s) for matching | | userAgents | Known user agent string(s) |

Install

npm install cloudflare-bot-directory

Usage

const bots = require('cloudflare-bot-directory')

// Get all bot slugs
console.log(bots.map(bot => bot.slug))
// ['2checkout', '360monitoring', 'googlebot', 'bingbot', ...]

// Find a specific bot
const googlebot = bots.find(bot => bot.slug === 'googlebot')

// Filter by category
const searchBots = bots.filter(bot => bot.category === 'Search Engine Crawler')

// Check if a user agent is a known bot
const isKnownBot = (userAgent) =>
  bots.some(bot => userAgent.includes(bot.name))

Related

• top-user-agents – An always up-to-date list of the top 100 HTTP user-agents most used over the Internet.

License

This project has two licenses, one for the code and one for the data. See NOTICE for the full statement.

Code (everything except src/index.json) © Microlink, released under the MIT License.

Data (src/index.json) is sourced from Cloudflare Radar Verified Bots by Cloudflare, Inc. and licensed under CC BY-NC 4.0. The data has been reformatted — fetched per bot, serialised into a single JSON array, with userAgents and userAgentPatterns trimmed of surrounding whitespace and empty entries removed. Values are otherwise unmodified. Use of the dataset is restricted to non-commercial purposes. For commercial use, contact [email protected].

Cloudflare's trademarks are not licensed under CC BY-NC 4.0. This project is not affiliated with, sponsored by, or endorsed by Cloudflare, Inc.

Authored and maintained by Kiko Beats with help from contributors.

microlink.io · GitHub microlinkhq · X @microlinkhq