CrawlPay SDK

Charge AI bots $0.001 USDC per page. Two lines of code.

GPTBot, ClaudeBot, PerplexityBot crawl your site constantly. They read your content, train models on it, and pay you nothing. CrawlPay fixes that - bots that handle HTTP correctly pay via x402, the rest pass through untouched.

Install

npm install @crawlpay/sdk

Quick start

Next.js

import { crawlpay } from "@crawlpay/sdk"
import { NextResponse } from "next/server"

const paywall = crawlpay({ wallet: "0xYourWallet" })

export function middleware(request) {
  return paywall(request) ?? NextResponse.next()
}

Express

import { crawlpayExpress } from "@crawlpay/sdk/express"
import express from "express"

const app = express()
app.use(crawlpayExpress({ wallet: "0xYourWallet" }))

Cloudflare Workers

import { crawlpayCloudflare } from "@crawlpay/sdk/cloudflare"

const paywall = crawlpayCloudflare({ wallet: "0xYourWallet" })

export default {
  async fetch(request, env, ctx) {
    // Returns 402 for unpaid bots; forwards humans and paid bots to origin
    return paywall.fetch(request)
  }
}

For manual control (you handle forwarding yourself):

import { crawlpayCheck } from "@crawlpay/sdk/cloudflare"

const check = crawlpayCheck({ wallet: "0xYourWallet" })

export default {
  async fetch(request) {
    const blocked = check(request)
    if (blocked) return blocked
    return fetch(request)
  }
}

Vault Mode (Story CDR)

Private encrypted datasets - content that doesn't exist in plaintext anywhere. Bots pay, Story Protocol decrypts.

crawlpay({
  wallet: "0xYourWallet",
  vault: process.env.CDR_VAULT_UUID
})

Configuration

| Option | Required | Default | Description | | -------- | -------- | -------------- | ------------------------------------- | | wallet | Yes | - | Your Arc wallet address | | price | No | "0.001" | Default price in USDC | | paths | No | - | Per-path prices via patterns | | network| No | "base" | Payment network (base or polygon) | | vault | No | - | Story CDR vault UUID |

Per-path pricing

crawlpay({
  wallet: "0xYourWallet",
  price: "0.001", // default for all pages
  paths: {
    "/premium/*": "0.01", // 10x for premium content
    "/api/*": "0.005", // 5x for API endpoints
    "/vault/*": "0.001", // CDR vault pages
  },
})

Supported AI bots

| Bot | Provider | | ----------------- | ----------- | | GPTBot | OpenAI | | ChatGPT-User | OpenAI | | ClaudeBot | Anthropic | | anthropic-ai | Anthropic | | GoogleOther | Google | | Google-Extended | Google | | PerplexityBot | Perplexity | | CCBot | Common Crawl| | Bytespider | ByteDance | | FacebookBot | Meta | | Applebot-Extended | Apple |

Links

• Demo: crawl-pay.com
• Dashboard: crawl-pay.com/dashboard
• GitHub: divergenttt/CrawlPay-Vault-SDK
• App: divergenttt/CrawlPay-Vault
• License: MIT