ai-signals-audit

You now set AI-crawler permissions in four or five places — robots.txt Disallow, Cloudflare's Content-Signal, the AIPREF Content-Usage header, X-Robots-Tag, an RSL license, a llms.txt. They drift apart, and when they disagree a crawler obeys whichever one it reads and ignores the rest. So you end up with policies like this and never notice:

$ npx @fernforge/ai-signals-audit nytimes.example

  [C1 error] AI training is granted but the crawlers are Disallowed
    robots.txt Content-Signal set ai-train=y, but robots.txt Disallows
    gptbot (OpenAI), ccbot (Common Crawl) — bots that perform AI training.
    They can never fetch the page, so the grant is unreachable.

  [C4 error] Carriers disagree on AI training
    robots.txt Content-Signal set train=y while X-Robots-Tag set train=n.
    Different crawlers read different carriers, so your intent is ambiguous.

It fetches a domain, reads every AI-control carrier, and prints the contradictions. Exit code is non-zero when it finds one, so you can drop it in CI and fail the build the moment your signals diverge.

npx @fernforge/ai-signals-audit example.com

No install, no config, no account. Node 18+.

When you'd run this

Most sites still have one AI-control signal: robots.txt. The conflicts start the moment you add a second. Run this when you:

• add an RSL license, llms.txt, a Content-Signal policy, or an X-Robots-Tag to a site that already has a robots.txt — before you ship, to confirm the new signal doesn't grant what the old one blocks;
• inherit a site and don't know what its AI rules already say across carriers;
• want a CI gate so the next edit to any one signal can't silently contradict the others.

What it checks

| Signal | Where | What it carries | |---|---|---| | robots.txt Disallow | /robots.txt | which crawlers may fetch at all | | Content-Signal | /robots.txt | Cloudflare's ai-train / ai-input / search policy | | Content-Usage | response header + /robots.txt | IETF AIPREF train-ai / search preferences | | X-Robots-Tag | response header | noai, noimageai, noindex | | RSL license | robots.txt License:, /.well-known/rsl.xml, <link rel=license> | what AI use the license permits or prohibits | | llms.txt | /llms.txt | whether you publish an LLM consumption guide |

The contradictions it catches

• C1 — unreachable grant. A carrier permits AI training (or AI input, or search) but robots.txt Disallows the very crawlers that do it. You said yes; they can't fetch the page to act on it.
• C2 — incomplete block. A directive says ai-train=n and you blocked some training bots, but left others crawlable. Crawlers that only read Disallow walk right in.
• C3 — license you also block. Your RSL license sells AI-training access, but robots.txt blocks the crawlers that would consume it. The license grants something no compliant crawler can reach.
• C4 — carriers disagree. Two signals state opposite values for the same axis (Content-Usage header says train, X-Robots-Tag says noai). Crawlers split on which to honor.
• C5 — llms.txt to no one. You publish /llms.txt to guide LLMs, but your other signals block AI crawlers. The guide reaches nothing.

Errors (C1, C3, C4) fail the build. Warnings (C2, C5) report without failing. --warn-only downgrades everything to advisory.

CLI

ai-signals-audit <domain> [options]

  --json          Machine-readable output for CI logs and dashboards.
  --quiet         Conflicts only; skip the signal summary.
  --warn-only     Exit 0 even with conflicts (report without failing CI).
  --timeout <ms>  Per-request timeout. Default 10000.

Exit codes: 0 clean (or --warn-only), 1 conflicts found, 2 nothing reachable / usage error.

GitHub Action

Catch a drift the day someone edits robots.txt:

- uses: fernforge/[email protected]
  with:
    domain: example.com
    warn-only: false   # set true to report without failing the job

Library

import { audit } from '@fernforge/ai-signals-audit';

const { signals, conflicts } = await audit('example.com');
for (const c of conflicts) console.log(c.id, c.title);

The parsers (parseRobots, parseContentUsage, parseXRobotsTag, parseRSL) and detectConflicts are exported too, if you want to feed them carriers you fetched yourself.

A note on the specs

These standards are at very different stages. robots.txt AI directives and llms.txt are in wide use; Cloudflare's Content-Signal policy and RSL 1.0 shipped in late 2025; the AIPREF Content-Usage header rides an IETF draft that's still moving. The conflict rules are versioned with the tool — when AIPREF's attachment draft lands or changes, the Content-Usage handling updates with it. If you only emit RSL, see the sibling rsl-licensing.

License

MIT

Built and published autonomously by an AI agent.