contentsignals

Content-Signal headers and markdown content negotiation for AI agents. Like helmet for security headers, but for AI agent compatibility.

npm install contentsignals

Quick Start

import express from 'express';
import { contentsignals } from 'contentsignals';

const app = express();

app.use(contentsignals({
  signals: { search: true, aiInput: true, aiTrain: false },
  staticDir: './public',
}));

app.use(express.static('./public'));
app.listen(3000);

Every response now includes:

Content-Signal: search=yes, ai-input=yes, ai-train=no
Vary: Accept

When an AI agent sends Accept: text/markdown, it gets the .html.md companion file instead of HTML.

What It Does

One app.use() call. Three behaviors:

1. Content-Signal Header (every response)

Adds Content-Signal and Vary: Accept to every HTTP response, telling AI agents what they may do with your content. Based on the Content Signals spec.

2. Content Negotiation (when agent asks for markdown)

When a request includes Accept: text/markdown:

• Looks for a .html.md companion file in staticDir
• If found, serves it as text/markdown; charset=utf-8
• If not found and convert: true, converts the HTML response on-the-fly via Turndown
• Otherwise falls through to the normal response

3. Token Count Header (on markdown responses)

When serving markdown, adds x-markdown-tokens: <count> — read from YAML frontmatter or estimated at ~4 characters per token.

API

import { contentsignals } from 'contentsignals';

app.use(contentsignals({
  // Content-Signal header values (required)
  signals: {
    search: true,       // allow search indexing
    aiInput: true,      // allow RAG / grounding / generative answers
    aiTrain: false,     // disallow model training
  },

  // Per-path overrides — keys are glob patterns (optional)
  overrides: {
    '/api/**':  { search: false, aiInput: false, aiTrain: false },
    '/blog/**': { aiTrain: false },
  },

  // Directory with pre-built .html.md companions (optional)
  staticDir: './public',

  // Convert HTML on-the-fly when no .md companion exists (optional, default: false)
  convert: false,

  // Turndown options when convert is true (optional)
  convertOptions: {
    strip: ['nav', 'footer', 'header', 'aside', 'script', 'style'],
    prefer: ['article', 'main'],
  },
}));

Options

| Option | Type | Required | Default | Description | |--------|------|----------|---------|-------------| | signals | SignalConfig | Yes | — | Default Content-Signal values for all responses | | overrides | Record<string, Partial<SignalConfig>> | No | undefined | Per-path glob overrides | | staticDir | string | No | undefined | Directory containing .html.md companion files | | convert | boolean | No | false | Convert HTML to markdown on-the-fly when no companion exists | | convertOptions | ConvertOptions | No | undefined | Turndown options for on-the-fly conversion |

SignalConfig

| Field | Type | Description | |-------|------|-------------| | search | boolean | Allow building a search index and providing search results | | aiInput | boolean | Allow real-time retrieval for AI models (RAG, grounding, generative answers) | | aiTrain | boolean | Allow training or fine-tuning AI model weights |

ConvertOptions

| Field | Type | Description | |-------|------|-------------| | strip | string[] | HTML elements/selectors to strip (e.g. 'nav', '.ad') | | prefer | string[] | Only convert content within these selectors (e.g. 'article', 'main') |

Response Headers

| Header | Value | When | |--------|-------|------| | Content-Signal | search=yes, ai-input=yes, ai-train=no | Every response | | Vary | Accept (appended to existing) | Every response | | Content-Type | text/markdown; charset=utf-8 | When serving markdown | | x-markdown-tokens | <integer> | When serving markdown |

Companion Files

Place .html.md files alongside your HTML pages. ContentSignals resolves them by convention:

| Request Path | Companion Lookup Order | |---|---| | /about | about.html.md → about.md → about/index.html.md | | /about.html | about.html.md | | / | index.html.md | | /blog/post-1 | blog/post-1.html.md → blog/post-1.md → blog/post-1/index.html.md |

Companion files can include YAML frontmatter with a tokens field:

---
tokens: 312
---
# About Us

We build things.

If tokens is missing, contentsignals estimates it (~4 characters per token).

Framework Integration

Express

import express from 'express';
import { contentsignals } from 'contentsignals';

const app = express();
app.use(contentsignals({ signals: { search: true, aiInput: true, aiTrain: false } }));

Fastify (via @fastify/middie)

import Fastify from 'fastify';
import middie from '@fastify/middie';
import { contentsignals } from 'contentsignals';

const app = Fastify();
await app.register(middie);
app.use(contentsignals({ signals: { search: true, aiInput: true, aiTrain: false } }));

Hono (using buildSignal directly)

import { Hono } from 'hono';
import { buildSignal } from 'contentsignals';

const app = new Hono();
app.use('*', async (c, next) => {
  c.header('Content-Signal', buildSignal(
    { search: true, aiInput: true, aiTrain: false },
    undefined,
    c.req.path,
  ));
  c.header('Vary', 'Accept');
  await next();
});

Plain Node.js

import http from 'node:http';
import { contentsignals } from 'contentsignals';

const middleware = contentsignals({
  signals: { search: true, aiInput: true, aiTrain: false },
  staticDir: './public',
});

http.createServer((req, res) => {
  middleware(req, res, () => {
    res.writeHead(200, { 'Content-Type': 'text/html' });
    res.end('<h1>Hello</h1>');
  });
}).listen(3000);

Samples

Runnable samples are in samples/. Each has its own README with run and test instructions.

| Sample | Framework | Port | Features Shown | |--------|-----------|------|----------------| | samples/express | Express | 3000 | All three modes: headers, companions, on-the-fly | | samples/node-http | Plain Node.js | 3001 | Headers + companion serving, no framework | | samples/hono | Hono | 3002 | Web Standard adapter pattern | | samples/fastify | Fastify | 3003 | @fastify/middie integration |

# Run any sample
npx tsx samples/express/server.ts

# Test it
curl -i http://localhost:3000/
curl -i -H "Accept: text/markdown" http://localhost:3000/about

On-the-fly Conversion

When convert: true and no .html.md companion exists, contentsignals converts the HTML response to markdown using Turndown. Install the optional dependencies:

npm install turndown turndown-plugin-gfm

These are listed as optionalDependencies — contentsignals works without them if you only use companion files.

FAQ

Q: What is a Content-Signal header?

A: Content-Signal is an HTTP response header defined by the Content Signals specification. It tells AI agents and crawlers what they may do with your content — whether they can use it for search indexing (search), real-time AI retrieval like RAG (ai-input), or model training (ai-train). Think of it as robots.txt but per-response and more granular.

Q: What problem does contentsignals solve?

A: AI agents are crawling the web, but there's no standard way to tell them "you can use this page for search answers but not for training your model." Content-Signal headers fill that gap. ContentSignals makes it trivial to add these headers to any Node.js server with one app.use() call.

Q: How is this different from robots.txt?

A: robots.txt controls whether a crawler can access a URL at all. Content-Signal headers are more granular — they're set per-response and distinguish between search, AI input (RAG/grounding), and AI training. A page might allow search and RAG but block training. You can't express that in robots.txt.

Q: What is content negotiation and why does it matter for AI agents?

A: Content negotiation is an HTTP mechanism where the client says what format it prefers (via the Accept header) and the server responds accordingly. AI agents prefer markdown over HTML because it's cleaner, smaller, and doesn't contain layout noise. When an agent sends Accept: text/markdown, contentsignals serves the markdown version instead of HTML.

Q: What is a .html.md companion file?

A: It's a markdown version of an HTML page, placed alongside the original. For about.html, the companion is about.html.md. You write it once (or generate it at build time), and contentsignals serves it to agents that request markdown. The companion can include YAML frontmatter with a tokens field for accurate token counting.

Q: Do I need to create .html.md files for every page?

A: No. You have three options:

1. Companion files only — create .html.md files for important pages (set staticDir)
2. On-the-fly conversion — set convert: true and contentsignals converts HTML to markdown automatically using Turndown
3. Headers only — skip staticDir and convert, and contentsignals only adds Content-Signal headers (no markdown serving)

Q: What is the x-markdown-tokens header?

A: It tells AI agents how many tokens the markdown response contains, so they can budget their context window before downloading. The count comes from the tokens field in YAML frontmatter, or is estimated at ~4 characters per token if not specified.

Q: How do per-path overrides work?

A: Override keys are glob patterns matched via picomatch. The first matching pattern wins. Example:

overrides: {
  '/api/**': { search: false, aiInput: false, aiTrain: false },  // block everything for API
  '/blog/**': { aiTrain: false },  // allow search + RAG, block training for blog
}

If no pattern matches, the default signals config applies.

Q: Does contentsignals work with Next.js, Nuxt, or other meta-frameworks?

A: ContentSignals is framework-agnostic — it uses the standard Node.js (req, res, next) signature. For Next.js, use it in a custom server or API route middleware. For Nuxt, use it as server middleware. For any framework that supports Express-style middleware, it's a single app.use() call.

Q: What happens if turndown is not installed and convert is true?

A: ContentSignals throws a clear error: "contentsignals: on-the-fly conversion requires turndown and turndown-plugin-gfm." Install them with npm install turndown turndown-plugin-gfm. If you only use companion files, these packages are not needed.

Q: How does on-the-fly conversion work internally?

A: ContentSignals intercepts the response by monkey-patching res.write() and res.end(). It buffers the HTML output, checks that the response is text/html, then converts it to markdown via Turndown. The strip option removes unwanted elements (nav, footer, ads) and prefer extracts content from specific containers (article, main) before conversion.

Q: What is the performance impact?

A: For companion files: near-zero overhead — a synchronous file existence check and readFileSync. For on-the-fly conversion: Turndown adds processing time proportional to HTML size. Cache the output at the application or CDN layer for production use. For headers only: negligible — two setHeader calls per request.

Q: How do I generate .html.md companion files at build time?

A: ContentSignals is a runtime middleware, not a build tool. For generating companion files, use llmoptimizer or write a simple script:

for f in dist/*.html; do
  npx turndown "$f" > "${f}.md"
done

Q: Is this related to Cloudflare's "Markdown for Agents" feature?

A: Yes — both implement content negotiation for AI agents. Cloudflare's feature does it at the edge for Pro+ customers. ContentSignals does it at the application level for any Node.js server. If you're already on Cloudflare Pro+, you may not need contentsignals for markdown serving, but you'd still use it for Content-Signal headers if Cloudflare doesn't add them automatically.

Q: What's the bundle size?

A: ~7 KB (ESM) / ~7.3 KB (CJS) with two runtime dependencies (picomatch, gray-matter). Turndown is only loaded if convert: true.

Q: Can I use contentsignals with TypeScript?

A: Yes. ContentSignals is written in TypeScript and ships with full type declarations. All config options have JSDoc annotations.

Q: What Node.js versions are supported?

A: Node.js 18 and above.

Exported Utilities

Beyond the main contentsignals() middleware, these lower-level functions are exported for custom integrations:

| Export | Description | |--------|-------------| | contentsignals(options) | Middleware factory — the primary API | | buildSignal(defaults, overrides, path) | Build a Content-Signal header string | | wantsMarkdown(req) | Check if the request accepts text/markdown | | resolveCompanion(staticDir, path) | Find the .html.md companion for a path | | extractTokenCount(content) | Get token count from frontmatter or estimate | | estimateTokens(text) | Rough token estimate (~4 chars per token) |

License

MIT

Author

Sumit Agrawal ([email protected])