@rebilder/gateway

The content-negotiation middleware SDK — the front door of the Rebilder compiler. Merchants drop it in front of their existing stack; it classifies every request (AI agent / human / search crawler / protocol client) and serves agents a clean markdown transformation of the canonical page, rendered from the merchant's source of truth. Humans and crawlers pass through untouched.

This is the package ARCHITECTURE.md calls "the only public npm artifact initially", and it is published: @rebilder/gateway 0.1.0, Apache-2.0, installable from a bare directory with no monorepo checkout. The API below is a shipped contract, not a plan — treat additions as additive and breaking changes as a major.

request ──► classifyRequest ──► 'markdown'  ──► render from sources ──► text/markdown Response
                            ├─► 'protocol'  ──► config.protocols hook (UCP/ACP/MCP) ──► Response, or pass through
                            └─► 'html'      ──► pass through (humans AND crawlers)
                              every request ──► RebilderEventV0 via onEvent

Dependency policy: zero external runtime deps

Merchants install this in production stacks, so we are ruthless (CLAUDE.md Working Style). Runtime dependencies are exactly the workspace packages:

• @rebilder/agent-detect — pure-compute request classification
• @rebilder/render-md — pure, deterministic source-of-truth → markdown rendering
• @rebilder/events — the observation-layer types (no runtime code)

Nothing else. No framework imports anywhere — the core is written against web-standard Request/Response, the Next.js adapter imports nothing from next (Next middleware and route handlers accept standard Request/Response), the Node adapter imports nothing from express, fastify, or even node:http (structural types describe what every Node framework hands its middleware), and the edge adapter is a plain fetch-handler factory. Because the three workspace packages are internal, the gateway re-exports their public types (ProductSource, PolicySource, CatalogItemSource, DocumentSource, CollectionSource, Fact, DetectionResult, RebilderEventV0, …) so a merchant needs only @rebilder/gateway installed.

Latency budget (the hot path)

Edge code budget: p95 < 50ms compute, no network calls on the hot path (CLAUDE.md Conventions).

• classifyRequest is pure compute — a handful of header string scans, no I/O, well under 1ms.
• handleRequest does no network calls and no dynamic imports at request time; rendering is pure string assembly. render_ms is measured with performance.now() and stamped on every event.
• Your source resolvers are the only variable: back them with your in-memory catalog or a fast cache, not an origin round-trip.

Classification order

Delegated to @rebilder/agent-detect (ARCHITECTURE.md § Request classification — cheapest first): Accept: text/markdown → Web Bot Auth headers (Signature-Agent; parsed here — cryptographic verification is the separate config.verification step, § Verification) → protocol route (/.well-known/ucp, /mcp, /acp) → UA heuristics (fallback only) → default human. The gateway maps the detection onto a serving path:

| Detection | Path | Behavior | |---|---|---| | Agent with Accept: text/markdown or an identified platform | markdown | Render from sources; Response returned | | Agent (signed/identified) on a protocol route, no markdown Accept | protocol | Routed to the protocols hook — see the routing note in § Protocols | | Agent, unidentified, no markdown Accept (e.g. bare Signature pair) | html | null — never guess a format nobody asked for | | Human | html | null — your HTML pipeline runs as if we weren't there | | Crawler (Googlebot, bingbot) | html | null, always — see the guardrail below | | Protocol route | protocol | config.protocols hook when wired (see § Protocols); null otherwise — event records the demand either way |

Sources: what a page can be

Five source types, and any page on any site is one of them. Three are commerce-shaped; two are universal, and exist because most of the web is not a store — a dental practice, a law firm, a council, a magazine all have pages worth serving to an agent and no product catalog to serve them from.

| Source | Returns | Rendered as | |---|---|---| | product | ProductSource | PDP: brand, price, availability, shipping, returns, variant table | | policies | PolicySource[] | Policy documents, bodies verbatim | | catalog | CatalogItemSource[] | Commerce listing table (title, price, availability) | | document | DocumentSource | Any other page: front-loaded facts, hours, contact, actions, then prose | | collection | CollectionSource | A listing of documents: link list, or a table when items carry facts |

All five are optional and independent. A dentist wires document + collection and nothing else; a shop wires the three commerce sources and nothing else; a shop with a blog and a stockist list wires all five.

// A non-commerce site's whole wiring. Same GatewayConfig, same adapters.
import type { GatewayConfig } from '@rebilder/gateway'

export const config: GatewayConfig = {
  storeId: 'store_123',
  sources: {
    document: (url) => getService(url.pathname),      // null when not a service page
    collection: (url) => getServiceIndex(url.pathname),
  },
}

A DocumentSource is a title, a URL, an ordered list of typed Facts (text, list, number, boolean, money, date, url, opening hours), optional contact details and actions, and optional prose. Facts render first, so an agent reading top-down has the whole machine-readable payload before any paragraph — and prose is the only thing a byte budget can take away. The field-by-field contract, the fixed label maps, and the access rule (a metered/subscriber document never emits its prose) live in packages/render-md/README.md.

Resolution order, and the worst case

Fixed order: product → policies → catalog → document → collection. The first source returning data wins, so product wins when several would match a URL. The two universal sources were appended below the three commerce ones, which is what makes them free to adopt: a store that adds site-wide documents keeps its PDPs rendering as PDPs, unchanged. Overlap is narrowed by narrowing the higher-precedence resolver — the same rule as product-over-policies. There is no new knob for it.

Worst case: five sequential resolver calls per markdown request (a URL that nothing matches, with all five wired). Not five per request in general — the chain stops at the first match, so a PDP costs one call — but five is the number to budget against, and it is why sources were added as two general types rather than one per vertical.

With sources.match: exactly one call. An optional pure, synchronous router from URL to source kind:

sources: {
  document:   (url) => getDocument(url.pathname),
  collection: (url) => getCollection(url.pathname),
  product:    (url) => getProduct(url.pathname),
  // One string comparison on the hot path replaces up to five awaits.
  match: (url) => {
    if (url.pathname.startsWith('/products/')) return 'product'
    if (url.pathname.startsWith('/guides/')) return 'document'
    if (url.pathname === '/guides') return 'collection'
    return null // fall through to the ordered chain
  },
}

When match names a kind, that resolver is the only one called — including when it returns no data, which passes through to HTML rather than resuming the chain. null means "use the normal order"; a router that throws or returns an unrecognised kind is treated as null. Keep it to path comparisons: it runs on the hot path, and it has to be cheaper than the awaits it saves.

Failure containment: a broken source cannot break your page

Every one of these is a no-match — resolution continues to the next source, and the request falls through to your HTML if nothing else answers:

• a resolver that throws or returns a rejecting promise;
• a resolver that hangs past sourceTimeoutMs (default 250ms), cut off with Promise.race;
• an empty policies/catalog array, or a collection whose items is empty or missing;
• a value of the wrong shape — a policies resolver that returns a bare object, a collection with no items array. These are array-guarded rather than trusted, because the alternative is a TypeError thrown inside the gateway on a live merchant site.

A render failure (e.g. @rebilder/render-md refusing a malformed money amount) passes the request through to HTML and does not fall down the order — serving the returns policy on a product URL would be worse than serving HTML.

The Gateway Rule (this package will never check a plan)

@rebilder/gateway and @rebilder/render-md contain no concept of a plan, licence, tier, or quota, and never will. Not a flag, not a lookup, not a "just for enterprise" branch. Everything on this page works the same for every merchant on every plan forever, and nothing here phones home.

This is a structural rule, not a pricing promise that could be revised (CLAUDE.md Hard Rule 8): a licence check on the request path is a network call, a failure mode, and a reason for a merchant's page to break — on the one piece of our code that runs in front of their store. Cost control belongs at ingest, where a slow decision is harmless. If you are reading this while adding an entitlement check to this package: don't. Add it to ingest instead.

The cloaking guardrail (Hard Rule 3)

Markdown responses are format transformations of the same substance as the canonical HTML page — never different prices, claims, or availability (that is enforced by @rebilder/render-md's injection-only contract). Googlebot always receives canonical HTML: a known crawler classifies as crawler even if the request sends Accept: text/markdown, so the markdown path is unreachable for crawlers by construction. UA sniffing alone never triggers substantive differences — it only ever selects a format transformation.

API

Public API from the root export only (no deep imports; adapters are the ./next, ./node, ./edge, and ./shopify subpath exports).

import {
  classifyRequest,   // (req: Request) => GatewayDecision — pure, <1ms, no I/O
  handleRequest,     // (req: Request, config: GatewayConfig) => Promise<Response | null>
  generateLlmsTxt,   // (config, options) => Promise<string> — deterministic llms.txt
  verifyWebBotAuth,  // pure Ed25519 crypto over injected keys — no network (§ Verification)
  acceptsMarkdownHeader, // (accept: string | null) => boolean — the classifier's own Accept
                         // predicate. A WILDCARD RANGE RETURNS FALSE: only an explicit
                         // text/markdown range with q > 0 counts. Exported so anything
                         // reporting on negotiation decides it with the same code that
                         // served the request, instead of a lookalike free to disagree.
  KNOWN_AGENT_DIRECTORY,        // ships EMPTY on purpose — see § Verification
  AGENT_VERIFIED_HEADER,        // 'x-rebilder-agent-verified'
  AGENT_VERIFIED_REASON_HEADER, // 'x-rebilder-agent-verified-reason'
  type GatewayConfig,
  type GatewaySources,
  type GatewayDecision,
  type GatewayPath,          // 'markdown' | 'protocol' | 'html'
  type GatewayVerification,  // { keys: AgentKeyRegistry; require?: 'protocol' }
  type SourceResolver,
  type SourceKind,           // 'product' | 'policies' | 'catalog' | 'document' | 'collection'
  type LlmsTxtOptions, type LlmsTxtSection, type LlmsTxtLink,
  // Re-exported dependency types:
  type ProductSource, type PolicySource, type CatalogItemSource,
  type DocumentSource, type CollectionSource, type Fact, type FactValue, type HoursSpec,
  type DetectionResult, type RebilderEventV0,
  type AgentKeyRegistry, type VerificationResult, type VerificationFailureReason,
} from '@rebilder/gateway'

GatewayConfig

interface GatewayConfig {
  storeId: string                     // stamped on every event
  sources: GatewaySources             // your wiring to the source of truth
  onEvent?: (event: RebilderEventV0) => void | Promise<void>
  maxBytes?: number                   // markdown size budget; default 5120
  maxBytesBySource?: Partial<Record<SourceKind, number>>  // per-source override of maxBytes
  sourceTimeoutMs?: number            // per-resolver cutoff; default 250, <= 0 disables
  protocols?: (req: Request) => Promise<Response | null>  // UCP/ACP/MCP hook — see § Protocols
  verification?: { keys: AgentKeyRegistry; require?: 'protocol' }  // Web Bot Auth — see § Verification
}

interface GatewaySources {            // all optional; missing source => pass through
  product?:    (url: URL) => ProductSource | null | Promise<ProductSource | null>
  policies?:   (url: URL) => PolicySource[] | null | Promise<PolicySource[] | null>
  catalog?:    (url: URL) => CatalogItemSource[] | null | Promise<CatalogItemSource[] | null>
  document?:   (url: URL) => DocumentSource | null | Promise<DocumentSource | null>
  collection?: (url: URL) => CollectionSource | null | Promise<CollectionSource | null>
  match?:      (url: URL) => SourceKind | null   // pure + sync; at most one resolver runs
}

maxBytesBySource exists because one global budget is the wrong shape once documents exist: a store can hold products to a tight budget on purpose (trymumm.com runs 8192 for policy reasons) without forcing the same ceiling onto a long article. A source with no entry falls back to maxBytes, then to render-md's default.

handleRequest(req, config)

• Response — markdown (text/markdown; charset=utf-8) rendered via @rebilder/render-md, with Vary: Accept (the same URL serves HTML to browsers — caches must key on Accept) and X-Rebilder-Path: markdown; or a protocol response served by your wired config.protocols handler (see § Protocols).
• null — serve your normal HTML. Covers humans, crawlers, protocol routes without a wired protocols handler (or where it answered null or threw), URLs no source matched, sources returning null, sources that threw, and sources that hung past sourceTimeoutMs. A broken source never breaks your site — errors are contained, slow ones are cut off, the event still fires, the request passes through.

Events

Every handled request — including pass-throughs — emits exactly one RebilderEventV0 when onEvent is configured: requester from detection (kind/platform/verified), request (url, accept, referrer), and response.path ('markdown', 'html-variant' for pass-throughs, 'protocol') with measured render_ms. Emission is fire-and-forget: never awaited, sync throws and async rejections are swallowed; a broken or slow event sink can neither block nor break a response.

One schema note: crawlers ride the human/HTML serving path but are recorded with the first-class kind: 'crawler' (with platform preserved — 'googlebot', 'bingbot'), keeping them distinguishable from humans in the warehouse. The kind union lives in packages/events and matches the events table check constraint in infra/sql/rebilder/0002_events.sql.

Protocols (UCP / ACP / MCP) — the protocols hook

Protocol endpoints live in @rebilder/protocols (spec-versioned adapters: /.well-known/ucp discovery + catalog + checkout handoff, /acp/v0/feed, /mcp JSON-RPC). The gateway serves them through the optional config.protocols hook:

import { handleRequest, type GatewayConfig } from '@rebilder/gateway'
import { createProtocolHandler } from '@rebilder/protocols'

const sources = { product, policies, catalog } // ProtocolSources is structurally
                                               // identical to GatewaySources —
                                               // one wiring object serves both

const config: GatewayConfig = {
  storeId: 'store_123',
  sources,
  protocols: createProtocolHandler({
    storeId: 'store_123',
    sources,
    checkout: { handoffUrl: (productUrl) => merchantCheckoutUrlFor(productUrl) },
    // leave onEvent unset here — the gateway already emits one event per request
  }),
  onEvent: (event) => queue(event),
}

Semantics: the hook is invoked only when a request classifies onto the 'protocol' path. A returned Response is served as-is, and the request's event records response.path: 'protocol' with measured render_ms (the hook's time included). null — or an unset hook, or a hook that throws (contained, like every gateway failure) — preserves the exact pre-hook pass-through behavior, event included.

Why isn't @rebilder/protocols a gateway dependency? Merchants install this SDK in production stacks, and we are ruthless about its dependency count (see the policy above): a store that only wants the markdown path shouldn't carry protocol adapters, and protocol spec versions must ship on their own cadence without version-bumping the gateway. So the merchant constructs the handler and passes it in — the hook is just a function type, and any (req: Request) => Promise<Response | null> satisfies it.

PSP note: checkout over these protocol endpoints is a redirect handoff onto the merchant's own PSP rails (Stripe/Adyen/Shop Pay/…). Neither this SDK nor @rebilder/protocols ever holds, moves, or custodies funds — the checkout response carries a handoff_url and nothing else.

Routing note (signed agents): @rebilder/agent-detect ranks Signature-Agent above the protocol route, so a Web Bot Auth-signed UCP/MCP call detects as kind: 'agent'. The gateway still routes it onto the 'protocol' path when the URL is a protocol route and the request did not explicitly ask for markdown — otherwise signed protocol clients (exactly the agents verification exists for) would never reach the protocol handler. An explicit Accept: text/markdown keeps its documented precedence, even on /mcp.

Verification (Web Bot Auth) — the verification config

Agent verification, before accepting protocol transactions. Wire it by injecting a key registry:

import { handleRequest, type GatewayConfig, type AgentKeyRegistry } from '@rebilder/gateway'

const registry: AgentKeyRegistry = { /* operator-populated — see the honesty note below */ }

const config: GatewayConfig = {
  storeId: 'store_123',
  sources,
  protocols: createProtocolHandler({ storeId: 'store_123', sources, checkout }),
  verification: { keys: registry },   // require?: 'protocol' — the default and only v0 scope
}

Semantics, exactly:

• When set and a request classifies onto the 'protocol' path with a wired protocols hook, the gateway runs verifyWebBotAuth (from @rebilder/agent-detect) before invoking the hook. This is pure Ed25519 crypto over the injected keys — no network call, no key-directory fetch, ever (the hot-path budget holds).
• The hook is invoked with a cloned request carrying the verdict: x-rebilder-agent-verified: 'true' | 'false', plus x-rebilder-agent-verified-reason: <reason> when unverified (no-signature, unknown-agent, bad-signature, expired, …). Client-sent values of these headers are always overwritten — a spoofed verdict cannot survive the gateway.
• Unverified requests still reach the hook. Read endpoints (discovery, catalog, feed, MCP tools) stay open to unverified agents; gating transactional endpoints on the verdict is the protocol handler's job — @rebilder/protocols' UCP checkout requires x-rebilder-agent-verified: true by default (checkout.requireVerified, default true).
• The emitted event records the outcome: requester.verified reflects the cryptographic verdict, and a verified platform identity fills requester.platform when detection had nothing better.
• When unset (the default): behavior is byte-for-byte the pre-verification gateway — no clone, no headers stamped, and (deliberately) no stripping of client-sent verdict headers. requester.verified stays false everywhere.

Honesty notes. (1) The shipped KNOWN_AGENT_DIRECTORY is empty — we do not invent production platform keys. Until the operator populates a registry (offline, from the platforms' published key directories — see @rebilder/agent-detect's README § Verification), every request verifies false/unknown-agent, and a default-configured checkout answers 403 verification_required; the operator's alternatives are to populate keys or to opt the checkout out (requireVerified: false). (2) The verdict header is a trusted channel between this gateway and the protocol handler: it is only meaningful when the handler sits behind a verification-configured gateway (which overwrites it) or behind an edge that strips it. Do not expose a header-gating protocol handler directly to the internet and call it verified.

Access control (Web Bot Auth) — the access config

Allow, deny, or rate-limit agents on your own site. Off by default: with access unset, every agent is served exactly as before this field existed.

export const gatewayConfig: GatewayConfig = {
  storeId: 'store_123',
  sources: { /* … */ },
  verification: { keys: myRegistry },      // required for platform-named rules
  access: {
    default: 'allow',
    rules: [
      { subject: 'chatgpt', action: 'allow' },
      { subject: 'unverified', action: 'limit', limit: { requests: 60, windowSeconds: 60 } },
      { subject: '*', action: 'allow' },
    ],
  },
}

Subjects, most specific first. A platform id ('chatgpt', 'claude', …) matches a verified agent of that platform. 'unverified' matches any agent that presented no verifiable identity. '*' matches every agent. When nothing matches, default applies ('allow' unless you say otherwise).

A rule naming a platform requires verification. Without it the compiler rejects that rule and reports why — it does not enforce it. Allow/deny keyed on a self-asserted Signature-Agent or User-Agent is not a control; it is a suggestion that anyone can walk past by editing a header, and shipping it as a control is a security misrepresentation. 'unverified' and '*' need no verification, because they are rules about the absence of proof rather than rules that trust a claim.

Nothing here can reach a human or a search crawler. Googlebot always gets canonical HTML (Hard Rule 3), under every policy, including default: 'deny'. A policy that could deindex your site is one that eventually would.

A denial costs you nothing. Enforcement runs before any source resolver, any protocol handler, and any rendering — so refusing a request is cheaper than serving one, which is the right way round. Denied requests answer 403 (429 with Retry-After for a rate limit), with a small JSON body: the recipient is a program, and a styled error page teaches it nothing.

Denied requests still emit an event, on the 'denied' serving path (@rebilder/events 0.3.0). A policy you cannot see the effect of is a policy you cannot tell from one that is blocking your best traffic.

Two limits worth knowing before you turn this on

The rate limiter is per process instance. A token bucket in memory is the only limiter available to code with no shared state, so an agent hitting ten edge locations gets ten buckets. This is a politeness brake and a defence against one runaway client — not a guarantee about global request volume. A hard global limit needs a shared counter, which means a network call per request, which is the one thing the edge budget forbids. We would rather ship the honest version than a global-looking number that is not one.

Policy is compiled once, and never fetched. access takes a value or a synchronous getter — deliberately not a Promise, which would be an invitation to fetch per request. To swap policy at runtime, poll and verify on your own schedule and assign the result to a variable your getter reads; the gateway recompiles only when the object identity changes, and the rate-limit counters survive the swap (otherwise a 30-second poll would silently reset every window). The compiled policy and the limiter are keyed on the access value itself, not the config wrapper — so a multi-tenant host that builds a fresh config per request keeps one budget per tenant by reusing the tenant's policy object, and two gateways handed distinct policy literals keep distinct budgets.

let current: AccessPolicy = BOOT_POLICY
setInterval(async () => { current = await fetchAndVerifyPolicy() }, 300_000)

export const gatewayConfig: GatewayConfig = { /* … */, access: () => current }

Validate where policy is authored, not where it is enforced. compileAccessPolicy(policy, { verificationConfigured }) is exported so an admin UI can show "3 rules rejected, here is why" at save time. A policy that saves clean and silently enforces two thirds of itself is the failure this export exists to prevent.

This is not a quota, and this package still has no plans. The Gateway Rule above is unchanged. A quota is us metering you, and it belongs at ingest. This is your policy about third-party agents on your site, handed in already decided — the gateway cannot tell whether you are entitled to author it, has no way to find out, and never asks.

Next.js adapter (@rebilder/gateway/next)

Imports nothing from next — Next middleware/proxy handlers and app-router route handlers speak web-standard Request/Response. This repo's apps use the Next 16 proxy.ts convention; the identical code works in a Next ≤15 middleware.ts.

Install

npm install @rebilder/gateway        # pnpm add / yarn add

Installing with a coding agent

Paste the block below into Claude Code, Cursor, Copilot or whatever you use. It is written for an agent working in your repository, and it front-loads the two things that are easy to get wrong and hard to notice later: passing the fallthrough, and checking every URL you wired rather than the first one.

It deliberately tells your agent to ask you for the data-mapping rather than guess it. Nobody else knows where your prices live, and a plausible guess about a price is the one failure mode this package exists to prevent.

Install @rebilder/gateway in this repository.

WHAT IT DOES
It is middleware. When an AI agent requests one of our URLs with
`Accept: text/markdown`, it answers with a clean markdown rendering of that
page's facts. Every other request — humans, Googlebot, anything that did not ask
for markdown — falls through to the existing pipeline untouched. Same URL, same
substance, different format.

STEPS
1. Read the README of the installed package before writing anything. Do not
   work from memory of this prompt; the API is in
   node_modules/@rebilder/gateway/README.md.
2. Add the dependency with the package manager this repo already uses.
3. Create a gateway config exporting a `GatewayConfig`. Its `sources` resolvers
   map a URL to our own data. There are five source kinds and every page on
   this site is one of them. Three are commerce shaped: `product`, `policies`,
   `catalog`. Two are universal: `document` for any other page, such as a
   guide, an article, a service, a location or an FAQ, and `collection` for an
   index of documents. A site with no catalog wires only the universal two, and
   a site with both wires all five. Do not skip a section of the site because
   it is not commerce. Wire the adapter for this repo's framework (`/next`,
   `/node` for Express and Fastify, `/edge` for Workers, `/shopify`).
4. Pass the existing fallthrough into the adapter rather than calling it
   yourself. That is what puts `Vary: Accept` on the HTML response. Without it a
   shared cache can serve markdown to a human or stale HTML to an agent, and it
   will not show up until it is in production behind a CDN.
5. Add a `match` router to `sources` if you can: one pure, synchronous function
   from URL to source kind. It keeps the hot path to a single lookup and it is
   what lets the adapter advertise the markdown alternate.

RULES — these are not style preferences
- Never invent a substantive value. Prices, stock, shipping costs, return
  windows, policy text and dates on a commerce page, and opening hours,
  turnaround times, eligibility rules, what a service includes and whether
  booking is required on any other page, are read from our source of truth and
  passed through unchanged. If you cannot find where a value lives,
  stop and ask me. Do not approximate, do not use a placeholder that looks
  real, and do not write an example value into a resolver.
- A resolver returns `null` when the URL is not that kind of page. `null` means
  "fall through to HTML", which is always a safe answer.
- Do not change what the HTML pages render. This is additive.
- Do not add a build step, a new service, or a runtime dependency.

WHEN YOU ARE DONE
Tell me which URL patterns you wired and which you deliberately skipped, then
run this against one real URL of EACH kind you wired — not just the first one:

  npx rebilder diff <url>     # shows the agent view and the browser view
  npx rebilder check <url>    # grades the page, names what is missing

An incomplete source map is the most common install defect and it is invisible
until an agent hits the page you missed. Both commands run locally, upload
nothing, and need no account.

If you cannot answer "which URLs now serve markdown", the install is not done.

Middleware / proxy

Two lines of plumbing; the integration is the sources block. The wiring below is genuinely four lines and never changes. What takes real time is the next snippet — mapping your URLs onto the source types, against your database, your API, or your CMS. Budget an afternoon for a first real catalog, not two minutes. It is the same work any structured-data integration asks for, and pretending otherwise just moves the surprise later.

// lib/gateway-config.ts — wire the gateway to your source of truth
import type { GatewayConfig } from '@rebilder/gateway'
import { getProduct, getPolicies, getCollection } from './catalog' // your code
import { getDocument, getDocumentIndex } from './content'          // your code

export const gatewayConfig: GatewayConfig = {
  storeId: 'store_123',
  sources: {
    // Commerce shaped. Wire the ones you have; a site with no catalog wires none.
    product:  (url) => getProduct(url.pathname),      // null when not a PDP
    policies: (url) => getPolicies(url.pathname),
    catalog:  (url) => getCollection(url.pathname),
    // Universal: any other page, and any index of them. Guides, services,
    // locations, articles. Most of a site usually lives here.
    document:   (url) => getDocument(url.pathname),
    collection: (url) => getDocumentIndex(url.pathname),
  },
  onEvent: (event) => { /* queue to your analytics sink; fire-and-forget */ },
}

// proxy.ts (Next 16) — middleware.ts on Next ≤15 is identical
import { NextResponse } from 'next/server'
import { createGatewayProxy } from '@rebilder/gateway/next'
import { gatewayConfig } from './lib/gateway-config'

// Pass your fallthrough. The proxy then returns a Response for every request,
// and the HTML half of each negotiated URL gets `Vary: Accept` — see below for
// why that matters more than it looks.
const gateway = createGatewayProxy(gatewayConfig, () => NextResponse.next())

export default gateway

export const config = { matcher: ['/products/:path*', '/policies/:path*', '/collections/:path*'] }

Response short-circuits with markdown, and your HTML pipeline runs unchanged for everyone else.

Pass the fallthrough — the HTML half has to declare the negotiation too

The one-argument form still works and still returns null, but it leaves one thing to you that is easy to miss and expensive to get wrong.

Your URL now has two representations. The markdown one is ours and carries Vary: Accept. The HTML one is your framework's, and carries whatever Vary your framework set — for Next, the RSC router tokens and no Accept at all. A shared cache that stores the HTML under a key without Accept can hand it to an agent that asked for markdown, or hand your markdown to a shopper's browser. It surfaces weeks later, through a CDN, as "a customer saw a wall of asterisks on our product page."

We shipped this bug ourselves — rebilder.com ran the gateway for months with Vary missing from the HTML half, while this README told you the rule. Passing the fallthrough is now the fix, and the adapters do it for you: the Express middleware sets the header on res before calling next(), and the Workers handler decorates the origin response, both without any change to your code.

Next.js needs one more step, and how far it gets depends on where you host. Set it in next.config.ts for the paths you serve markdown on:

async headers() {
  return ['/', '/products/:path*', '/policies/:path*'].map((source) => ({
    source,
    headers: [
      {
        key: 'Vary',
        value:
          'rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch, Accept',
      },
    ],
  }))
}

Repeat Next's RSC tokens rather than sending Accept alone — dropping them would break client-side navigation, and a duplicate token is a no-op because Vary is a set.

Why the config and not middleware: Next's app router calls res.appendHeader('vary', …) on the final response (base-server.js), so it never replaces yours — but a Vary set in middleware does not survive to the wire, while a Link set on the same line does. The config entry lands in routes-manifest.json and is applied at the routing layer.

On Vercel the edge then strips it, and that is not a bug you can fix. Verified on a fresh PRERENDER response with age: 0: the manifest carries …, Accept, the wire does not. Vercel's Edge Network keys its own cache and normalises Vary to the tokens it understands.

You are still safe on Vercel, for a structural reason rather than luck: middleware runs before the cache lookup, so an agent's request is intercepted and served markdown even when that exact URL's HTML is a warm cache hit. We verified it — /learn on rebilder.com returns cached HTML to a browser (x-vercel-cache: HIT, age: 519) and text/markdown to an agent on the same request, with no cache involvement at all.

Where it matters is a cache you put in front of an origin — Cloudflare, Fastly, a corporate proxy — because that cache sits outside your middleware and will happily serve HTML to an agent. There, the config header does the job, and if your CDN also ignores Vary, add Accept to the cache key directly (Cloudflare: Cache Rules → custom cache key; Fastly: vcl_hash).

Express and Workers need none of this — those adapters own the response and nothing rewrites it. Check whichever you run: curl -sSI https://your-store.example/products/x | grep -i vary. A test asserting the adapter sets the header is not the same claim as the header being on the wire; that gap is exactly how this shipped wrong twice.

Keeping the ?? NextResponse.next() shape? Wrap it:

import { createGatewayProxy, withNegotiationHeaders } from '@rebilder/gateway/next'

const gateway = createGatewayProxy(gatewayConfig)

export default async function proxy(req: Request) {
  return (
    (await gateway(req)) ??
    withNegotiationHeaders(NextResponse.next(), gatewayConfig, new URL(req.url))
  )
}

Vary: Accept is added to every response the middleware passes on, whether or not a source matched today — under-declaring it is the cache bug, and a source you add tomorrow must not be served out of an entry cached today. The Link: rel="alternate" advertisement is added only when your config has a match router that confirms the URL has a source: pointing an agent at a representation that does not exist is worse than saying nothing.

Check that it worked

Do not skip this. Our own install passed review and still missed the site's most important URL, because nobody ran the check.

npx rebilder diff https://your-store.example/products/some-product

It fetches the URL twice — once as a browser, once as an agent — and shows you both. What you want to see:

• the agent view is markdown, with your real price and availability in the first few lines;
• the browser view is your HTML, unchanged;
• every URL you meant to wire returns markdown, not just the first one you tried. A source map with a gap is the most common install defect, and it is invisible until an agent hits the missing page.

npx rebilder check <url> then grades the page and tells you what is still missing. Both run on your machine, upload nothing, and need no account.

First live integration (dogfood): trymumm.com wires exactly this in apps/mumm-web/proxy.ts — gateway first, ?? updateSession(request) fallthrough — with its config in apps/mumm-web/lib/gateway/config.ts.

Optional dedicated markdown route

A stable always-markdown URL (agent permalink, "what agents see" preview) — renders markdown for any requester; 404 JSON when no source matches:

// app/md/[[...path]]/route.ts
import { createGatewayRouteHandler } from '@rebilder/gateway/next'
import { gatewayConfig } from '../../../lib/gateway-config'

export const GET = createGatewayRouteHandler(gatewayConfig, { stripPrefix: '/md' })

stripPrefix removes the route prefix (whole path segments only) before consulting your sources, so /md/products/x resolves against sources keyed by canonical paths (/products/x). This route is its own URL, so it doesn't conflict with the cloaking guardrail — the guardrail is about serving different substance on the same URL.

Shopify adapter (@rebilder/gateway/shopify)

The largest reachable merchant pool, served through a Shopify App Proxy — no theme edits, no Liquid, no new dependencies (Web Crypto is ambient).

What an app proxy is. Shopify lets an app claim a subpath on the merchant's own storefront domain (default here: /apps/rebilder/...). Requests to that subpath are forwarded server-side by Shopify to a URL the app hosts, and the response is returned to the requester from the shop's domain. So an agent fetching https://acme.myshopify.com/apps/rebilder/products/x is answered by this gateway — the agent-facing URL lives on the store, the serving lives with us.

Merchant setup (Shopify Partners)

1. In the Partners dashboard → your app → Configuration → App proxy: set Subpath prefix apps, Subpath rebilder (⇒ storefront path /apps/rebilder), and Proxy URL to your deployed handler endpoint.
2. Copy the app's Client secret — it is the HMAC key for proxy signatures. Provide it to the handler via env; never hardcode it.
3. Deploy the handler on any web-standard runtime:

// e.g. app/apps/rebilder/[[...path]]/route.ts — any Request/Response runtime works
import { createShopifyAppProxyHandler } from '@rebilder/gateway/shopify'
import { gatewayConfig } from '../../lib/gateway-config'

export const GET = createShopifyAppProxyHandler(gatewayConfig, {
  sharedSecret: process.env.SHOPIFY_APP_SECRET!,
  pathPrefix: '/apps/rebilder', // as the path arrives at YOUR endpoint; strip is a no-op if absent
})

Security model

Shopify appends query params (shop, path_prefix, timestamp, logged_in_customer_id) plus signature: a hex HMAC-SHA256, keyed with the app's shared secret, over the other params canonicalized as sorted key=value strings concatenated with no separator (values of a repeated key joined with ,). The handler:

1. Verifies the signature with Web Crypto (crypto.subtle — edge-safe, no node:crypto) and a constant-time comparison. Failure → 401 JSON. Content is never served on an unverified proxy request — verifyAppProxySignature(url, sharedSecret) is exported for reuse.
2. Checks timestamp freshness: a signed timestamp older than 90s (±5s clock-skew tolerance) → 401 JSON, bounding the replay window of a captured signed URL.
3. Only then reconstructs the canonical storefront URL — strips pathPrefix and Shopify's injected params, host from the signed shop param — and runs the core handleRequest. Your sources and emitted events see the same canonical URLs (https://{shop}/products/x) as every other adapter.

Because the app proxy is the endpoint — there is no downstream HTML to fall through to — every handleRequest pass-through (null) becomes 404 JSON { "error": "no_source" }: unmatched URLs, sources that returned null or threw, and non-agent requesters (a human or crawler hitting the proxy subpath gets 404, not markdown; the storefront's real pages live at their canonical URLs).

Scope: this adapter is the serving path. A full Shopify app — OAuth install flow, billing, embedded admin — is later work; until it exists, the shared secret and proxy config are set up manually per merchant in Partners.

Node adapter (@rebilder/gateway/node) — Express & Fastify

Connect-style middleware for Node HTTP servers. Imports nothing from express, fastify, or even node:http — the adapter is written against structural types that any real Node request/response object satisfies, so the zero-external-deps invariant holds.

Install & Express integration

npm install @rebilder/gateway        # pnpm add / yarn add

import express from 'express'
import { createGatewayMiddleware } from '@rebilder/gateway/node'
import { gatewayConfig } from './gateway-config'   // same GatewayConfig as every adapter

const app = express()
app.use(createGatewayMiddleware(gatewayConfig))    // before your routes
// ... your existing routes serve HTML exactly as before

Semantics are handleRequest's, mapped onto the middleware contract:

• Markdown path (agent + matching source): status + headers copied from the core response (text/markdown; charset=utf-8, Vary: Accept, X-Rebilder-Path: markdown), body written with res.end(). next() is not called — the gateway answered. The body is buffered before writing (no streaming) — deliberately fine: rendered markdown is capped by maxBytes (default 5KB; trymumm runs 8KB), so there is nothing worth streaming.
• Pass-through (null): next() — humans, crawlers, unmatched URLs, throwing sources; your pipeline runs untouched.
• Any adapter error (e.g. an unparseable Host header): next(). The gateway never crashes a merchant's server and never leaves a request hanging.

The web-standard Request is built from the Node request: host from the Host header, protocol from x-forwarded-proto (first value — set by your proxy/LB) → Express's req.protocol → socket.encrypted, and Express's originalUrl preferred over url (mounted routers rewrite url; sources must see the real path). Repeated (array) headers are appended per value. toWebRequest(nodeReq, options?) is exported for reuse if you want core handleRequest semantics against a raw Node request yourself.

Fastify

Two options, no fastify-specific code in this package:

// Option A — Fastify's middleware compat layer (@fastify/middie):
import middie from '@fastify/middie'
await fastify.register(middie)
fastify.use(createGatewayMiddleware(gatewayConfig))

// Option B — a 3-line onRequest hook over the raw req/res (no plugin needed):
const gateway = createGatewayMiddleware(gatewayConfig)
fastify.addHook('onRequest', (req, reply, done) => {
  gateway(req.raw, reply.raw, done) // markdown answered on res; otherwise done() continues
})

With Option B, Fastify's routing never sees gateway-answered requests (the response is written on the raw socket), and every pass-through continues through done() into your normal routes.

Cloudflare Worker / edge adapter (@rebilder/gateway/edge)

A fetch-handler factory for web-standard edge runtimes. No Cloudflare imports — (req: Request) => Promise<Response> is the whole contract.

// worker.ts — deploy on a route in front of the store, e.g. store.example.com/*
import { createGatewayFetchHandler } from '@rebilder/gateway/edge'
import { gatewayConfig } from './gateway-config'

export default { fetch: createGatewayFetchHandler(gatewayConfig) }

That is a complete worker: agent traffic with a matching source gets the standard markdown response; everything else is fetch(req)-ed to the origin unchanged — the standard CF Worker reverse-proxy pattern — so browsers and crawlers get the canonical HTML exactly as if the worker weren't there. Pass options.fallback: (req) => Response | Promise<Response> to serve the non-markdown path yourself instead of proxying.

One honest latency note: this is the one adapter where a network call happens — the origin pass-through — and only on the non-markdown path (a round-trip that would occur without the worker anyway). The markdown path stays pure compute within the edge budget. A gateway-internal error is contained like every other failure: the request falls through to fallback/origin, never a 500.

Vercel Edge note: the same plain fetch handler works on Vercel Edge Functions (or any WinterCG runtime) — export it as the handler; use options.fallback there, since Vercel Edge functions don't sit in front of an origin the way a route-mounted CF Worker does (in Next.js projects, prefer the ./next adapter, which plugs into middleware natively).

llms.txt

One honest paragraph first: content negotiation (Accept: text/markdown) measured ~4.2x more effective than llms.txt for accurate retrieval (300k-domain study, mid-2026), and llms.txt alone shows no citation lift. We ship it because it's cheap to generate, free to serve, and merchants ask for it — but it is not the strategy (VISION § Market Context); the gateway's markdown path is.

generateLlmsTxt(config, options) (root export) produces spec-shaped markdown — H1 site name, blockquote description, H2 sections of link lists — deterministically, with injected values only, no LLM anywhere. If config.sources.catalog / config.sources.collection / config.sources.policies enumerate for new URL(options.baseUrl) (they may return null or throw; both are tolerated as "nothing to list"), a ## Products section (title, URL, price), a ## Pages section (title, URL, and the item summary when it fits on one line), and a ## Policies section are generated automatically, in that order; options.sections appends manual sections in order after them. A config with no collection source produces byte-identical output to before ## Pages existed.

// app/llms.txt/route.ts
import { createLlmsTxtRouteHandler } from '@rebilder/gateway/next'
import { gatewayConfig } from '../../lib/gateway-config'

export const GET = createLlmsTxtRouteHandler(gatewayConfig, {
  baseUrl: 'https://store.example.com',
  siteName: 'Acme Outdoors',
  description: 'Trail footwear and gear, shipped from Bend, OR.',
  sections: [{ title: 'Guides', links: [{ title: 'Sizing guide', url: 'https://store.example.com/pages/sizing' }] }],
})

The route handler serves text/plain; charset=utf-8 with long shared-cache headers (s-maxage=3600) — safe because the output is deterministic.

What this package does, and what it does not

| Capability | Status | |---|---| | Markdown path: classify + render product / policies / catalog | Shipped | | Universal path: document / collection sources, match router, per-source budgets | Shipped | | Event emission via onEvent (types from @rebilder/events) | Shipped — the emission client and ingest pipeline live in @rebilder/events | | Next.js adapter (./next) | Shipped | | Shopify adapter (app proxy, ./shopify) | Shipped | | llms.txt generation (generateLlmsTxt + Next route helper) | Shipped | | Express/Fastify (./node) and Cloudflare Worker / edge (./edge) adapters | Shipped | | Protocol endpoints (UCP / ACP / MCP via config.protocols) | Shipped — see § Protocols. Without the hook, protocol routes pass through and events record the demand | | Web Bot Auth verification on the protocol path (config.verification) | Shipped — see § Verification. Enforced once you populate the key registry; the shipped directory is empty on purpose | | Preference-payload personalization | Not shipped | | Human variant serving (html-variant with variant_id) | Not shipped |

Scripts

pnpm --filter @rebilder/gateway lint        # eslint (flat config)
pnpm --filter @rebilder/gateway typecheck   # tsc --noEmit (strict)
pnpm --filter @rebilder/gateway test        # vitest — every behavioral rule above is a test