@growth-labs/seo

Astro integration for complete SEO infrastructure on Cloudflare. Handles JSON-LD structured data, meta tags, sitemaps, RSS/podcast/Apple News feeds, AEO (Answer Engine Optimization) with crawler-class dispatch, multilingual support, robots.txt, llms.txt / llms-full.txt, and build-time validation.

Quick start — prerendered content site (WarFronts pattern)

For immutable-after-publish content (articles, docs, archived posts). Zero bindings required.

import seo from '@growth-labs/seo'

export default defineConfig({
  integrations: [
    seo({
      site: 'https://warfronts.channel',
      organization: {
        name: 'WarFronts',
        logo: 'https://media.warfronts.channel/logos/header.png',
      },
      aeoTwins: true,  // → { mode: 'static' }
      llmsTxt: true,
      rss: true,
      // Module specifier (not function). Required for the Astro Cloudflare adapter's
      // prerender Worker, which doesn't re-execute astro.config.mjs so a captured
      // function reference never reaches it. The module must default-export the
      // ContentProvider.
      contentProviderModule: '/src/lib/content-provider.mjs',
    }),
  ],
})

// src/lib/content-provider.mjs
import { getCollection } from 'astro:content'

export default async function contentProvider({ type, slugs }) {
  const entries = await getCollection(type === 'articles' ? 'articles' : 'pages')
  return entries
    .filter((e) => !slugs || slugs.includes(e.slug))
    .map((e) => ({ url: `https://example.com/${e.slug}`, title: e.data.title, ... }))
}

mode: 'static' emits .md twin files at build time (via an injected prerender route) to dist/client/article/<slug>.md. Cloudflare Assets (or any static host) serves them directly. The build hook also writes a managed dist/client/_headers block so prerendered HTML responses carry the Link: <canonical-url>.md; rel="alternate"; type="text/markdown" markdown-alternate header. To stay under Cloudflare's 100-rule _headers limit on large sites, a route family of single-segment twins (e.g. /article/<slug>) collapses to one placeholder rule (/article/:slug → Link: <…/article/:slug.md>) rather than one rule per page; irregular paths fall back to literal rules, and any overflow past 100 rules is dropped with a build warning.

Premium publisher (fronts.co pattern) — SSR, gated content, Flexible Sampling

For paid publications where members see the full body and verified Googlebot gets the paywall-marked full body under Google's Flexible Sampling policy.

seo({
  site: { envVar: 'SITE_URL' },
  organization: { name: 'Fronts', logo: 'https://fronts.co/logo.png' },
  aeoTwins: {
    mode: 'middleware',
    onDemandRevalidation: true,
    revalidateToken: import.meta.env.SEO_REVALIDATE_TOKEN,  // ≥32 random bytes
    freshLayer: { bindingName: 'AEO_TWINS', type: 'r2' },
  },
  flexibleSampling: { enabled: true, sampleMode: 'lead-in', leadInParagraphs: 2 },
  llmsTxt: true,
  llmsFullTxt: true,
  contentProviderModule: '/src/lib/content-provider.mjs',
})

Gated articles need prerender: false on their route file — enforced at build time via the prerender-gated-content guard.

Foundry CTR title/meta/canonical overrides

Sites can opt into the Foundry-managed CTR experiment layer without giving agents CMS write access:

seo({
  site: 'https://homefronts.pub',
  organization: { name: 'HomeFronts', logo: 'https://homefronts.pub/logo.png' },
  seoOverrides: {
    siteId: 'homefronts',
  },
})

This injects POST /admin/api/seo/overrides, reads/writes gl_seo_overrides in the configured D1 binding (SITE_DB by default), and uses SEO_OVERRIDE_HMAC_SECRET for Foundry HMAC auth. Apply payloads may include canonicalUrl; the route rejects malformed, cross-host, query-bearing, fragment-bearing, or path-policy-blocked canonicals before D1 writes. <SeoHead /> applies active overrides to SERP/social copy (<title>, meta description, OG/Twitter) and canonical URL surfaces (<link rel="canonical">, og:url, JSON-LD URL); the original article title still feeds H1 and NewsArticle.headline. Existing D1 users should apply migrations/0002_gl_seo_overrides_canonical_url.sql before sending canonical override payloads; new installs get the columns in 0001.

Host guard

Preview and staging surfaces (*.workers.dev, staging.example.com) serve the same bytes as the canonical site. Without a guard, Google can index them as duplicate content competing with the real domain — the estate SEO audit found ten channel workers.dev subdomains and multiple staging hosts serving indexable duplicates.

hostGuard is on by default. When a request's Host differs from the configured site host (and isn't loopback/localhost or a dev build):

• GET /robots.txt returns User-agent: *\nDisallow: / (200, text/plain).
• Every other HTML response gets X-Robots-Tag: noindex, nofollow appended.

seo({
  site: 'https://example.com',
  organization: { /* … */ },
  // Default: hostGuard: true. Allow extra canonical hosts (a CDN, a vanity domain):
  hostGuard: { allowedHosts: ['cdn.example.net'] },
  // Or turn it off entirely:
  // hostGuard: false,
})

The canonical host is taken from site (the { envVar } form is resolved from the Worker env at request time). Localhost and astro dev are never blocked.

Publisher logo asset guidance

requirePublisherLogo (default on) errors when an Article/NewsArticle ships without publisher.logo. The organization.logo you pass must satisfy Google's publisher-logo constraints:

• An absolute URL (not a relative path).
• Ideally a wordmark / rectangular mark on a transparent or solid background — Google renders it at up to 600×60 in rich results, so a square favicon reads poorly. A square variant is acceptable but a wide wordmark is preferred.
• Served from a host the crawler can fetch (not behind auth / the host guard).

www → apex and trailing slashes on Cloudflare Assets

Cloudflare Assets' html_handling emits 307/308 for trailing-slash normalization, and a www. → apex move needs a redirect the origin never sees. Handle both at the edge, not in the app:

• www → apex: a Cloudflare Bulk Redirect (or a Redirect Rule) from www.example.com/* → https://example.com/$1, 301, preserve-path/query. Keep site set to the apex so canonicals and the host guard agree.
• Trailing slash: pick one policy and set trailingSlash in the SEO options to match your astro.config trailingSlash; let html_handling's 308 do the normalization rather than emitting per-page redirects.

llms.txt auto-population

With llmsTxt: true and no hand-wired llmsContent, /llms.txt derives its body from the content provider instead of shipping a one-line stub:

• Public articles → an Article Markdown Twins section, each link pointing at the item's .md twin URL with its description.
• Videos / Podcasts sections when the provider returns those types.
• The > … site description is taken from organization.tagline.
• access: 'members' items are excluded.

An explicit llmsContent always wins. When aeoTwins and llmsTxt are both enabled but the rendered llms.txt has zero content links, the build warns; validation.requireLlmsEntries: true upgrades that to a build-failing error.

seo({
  organization: { name: 'Example', logo: '…', tagline: 'The place for great articles.' },
  aeoTwins: true,
  llmsTxt: true,           // auto-populated from the provider
  contentProviderModule: '/src/lib/content-provider.mjs',
})

Vary hygiene

Vary tokens are added only when a feature that branches the response on that request header is active — an over-broad Vary makes the HTML surface edge-uncacheable for features a site doesn't use.

| Token | Added when | |---|---| | Accept | AEO mode is middleware or both (content negotiation on the same URL) | | User-Agent, CF-Connecting-IP | crawler classification is consulted: LLM-training-crawler blocking, AEO middleware dispatch, or Flexible Sampling | | Cookie | an auth-segment resolver is wired (the response branches on cookies) |

A static-mode-twins-only site (no crawler gating, no middleware negotiation, no auth segment) adds no Vary, so its HTML stays cacheable at the edge.

Cache audiences

Middleware writes Astro.locals.seoCacheAudience and Astro.locals.seoCacheKey for consumers that build their own cache layer. Public and verified crawler renders never share a key. member remains accepted as the legacy subscriber segment and maps to the subscriber cache audience; registered, metered, subscriber, and admin responses are forced to Cache-Control: private, no-store.

Validators

All thresholds live under the validation option and respect the noindex exemption. New in 0.9.0:

| Option | Default | Effect | |---|---|---| | descriptionMinLength | 50 | Warn when a present meta description is shorter (a missing one keeps its own warning) | | titleMinWords | 12 | Warn when the <title> has fewer words (counted before the | suffix) | | requirePublisherLogo | true | Error when Article/NewsArticle JSON-LD lacks publisher.logo | | requireDateModified | true | Error when article-schema JSON-LD lacks dateModified | | articleImageMinCount | 3 | Warn when the article JSON-LD image array has fewer entries (Discover trio) |

Plus two checks with no threshold: a generic-title detector (error when the pre-suffix title is home/index/untitled/welcome on an indexable page) and a heading-hierarchy warning (heading levels that skip, e.g. h2→h4).

AEO frontmatter fields

ContentItem carries optional enrichment that generateAeoMarkdown emits into the .md twin frontmatter (omitted entirely when absent — no empty keys):

{
  topic: 'Federal retirement',
  namedEntities: ['TSP', 'FERS', 'OPM'],
  sources: [
    { title: 'OPM Handbook', url: 'https://opm.gov/handbook' },
    'https://tsp.gov',   // bare URL form also accepted
  ],
}

emits:

topic: Federal retirement
named_entities:
  - TSP
  - FERS
  - OPM
sources:
  - url: https://opm.gov/handbook
    title: OPM Handbook
  - url: https://tsp.gov

RSS feed shape

/feed.xml normalizes each item so <description> carries a summary (item.summary → item.description → a ~300-char word-boundary truncation of the full body) and the full body moves to <content:encoded> (CDATA) when a body resolver is provided. /rss.xml is injected as a 301 alias to /feed.xml.

seo({
  rss: true,
  feed: { fullContent: true },   // default; false = summary-only, no content:encoded
})

The content:encoded body is supplied by a contentHtml resolver when calling generateRssFeed directly (mirrors the Apple News feed contract).

Site URL Resolution

site accepts three forms:

seo({ site: 'https://fronts.co', ... })
seo({ site: () => 'https://fronts.co', ... })
seo({ site: { envVar: 'SITE_URL' }, ... })

String values and resolver function return values are validated as URLs when Astro config is parsed. { envVar } is resolved by runtime routes and head components from the standard Cloudflare Workers env binding (cloudflare:workers), then validated as a URL. process.env is only a Node/test fallback for local tooling; consumers do not need nodejs_compat, process.env, or a Fronts-local runtime shim for site URL resolution.

wrangler.toml (required for middleware/both modes)

# Fresh-twin storage (R2 preferred, KV acceptable)
[[r2_buckets]]
binding = "AEO_TWINS"
bucket_name = "my-site-aeo-twins"

# Revalidation Coordinator — rate limit + per-slug lock + idempotency
[[durable_objects.bindings]]
name = "AEO_REVALIDATION_COORD"
class_name = "AeoRevalidationCoordinator"

[[migrations]]
tag = "v1"
new_sqlite_classes = ["AeoRevalidationCoordinator"]

# Version metadata (R2 key prefixing for rollback safety)
[version_metadata]
binding = "CF_VERSION_METADATA"

# Daily prune cron — deletes old-version R2 entries
[triggers]
crons = ["0 3 * * *"]

# Assets binding — MUST set not_found_handling to "none" or middleware's
# env.ASSETS.fetch() for missing twins will return the SPA fallback.
[assets]
binding = "ASSETS"
directory = "./dist/client"
not_found_handling = "none"

Re-export the DO class + scheduled handler from your Worker entrypoint:

export { AeoRevalidationCoordinator } from '@growth-labs/seo/durable-objects'
export { pruneAeoR2 } from '@growth-labs/seo/cron'

export default {
  async scheduled(event, env, ctx) {
    await pruneAeoR2({ env })
  },
}

What it injects

Middleware (order: post):

• Enforces the host guard (default on): off-host requests (*.workers.dev, staging) get a Disallow robots.txt and X-Robots-Tag: noindex, nofollow. See "Host guard" below.
• Classifies every request: verified search crawler (known search crawler UA + FCrDNS), LLM training crawler, user-directed LLM agent, anonymous.
• Sets Astro.locals.crawlerClass, effectiveAuthSegment, seoCacheAudience, and seoCacheKey for consumer cache-key builders.
• 403s LLM training crawlers on access: 'members' items.
• Adds Content-Signal header on every response.
• Adds Vary tokens only for the axes an active feature branches on — a static-mode-twins-only site adds none. See "Vary hygiene" below.
• Adds Link: rel="alternate"; type="text/markdown" on HTML responses (suppressed for members items).
• Serves .md twins via Accept: text/markdown content-negotiation in middleware/both modes (R2 → Assets → 503 stub + background render fallthrough).

Routes:

• /sitemap-index.xml and /sitemap.xml + sitemap-articles.xml, sitemap-pages.xml, sitemap-videos.xml, sitemap-products.xml
• /sitemap-markdown.xml — twin URL sitemap (static/both modes only)
• /robots.txt
• /llms.txt, /llms-full.txt
• /feed.xml (RSS) and /rss.xml (301 → /feed.xml)
• /apple-news.xml (Apple News Publisher RSS, if enabled)
• /podcast.xml, /listen.xml
• POST /_seo/revalidate — CMS webhook target (when onDemandRevalidation: true)

Consumers that own an injected sitemap path can disable only that route with injectedRoutes, for example injectedRoutes: { sitemapVideos: false }. The generated sitemap index omits disabled child sitemaps and, when a content provider is wired, child sitemaps with no sitemap-eligible entries.

Head-tag components:

• <SeoHead /> emits <title>, description, canonical, robots with max-image-preview:large, hreflang, OG/Twitter fields, Apple News discovery, markdown twin links, and JSON-LD. Use it in layouts instead of local meta shims.
• JSON-LD includes WebSite for non-content pages, Article/NewsArticle or Product for content pages, and VideoObject / AudioObject when ContentItem.video or ContentItem.audio is present.
• For @growth-labs/opengraph, pass getOgImageUrl() output as item.image or defaults.defaultImage; SeoHead uses that URL for og:image and twitter:image.
• <AeoHead /> remains available standalone when a site only wants Apple News discovery and markdown twin links.

Runtime behavior

@growth-labs/seo self-seeds config through virtual:growth-labs/seo/config. Runtime entrypoints resolve bindings from the standard Cloudflare surfaces:

• cloudflare:workers for env bindings
• Astro.locals.cfContext.waitUntil() for background tasks

For site: { envVar: 'SITE_URL' }, the package reads SITE_URL from the Worker env binding at request time. The Node process.env fallback exists only for tests and build tooling.

Build-time:

• Emits .md twins + summary twins for public items (static/both modes) under dist/client/.
• Writes Cloudflare Static Assets _headers rules for primary .md twins so prerendered HTML carries the same markdown alternate HTTP Link header that middleware adds to Worker-rendered pages. Single-segment route families collapse to one Cloudflare :slug placeholder rule to stay under the 100-rule _headers limit; overflow beyond 100 rules is dropped with a build warning.
• Validates hreflang reciprocity.
• Validates no prerendered route serves a members-gated item (when Flexible Sampling is enabled).
• Per-page HTML validation (title length + word floor + generic-title detector, meta description min/max length, canonical, H1, heading-hierarchy skips, hero image, Article JSON-LD publisher.logo + dateModified + image-array trio, and max-image-preview:large). Hard validation errors fail the build; warnings remain informational. See "Validators" below.
• Warns when aeoTwins + llmsTxt are enabled but the rendered /llms.txt carries zero content links (validation.requireLlmsEntries: true makes it a build error).

Rendered responses: <AeoHead /> sets the markdown alternate HTTP Link header when it emits the matching <link rel="alternate" type="text/markdown"> head tag. The SEO middleware appends the same header for middleware-mode AEO responses and deduplicates when the component already set it.

Crawler classes

| Class | What they see | Notes | |---|---|---| | verifiedSearchCrawler | Full body (+ paywall JSON-LD on gated items under Flexible Sampling) | Known search crawler UA plus FCrDNS-verified Googlebot/Bingbot/Applebot/DuckDuckBot IP. Cloudflare Bot Management metadata is advisory only and never promotes by itself. | | llmTrainingCrawler | 403 on members items, public body on public items | GPTBot, ClaudeBot, CCBot, PerplexityBot, Applebot-Extended, etc. | | userDirectedLlmAgent | Anonymous body only, regardless of cookies | ChatGPT-User, Claude-User, PerplexityBot-User, Google-NotebookLM. Load-bearing override prevents cookie-based leakage. | | anonymous | Public body or gate | Everything else. |

Standalone utilities

All pure-function utilities are available without the integration:

import { generateArticleJsonLd, generateProductJsonLd } from '@growth-labs/seo/utils'
import { generateMeta } from '@growth-labs/seo/utils/meta'
import { generateAppleNewsRss, generateAppleNewsAnf } from '@growth-labs/seo/utils'
import { classifyRequest, createFcrdnsVerifier } from '@growth-labs/seo/utils'
import { buildSeoCacheKey, computeEffectiveAuthSegment } from '@growth-labs/seo/utils'

JSON-LD generators: Article, NewsArticle, BlogPosting, FAQPage, VideoObject, AudioObject, Person, HowTo, Product, BreadcrumbList, Organization, WebSite, ItemList, SpeakableSpecification

Feed generators: RSS, Apple News Publisher RSS, Apple News Format (ANF) JSON, podcast RSS, listen feed, llms.txt, llms-full.txt

Utilities: OG + Twitter Card meta, sitemap XML, markdown sitemap, hreflang, robots.txt, AEO markdown generator with RAG chunk markers, summary twin generator, content-hash staleness.

Non-Cloudflare hosts

aeoTwins: { mode: 'static' } works on any host that serves static files (Vercel, Netlify, GitHub Pages, S3+CloudFront). Other modes ('middleware', 'both') and onDemandRevalidation require Cloudflare Workers + R2/KV + Durable Objects. See packages-seo-SPEC-v2.md "Deployment Targets" for details.

Key patterns

• Virtual module: virtual:growth-labs/seo/config
• Runtime routes and middleware resolve bindings from standard Cloudflare surfaces
• AI crawler blocking: enforced at robots.txt AND per-request 403 on members items
• .md twin canonical: emitted with X-Robots-Tag: noindex + Link: <html-url>; rel="canonical" — prevents Google from clustering the .md as a duplicate of the HTML
• Summary twins: .summary.md companion emitted when summaryTwin: true (default), with a 4-tier fallback (item.summary → bullets → first-sentence-per-section → description-only)
• Build-time validation: fails on required structural SEO defects and gated content on prerendered routes

Full spec

See packages-seo-SPEC-v2.md for the complete 2,400-line specification including the test matrix, architectural rationale, and worked examples for every code path.