@vulcantech/seo-files

CMS-driven robots.txt and llms.txt for VulcanTech consumer sites. Reads each file from the CMS with Next.js ISR tags, and falls back to the app's own compile-time content whenever the CMS has nothing saved, is unreachable, or errors — so the routes always serve valid text. Shared so each site stops hand-rolling its robots.ts / llms.txt route.

Exports: createSeoFileRoute (the route-handler factory), fetchSeoFiles (the raw CMS fetch), tagsForSeoFiles (the ISR tag helper), and the SeoFileType / SeoFilesPayload types.

Server-only — uses VULCANTECH_API_KEY; import from route handlers only.

What the exports return

createSeoFileRoute({ type, fallback }) → { GET }

The factory returns an object with a single GET route handler. That handler always resolves to a 200 Response — there is no error branch, no 404, no 500. The only thing that varies is which text is in the body:

Status:        200 (always)
Content-Type:  text/plain; charset=utf-8
Cache-Control: public, s-maxage=60, stale-while-revalidate=300
Body:          CMS content if it is non-null and has non-whitespace characters,
               otherwise the `fallback` string you passed in.
               Either way: CRLF → LF, exactly one trailing newline.

Two consequences worth planning around:

• You never need error handling around the route. No try/catch, no status branch, no "SEO files unavailable" state — none of it can fire.
• The fallback covers empty and unreachable, not invalid. Whatever the CMS has saved is served verbatim (bar newline normalization). The CMS does not validate on write — the dashboard shows advisory amber warnings (missing User-agent:, HTML pasted in, blanket Disallow: /, >100 KiB) but never blocks a save. A malformed robots.txt in the CMS is published as a malformed robots.txt.

fetchSeoFiles() → SeoFilesPayload | null

The raw fetch behind the factory. One call returns both files:

interface SeoFilesPayload {
  robotsTxt: string | null;  // the saved robots.txt body; null when never saved
  llmsTxt: string | null;    // the saved llms.txt body; null when never saved
  updatedAt: string | null;  // ISO 8601, when the pair was last saved; advisory
}

Field notes, verified against the CMS route (/api/seo-files GET) and the dashboard editor that writes projects/{projectId}.seoFiles:

• "" is a real value, distinct from null. The editor saves an emptied draft verbatim, so a file the user cleared comes back as an empty string, not null. That is exactly why the route's usability test is .trim().length > 0 rather than a null check — treat empty and whitespace-only as "nothing saved".
• An unconfigured project is a 200, not a 404. A project that has never opened the SEO-files page returns { robotsTxt: null, llmsTxt: null, updatedAt: null }. Do not expect a 404 to signal "not set up".
• updatedAt covers the pair, not a single file — the dashboard stamps one new Date().toISOString() over both bodies on save. This package does not read it; it is there for consumers that want a "last synced" line.
• Both routes share one upstream request. robots.txt and llms.txt hit the same URL with the same tags, so they share one Next Data Cache entry — wiring both routes does not double CMS traffic.

Caching and revalidation

• The upstream CMS fetch is issued with next: { revalidate: 3600, tags: ["seo-files:{projectId}"] }, so CMS content is served from the Next Data Cache for up to 1 hour.
• The route handler itself is not cached. Neither this package nor the consumer routes set dynamic / revalidate route-segment config, and Next does not cache Route Handlers by default — so the handler runs per request and the freshness window comes from the Data Cache entry plus the 60s CDN s-maxage on the response.
• On save the CMS pushes. It POSTs to every webhook in the project's formConsumers (the same fan-out forms and banners use) with { projectId, tags: ["seo-files:{projectId}"], paths: ["/robots.txt", "/llms.txt"] }, fire-and-forget with a 5s timeout. Your site therefore needs src/app/api/revalidate/route.ts, a matching VULCANTECH_REVALIDATE_SECRET, and to be registered in that project's formConsumers (Dashboard → Settings → Form Consumers (ISR Revalidation); the list serves every integration, not just forms) — see Integration step 5. Without all three, edits are not lost, just slow: they appear within the 1-hour Data Cache window (plus up to 60s of CDN staleness).
• There is no cache-bypass env var for this package (unlike VULCANTECH_BLOG_NO_CACHE in @vulcantech/blog). To see a CMS edit immediately in local dev, restart the dev server or trigger the revalidate webhook by hand.

Failure mode

Nothing in this package throws, and no failure is visible to a visitor. fetchSeoFiles() returns null — after a single console.warn on the server — in every one of these cases, and the route then serves your fallback:

| What went wrong | CMS response | | --- | --- | | VULCANTECH_PROJECT_ID or VULCANTECH_API_KEY unset | (no request made) | | projectId / apiKey missing from the query | 400 | | API key invalid, or the key's project not found | 401 | | Key lacks the seo-files:read scope | 403 | | Request IP not in the key's allowlist | 403 | | Project document does not exist | 404 | | CMS Firestore read failed | 500 | | CMS unreachable / DNS / timeout | (fetch throws) |

The practical trap: a key missing the seo-files:read scope fails silently. The site keeps serving the compile-time fallback and looks healthy — the only evidence is a [cms/seo-files] warning in the server log. If CMS edits are not showing up, check the key's scopes before anything else.

Integration

Wire SEO files into a consumer site (Next.js App Router). Each step is concrete and verifiable:

1. Add the dependency:

npm install @vulcantech/seo-files

(Inside the VulcanTech monorepo, use the workspace wiring in MONOREPO.md instead.)

2. Create the fallbacks src/lib/seo/fallbacks.ts — export ROBOTS_FALLBACK and LLMS_FALLBACK, seeded verbatim from the app's current hardcoded robots / llms.txt content so nothing drifts when the CMS is empty:

export const ROBOTS_FALLBACK = `User-Agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml`;

export const LLMS_FALLBACK = `# Example Site
...`;

3. Create the routes src/app/robots.txt/route.ts and src/app/llms.txt/route.ts via the factory, then DELETE src/app/robots.ts — a Metadata robots.ts and a literal robots.txt/route.ts conflict and Next will fail the build:

// src/app/robots.txt/route.ts
import { createSeoFileRoute } from "@vulcantech/seo-files";
import { ROBOTS_FALLBACK } from "@/lib/seo/fallbacks";
export const { GET } = createSeoFileRoute({ type: "robots", fallback: ROBOTS_FALLBACK });

// src/app/llms.txt/route.ts
import { createSeoFileRoute } from "@vulcantech/seo-files";
import { LLMS_FALLBACK } from "@/lib/seo/fallbacks";
export const { GET } = createSeoFileRoute({ type: "llms", fallback: LLMS_FALLBACK });

4. Set the env (already present on sites using @vulcantech/blog):

• VULCANTECH_PROJECT_ID (required)
• VULCANTECH_API_KEY (required — the key needs the seo-files:read scope)
• VULCANTECH_CMS_URL (optional, default https://cms.vulcantech.io)

5. ISR invalidation — ensure src/app/api/revalidate/route.ts exists; on save the CMS purges the tag seo-files:{projectId} and the paths /robots.txt and /llms.txt for this project.

Fallback semantics

• CMS returns non-empty content → that content is served.
• CMS content is empty / whitespace-only, OR the fetch fails / errors → the app's fallback is served instead.
• An empty robots.txt is impossible by construction: whitespace-only CMS content is treated as "nothing saved" and the fallback takes over.
• Bodies are normalized on the way out: CRLF → LF, with exactly one trailing newline.

Required env

VULCANTECH_PROJECT_ID, VULCANTECH_API_KEY (required) · VULCANTECH_CMS_URL (optional, defaults to https://cms.vulcantech.io).

Per-app values

The ROBOTS_FALLBACK / LLMS_FALLBACK strings in src/lib/seo/fallbacks.ts — everything else is shared and config-driven from the CMS.

Notes

• tagsForSeoFiles(projectId) must match the CMS-side tag exactly (seo-files:{projectId}) — keep them in sync.