@vulcantech/seo-files

CMS-driven robots.txt and llms.txt for VulcanTech consumer sites. Reads each file from the CMS with Next.js ISR tags, and falls back to the app's own compile-time content whenever the CMS has nothing saved, is unreachable, or errors — so the routes always serve valid text. Shared so each site stops hand-rolling its robots.ts / llms.txt route.

Exports: createSeoFileRoute (the route-handler factory), fetchSeoFiles (the raw CMS fetch), tagsForSeoFiles (the ISR tag helper), and the SeoFileType / SeoFilesPayload types.

Server-only — uses VULCANTECH_API_KEY; import from route handlers only.

Integration

Wire SEO files into a consumer site (Next.js App Router). Each step is concrete and verifiable:

1. Add the dependency:

npm install @vulcantech/seo-files

(Inside the VulcanTech monorepo, use the workspace wiring in MONOREPO.md instead.)

2. Create the fallbacks src/lib/seo/fallbacks.ts — export ROBOTS_FALLBACK and LLMS_FALLBACK, seeded verbatim from the app's current hardcoded robots / llms.txt content so nothing drifts when the CMS is empty:

export const ROBOTS_FALLBACK = `User-Agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml`;

export const LLMS_FALLBACK = `# Example Site
...`;

3. Create the routes src/app/robots.txt/route.ts and src/app/llms.txt/route.ts via the factory, then DELETE src/app/robots.ts — a Metadata robots.ts and a literal robots.txt/route.ts conflict and Next will fail the build:

// src/app/robots.txt/route.ts
import { createSeoFileRoute } from "@vulcantech/seo-files";
import { ROBOTS_FALLBACK } from "@/lib/seo/fallbacks";
export const { GET } = createSeoFileRoute({ type: "robots", fallback: ROBOTS_FALLBACK });

// src/app/llms.txt/route.ts
import { createSeoFileRoute } from "@vulcantech/seo-files";
import { LLMS_FALLBACK } from "@/lib/seo/fallbacks";
export const { GET } = createSeoFileRoute({ type: "llms", fallback: LLMS_FALLBACK });

4. Set the env (already present on sites using @vulcantech/blog):

• VULCANTECH_PROJECT_ID (required)
• VULCANTECH_API_KEY (required — the key needs the seo-files:read scope)
• VULCANTECH_CMS_URL (optional, default https://cms.vulcantech.io)

5. ISR invalidation — ensure src/app/api/revalidate/route.ts exists; on save the CMS purges the tag seo-files:{projectId} and the paths /robots.txt and /llms.txt for this project.

Fallback semantics

• CMS returns non-empty content → that content is served.
• CMS content is empty / whitespace-only, OR the fetch fails / errors → the app's fallback is served instead.
• An empty robots.txt is impossible by construction: whitespace-only CMS content is treated as "nothing saved" and the fallback takes over.
• Bodies are normalized on the way out: CRLF → LF, with exactly one trailing newline.

Required env

VULCANTECH_PROJECT_ID, VULCANTECH_API_KEY (required) · VULCANTECH_CMS_URL (optional, defaults to https://cms.vulcantech.io).

Per-app values

The ROBOTS_FALLBACK / LLMS_FALLBACK strings in src/lib/seo/fallbacks.ts — everything else is shared and config-driven from the CMS.

Notes

• tagsForSeoFiles(projectId) must match the CMS-side tag exactly (seo-files:{projectId}) — keep them in sync.