@se-studio/site-check

Validate SE marketing sites (sitemap.xml, sitemap-unindexed.xml, llms.txt) and download all markdown files into a local directory, preserving URL path structure.

Usage

npx @se-studio/site-check <baseUrl> [-o dir] [-H "Name: value"] [--vercel-bypass [secret]] [--check-against <devUrl>] [--check-rewrites] [--ignore-missing-child-sitemaps]

Examples

# Local site
npx @se-studio/site-check http://localhost:3015

# Custom output directory
npx @se-studio/site-check http://localhost:3015 -o ./out

# Compare production vs development: use production sitemap, check pages exist on dev
npx @se-studio/site-check https://example.com --check-against https://example.localhost

# Enable rewrite check (optional; off by default)
npx @se-studio/site-check https://example.com --check-rewrites

# Vercel Deployment Protection: use env secret (preferred name)
VERCEL_PROTECTION_BYPASS_TOKEN=your-secret npx @se-studio/site-check https://preview.vercel.app --vercel-bypass

# Legacy alias still works
VERCEL_AUTOMATION_BYPASS_SECRET=your-secret npx @se-studio/site-check https://preview.vercel.app --vercel-bypass

# Vercel bypass with explicit secret
npx @se-studio/site-check https://preview.vercel.app --vercel-bypass your-secret

# Custom headers
npx @se-studio/site-check https://example.com -H "Authorization: Bearer token" -H "X-Custom: value"

# Sitemap index lists a child that does not exist locally (404) — skip it and use remaining children
npx @se-studio/site-check http://localhost:3016 --ignore-missing-child-sitemaps -o ./out

What it does

1. Validates the site by fetching:

   • sitemap.xml (required) — must return 200 and contain <loc> entries; if it is a sitemap index, child sitemaps are fetched and must return valid urlset content (or use --ignore-missing-child-sitemaps to skip child URLs that return 404)
   • sitemap-unindexed.xml (optional) — if 404, a warning is printed and the run continues. When it exists (returns 200), its page URLs are included in all checks (existence on dev, optional rewrite check, markdown collection).
   • llms.txt (optional by default) — if 404, a warning is printed and the run continues; use --strict-discovery to require all four discovery endpoints (llms.txt, markdown-index.txt, cms.txt, site-info.md) with content validation

2. Collects markdown URLs from sitemap.xml and sitemap-unindexed.xml (Option B): when either URL returns a sitemap index (e.g. <sitemapindex> with child <sitemap><loc>...</loc></sitemap>), the tool follows those links and collects page URLs from each child sitemap’s urlset (one level only). When the response is a normal urlset, every <loc> is treated as a page URL. Each page URL is then converted to a .md URL (path + .md). This ensures all indexed and unindexed pages are checked.

3. Checks for unexpected rewrites (only when --check-rewrites is set): For each page URL from the sitemaps, the tool sends a HEAD request and reads the x-nextjs-rewritten-path response header. If the header is present and the rewritten path is different from the requested path (after normalisation), the run fails (exit 1). This catches cases where the sitemap lists one URL (e.g. /learning-hub/blog/) but the app rewrites to another (e.g. /articles/blog). Rewrites that only add or remove a locale prefix are allowed when SITE_CHECK_LOCALES is set (see Options). By default the rewrite check is disabled.

4. Checks and downloads: For each markdown URL, the tool fetches it. If any return non-2xx, the run fails (exit 1) and reports which URLs are missing or errored. Otherwise it saves each response to the output directory, preserving path structure (e.g. blog/foo.md → ./markdown-export/blog/foo.md, es-US/about.md → ./markdown-export/es-US/about.md).

Compare mode (--check-against)

When --check-against <devUrl> is set, the tool runs in compare mode:

• Production (the <baseUrl>) is the source of the URL list: sitemap.xml and sitemap-unindexed.xml are fetched from production. Validation (and llms.txt) applies to production.
• Development (<devUrl>) is the site checked: each production sitemap page URL (including from sitemap-unindexed.xml when present) is mapped to the same path on the development origin. The tool then checks that each of those development URLs returns 2xx (HEAD). If --check-rewrites is set, it also runs the rewrite check against the development site. Markdown collection and download are not performed in compare mode.

Use this to ensure a development or staging build has the same pages as production (e.g. before release). Page title comparison is not yet implemented; a future option may add it.

Options

| Option | Description | |--------|-------------| | baseUrl | Base URL of the site (required). In compare mode this is the production URL (sitemap source). Trailing slash is stripped. | | -o, --out <dir> | Output directory for markdown files (default: ./markdown-export). Ignored in compare mode. | | -H, --header "Name: value" | Add a request header (repeatable). Applied to both production and development requests in compare mode. | | --check-against <devUrl> | Compare mode: use production sitemap(s) but check that each page exists on development (same path on <devUrl>). No markdown download. | | --check-rewrites | Enable checking for unexpected rewrites (x-nextjs-rewritten-path). Default: off. When set, the run fails if any sitemap page URL rewrites to a different path (locale-only rewrites allowed with SITE_CHECK_LOCALES). | | --ignore-missing-child-sitemaps | When the root sitemap (or sitemap-unindexed.xml) is a sitemap index, any child <loc> that returns 404 is skipped instead of failing validation. Other non-2xx responses still fail. After skipping, at least one page URL must still be collected from the remaining children. | | --vercel-bypass [secret] | Set x-vercel-protection-bypass for Vercel Deployment Protection. If secret is omitted, uses VERCEL_PROTECTION_BYPASS_TOKEN from the environment (aliases: VERCEL_AUTOMATION_BYPASS_SECRET, VERCEL_BYPASS_TOKEN). |

Headers from -H override the Vercel bypass header if the same name is used.

Environment

| Variable | Description | |----------|-------------| | SITE_CHECK_LOCALES | Comma-separated list of locale path segments (e.g. en,en-gb,de). When set, a rewrite is considered acceptable if the only difference between the requested path and the rewritten path is a leading locale segment. If unset, any rewrite to a different path fails. | | PREVIEW_SITE_URL | Vercel develop/preview deployment base URL (preview smoke tests, protected preview fetches). Trailing slash is stripped. | | SITEMAP_PROD_URL | Public production site base URL (sitemap-validate scripts, smoke case curation). Trailing slash is stripped. | | PRODUCTION_SITE_URL | Production site base URL for audits and tooling that need a canonical prod origin. Trailing slash is stripped. | | VERCEL_PROTECTION_BYPASS_TOKEN | Vercel Deployment Protection bypass secret (Vercel → Settings → Deployment Protection). Sent as x-vercel-protection-bypass. Aliases: VERCEL_AUTOMATION_BYPASS_SECRET, VERCEL_BYPASS_TOKEN. |

Production HTML audit (@se-studio/site-check/production-audit)

Programmatic module for live production SEO audits (used by per-app pnpm seo:audit:production scripts). Import from @se-studio/site-check/production-audit:

• runProductionSeoAudit, writeProductionSeoAuditOutputs, buildRichResultsSpotChecks, getProductionAuditExitCode

URL sources: sitemap.xml and sitemap-unindexed.xml (same resolver as the CLI — follows sitemap indexes). sitemap-unindexed.xml 404 is optional.

Markdown checks: Every sitemap page URL must return 200 for its .md mirror (/about/ → /about.md, / → /index.md). Excluded: /download/ paths and static file extensions (.pdf, etc.) via isNonPageAssetUrl. Additional exclusions: skipMarkdownPathPrefixes in config.

Discovery endpoints: Production audit validates all four discovery routes (llms.txt, markdown-index.txt, cms.txt, site-info.md) with content checks; issues appear in discoveryIssues on the audit report.

markdown-index.txt: Fetched for a supplementary missing_from_markdown_index report; not used to gate .md existence checks.

Exit code: getProductionAuditExitCode returns 1 when any row has severity: 'error' (including markdown_missing).

Smoke tests (@se-studio/site-check/smoke-test)

Curated local smoke tests for marketing sites: verify HTML and .md for URLs listed in smoke.cases.json, plus optional discovery endpoint checks (llms.txt, markdown-index.txt, cms.txt, site-info.md) via the discovery block (see skill se-marketing-sites-smoke-test-setup).

Optional cmsIntegrity in smoke.cases.json adds a local-only Contentful article-link check (parallel with pnpm smoke-test server boot). Deployment / live smoke remains HTTP-only.

Requires ^2.6.1 when using cmsIntegrity or pnpm smoke-test with integrity enabled.

Import from @se-studio/site-check/smoke-test:

• loadStaticSmokeConfig, runStaticSmokeTest, runStaticSmokeTestWithIntegrity, runPreviewStaticSmokeTest, runSmokeTest
• formatSmokeTestReport, formatCombinedSmokeReport, getSmokeTestExitCode, getCombinedSmokeExitCode

Per-app usage (see apps/example-empty):

pnpm build --filter @se-studio/site-check   # once, or after package changes
cd apps/example-empty
pnpm smoke-test:run                         # dev server must be running on app port
pnpm smoke-test                             # dev server + smoke (default dev:dev) + stop
pnpm smoke-test:preview                     # curated cases against Vercel preview (no local server)

Add smoke.cases.json, scripts/smoke-test-run.ts, and optionally scripts/smoke-test-preview.ts (or use node ../../scripts/smoke-test-preview.mjs from the app directory). No Contentful or cms-server imports required. Set SMOKE_TEST_IGNORE=true in .env.local to skip the full smoke-test CI script.

Preview smoke (runPreviewStaticSmokeTest): reads PREVIEW_SITE_URL and VERCEL_PROTECTION_BYPASS_TOKEN from .env.local (or process.env), applies the bypass header, and runs the same curated cases against the live preview deployment. No local dev server required.

smoke-test-one env:

| Variable | Purpose | |----------|---------| | SMOKE_TEST_SERVER_SCRIPT=dev:dev | Default dev server; use start:dev for prod-like next start (runs pnpm build first; not portless) | | SMOKE_TEST_SKIP_BUILD=true | Skip pnpm build when SERVER_SCRIPT=start:dev (use after Vercel next build) | | SMOKE_TEST_VERIFY_CACHE=true | Run smoke twice; assert x-nextjs-cache on HTML (meaningful with start:dev, soft in next dev) | | SMOKE_TEST_AUDIT_CACHE_LOGS=true | Pipe server/build logs and scan for 2MB cache failures, cache skip, and hard refresh | | SMOKE_TEST_CACHE_SKIP_SEVERITY=warn | warn or error for cache-skip log lines (default warn) | | SMOKE_TEST_READY_PATH=/ | Path polled until the server responds | | SMOKE_TEST_USE_RUNNING_SERVER=false | Force a fresh server (default true for local dev) | | SMOKE_TEST_SKIP_INTEGRITY=true | HTTP smoke only (set by smoke-test-one when cmsIntegrity runs in parallel) |

LOG_CMS_FETCH=1 is set when verify-cache or audit-cache-logs is enabled. The smoke HTTP client never sends Cache-Control: no-cache.

CMS article-link integrity (cmsIntegrity in smoke.cases.json)

{
  "cmsIntegrity": {
    "enabled": true,
    "routing": {
      "articleTypesBasePath": "/resources",
      "tagsBasePath": "/topics",
      "peopleBasePath": "/team",
      "defaultTopic": "other",
      "enablePrimaryTagPartOfSlug": true,
      "topiclessArticleTypeSlugs": ["ebooks"]
    }
  }
}

Map routing from the app’s constants.ts / URL rules. Use runStaticSmokeTestWithIntegrity in scripts/smoke-test-run.ts (see apps/example-empty).

Import from @se-studio/site-check/cms-integrity for standalone checks: runArticleLinkIntegrityFromSmokeConfig, evaluateArticleLinkIntegrity, getArticleLinkIntegrityExitCode.

Vercel build cache (VERCEL_FORCE_NO_BUILD_CACHE)

Not a smoke variable — set on the Vercel project when next build fails intermittently with webpack WasmHash errors (often with stale remote build cache; local pnpm build may still pass).

vercel env add VERCEL_FORCE_NO_BUILD_CACHE preview develop --value 1 --yes
vercel env add VERCEL_FORCE_NO_BUILD_CACHE production --value 1 --yes

Cannot be set in vercel.json — Vercel restores cache before install/build commands. Tradeoff: slower cold builds, more reliable compiles.

Deploy check (smoke-test-deploy-check): post-build gate for Vercel — starts pnpm start without rebuilding, runs pnpm smoke-test:run, exits non-zero on failure. Port defaults from smoke.cases.json. Set SMOKE_TEST_IGNORE=true in Vercel env to bypass in an emergency.

{
  "buildCommand": "pnpm build && pnpm smoke-test:deploy-check"
}

Per-app script: "smoke-test:deploy-check": "smoke-test-deploy-check" (or node ../../scripts/smoke-test-deploy-check.mjs in the monorepo).

Cross-site develop timing (develop-sites-timing): probes all client marketing sites (smoke paths from each repo’s smoke.cases.json) against develop and production. Runs N sequential full passes (default 5) over every path — one request at a time. Reads VERCEL_PROTECTION_BYPASS_TOKEN from each site’s .env.local; never hardcodes secrets.

pnpm develop-sites-timing
pnpm develop-sites-timing --site pointme --passes 3 --output /tmp/timing.json

| Option | Purpose | |--------|---------| | --site <key> | One site: se2026, om1, brightline, brightlifekids, pedestal, headwater, pointme | | --passes <n> | Full passes over all paths (default 5) | | --output <path> | JSON report (default ./develop-sites-timing.json) | | --delay-ms <n> | Pause between requests | | --develop-only / --production-only | Probe one environment | | --source-root <path> | Client repos root (default ~/source, or SE_SOURCE_ROOT) |

Live deployment smoke (smoke-test-live / runLiveSmokeTest): HTTP checks against a deployed URL — use with Vercel repository_dispatch (vercel.deployment.ready) and Deployment Checks to hold production domain aliasing until smoke passes.

Discovery URL origins on production deploys: Production builds often emit canonical URLs in llms.txt / markdown-index.txt (e.g. https://www.example.com/...) while Deployment Checks smoke the pre-alias *.vercel.app hostname. By default discovery.urlOrigin is auto: when every discovery URL shares one consistent HTTPS origin, content validation accepts that canonical origin. Key pages spot-checks then fetch the same paths on the smoke deployment base URL (not live CDN/www, which may still be the previous alias or an incomplete edge). Set urlOrigin: "deployment" for strict matching of URLs in the body, or canonicalBaseUrl to pin the expected content origin explicitly.

| Variable | Purpose | |----------|---------| | DEPLOYMENT_URL | Live deployment base URL (e.g. github.event.client_payload.url) | | SMOKE_TEST_BASE_URL | Alias for DEPLOYMENT_URL | | PREVIEW_SITE_URL | Fallback for local runs (same as preview smoke) |

Per-app script: "smoke-test:live": "smoke-test-live". GitHub Actions: set DEPLOYMENT_URL from the dispatch payload, report status with vercel/repository-dispatch/actions/status@v1, register the check name in Vercel Deployment Checks settings.

Content assertions (contentAssert)

Optional per-case HTML checks for release verification (key blocks must render, not only HTTP 200):

{
  "category": "person",
  "label": "Provider hero",
  "path": "/providers/jane-doe/",
  "expectMarkdown": true,
  "contentAssert": {
    "mustInclude": ["Jane Doe"],
    "mustHaveSelector": ["img"],
    "imagesMustLoad": true
  }
}

Publish-state check (CDA vs CPA)

Before promoting when Contentful entries changed, compare Delivery vs Preview:

check-publish-state --id <entryId> --id <otherId> --include-links
check-publish-state --type template --include-links

Programmatic: import { checkPublishState } from '@se-studio/site-check/publish-state'. Requires CONTENTFUL_SPACE_ID, CONTENTFUL_ACCESS_TOKEN, CONTENTFUL_PREVIEW_ACCESS_TOKEN, and optionally CONTENTFUL_ENVIRONMENT. See docs/RELEASE_VERIFICATION.md.

Exit codes

• 0 — Validation passed (sitemap.xml required; llms.txt and sitemap-unindexed.xml optional). If --check-rewrites was set, no unexpected rewrites. Every sitemap page returned 200 for its .md URL and files were saved (or in compare mode: all production sitemap pages exist on development and, if --check-rewrites, no unexpected rewrites).
• 1 — Usage error (missing baseUrl), validation failed (sitemap.xml), one or more sitemap pages rewrite to a different path when --check-rewrites was set, one or more markdown URLs returned non-2xx (pages missing or broken), or in compare mode one or more development URLs did not return 2xx. Validation does not fail when llms.txt returns 404.