Frontify Guideline Scraper

Scrape a Frontify brand portal and save every guideline page as a PDF and clean semantic HTML.

How it works

1. Discover — Loads the portal homepage to read the x-csrf-token meta tag (required by some hubs such as SBB), then queries Frontify's portal and document navigation APIs to build the full site tree (documents, pages, groups, headings, external links).
2. Crawl — Visits every page with Playwright, expands accordions, forces lazy images to load, dismisses cookie/overlay dialogs, then saves a PDF and raw HTML snapshot.
3. Clean — Strips the raw HTML down to semantic content (headings, text, images, tables) with no scripts, styles, or navigation chrome.

Setup

npm install
npx playwright install chromium

Usage

You need a portal URL (argument, --url, or URL in the environment). Running node crawl.mjs alone prints a short hint and exits.

node crawl.mjs --url brand.uber.com

Or pass a full URL:

node crawl.mjs --url https://developer.frontify.com

With pnpm or npm, use -- so arguments reach the script:

pnpm start -- https://brand.sbb.ch --hub 2 -c 'frontify-session-id=…'

If you run node crawl.mjs -- … (a -- right after the script name), that is supported too: the scraper strips that separator so flags like --hub still parse correctly.

URLs with # (hash routes): In the shell, # starts a comment. Quote the whole URL, or drop the hash (only the origin matters for discovery):

pnpm start -- 'https://brand.sbb.ch/d/…/marke#/section/page' --hub 2
# same effect for discovery:
pnpm start -- https://brand.sbb.ch/d/…/marke --hub 2

Options

| Flag | Short | Description | |------|-------|-------------| | --url <url> | -u | Portal domain or full URL | | --hub <id> | -h | Hub ID (auto-detected if omitted) | | --cookie <str> | -c | Cookie pair; repeat -c for each (-c a=1 -c b=2) or one -c "a=1; b=2" | | --csrf <token> | | Optional. X-CSRF-Token from your browser (see below). If omitted, discovery fetches it from the portal homepage. | | --help | | Show help |

These can also be set via environment variables or a .env file:

URL=brand.uber.com
HUB_ID=25
COOKIE=frontify-session-id=your-session-id
CSRF_TOKEN=…

Passing CSRF from the browser (optional)

On hubs that protect APIs with CSRF, the token is in the page HTML:

<meta name="x-csrf-token" content="…">

Copy the content value and pass:

node crawl.mjs --url https://brand.sbb.ch --hub 2 --csrf 'paste-token-here'

Or set CSRF_TOKEN in the environment. If you omit --csrf, discovery reads the meta tag from the portal.

If you pass -c / cookies, discovery always reloads CSRF using that cookie session and uses your --url (without #…) as Referer on API calls, so the token matches the session (avoids 403 from a stale or mismatched --csrf).

Output

output/{domain}/
  pdf/
    Group Name/
      Document Title.pdf
      Document Title/
        Page Title.pdf
  html/
    Group Name/
      Document Title.html
      ...

• PDF — Full-page A4 captures with background graphics, expanded accordions, and loaded lazy images.
• HTML — Cleaned semantic HTML: headings, paragraphs, images, tables. No scripts, styles, classes, or navigation elements. Wrapped in minimal readable CSS.

Discover only

Run the discovery step standalone to inspect or save the navigation tree:

node discover.mjs --url brand.uber.com --output brand.uber.com.json

This outputs a JSON tree of the portal's structure without crawling any pages.

Clean HTML only

Re-clean previously scraped raw HTML:

node purge-html.mjs output/.raw/html output/clean

Authenticated portals

For portals that require login, copy the Cookie header from browser dev tools (Application → Cookies, or the request headers for /api/portal-navigation/…) and pass it:

node crawl.mjs --url brand.uber.com --cookie "frontify-session-id=…; other=value"

Or pass each pair separately (they are merged into one header):

node crawl.mjs --url brand.sbb.ch --hub 2 \
  -c 'frontify-session-id=…' \
  -c 'terms_accepted=…' \
  -c 'pa_privacy="optin"'

Or add to .env:

COOKIE=frontify-session-id=…; terms_accepted=…

Note: Node’s CLI used to keep only the last -c if you repeated --cookie; use multiple -c flags (supported now) or a single semicolon-separated string.

See .env.example for reference.