index-squarespace

Build, patch, and verify the LZString-compressed static-index file the Squarespace "Custom Filter" plugin's customIndexUrl option reads. If you've turned a Squarespace blog collection into a large filterable directory (see seed-squarespace for one way to seed one), the plugin's default behavior is to fetch every record through hundreds of paginated ?format=page-context requests on every visit, then try to cache the result in localStorage — slow, per-browser, and it silently gives up once the payload outgrows the ~5MB quota. Pointing customIndexUrl at one pre-built, LZString-compressed file turns that into a single cached fetch from Squarespace's own CDN.

Pairs with seed-squarespace: seed the collection, then index it.

Proven at ~14,800-record scale (a grant-recipient directory) where it cut cold-load time from 5–10 minutes to about 5 seconds.

Zero npm dependencies (LZString is vendored, see Credits), Node 18+, CLI only.

Why a separate tool

This is the 5th stage of a larger CSV-to-Squarespace-directory pipeline (raw CSV → sort-large-csv → offline-csv-geocoder → csv-grep-rinse → seed-squarespace → this), but it's a distinct concern: the earlier tools run once, locally, against CSV-derived text to create content. This one runs repeatedly against a live site to reflect content that's already there — it harvests, it doesn't emit anything for import. It also has nothing WordPress- or WXR-specific about it; any Squarespace directory built any other way still needs this stage to get a fast, global static index.

The file format

A static-index file is plain text (conventionally saved with a .woff2 extension — see "Why .woff2?" below) consisting of:

LZString<compressed body>

where <compressed body> is LZString.compressToEncodedURIComponent(JSON.stringify(payload)) and payload is:

{
  "collection": { "id": "...", "typeName": "..." },
  "items": [
    { "id": "...", "title": "...", "fullUrl": "...", "urlId": "...", "publishOn": 0, "categories": [], "tags": [] }
  ],
  "html": "<article>...</article><article>...</article>..."
}

• items[] is a trimmed, arbitrary field set (defaults to the 7 fields shown — urlId is load-bearing if your plugin resolves detail-card matches against it; check your own plugin's slug-resolution logic before dropping it).
• html is every record's rendered card markup, concatenated into one string in the same order as items[].
• The exact field set, the collection id/typeName, and the html trims are all parameters — nothing here is specific to any one site or dataset. See build --help.

Why .woff2?

Squarespace's Custom Files uploader only accepts font/image file types. The index is plain text, not a font — but a .woff2 extension gets it through the uploader and, as a side effect, served with the CORS headers a cross-origin font request needs, which a plain data file wouldn't get. Your plugin reads it via fetch().then(r => r.text()), so the extension is cosmetic. If your setup doesn't have this constraint, any extension works.

Quick start

# 1. Get raw records into a harvest file somehow (see "Fetch / auth caveat")
#    -- or write a fetcher module and let `build` paginate for you.

# 2. Build the index
index-squarespace build \
  --harvest raw-harvest.json \
  --collection-id 695ef13009633d220bfc2ac9 \
  --collection-type blog-basic-grid \
  --collection-path partner-directory \
  -o index.woff2

# 3. Verify it
index-squarespace verify -i index.woff2 --checks my-checks.json

# 4. Upload index.woff2 to Squarespace Custom Files, paste the returned
#    static1.squarespace.com URL into your customIndexUrl header code.

# Later, a surgical fix without a full re-harvest:
index-squarespace patch -i index.woff2 -p my-patch.json -o index-v2.woff2 \
  --collection-path partner-directory

Commands

build

Harvests records, trims them to a fixed field set, applies safe html trims, and emits a compressed index.

index-squarespace build -o index.woff2 \
  --collection-id <id> --collection-type <typeName> \
  [--harvest raw.json | --fetcher fetch-page.mjs --source-url https://example.com/blog] \
  [--collection-path <path>] [--fields id,title,fullUrl,urlId,publishOn,categories,tags]

Full option list: index-squarespace build --help.

The inclusive cursor. If your records paginate by a timestamp-like field and any two records can share the exact same value (same-second publish timestamps are the classic case), a naive “strictly older than” cursor silently skips whichever record loses the tie at a page boundary — this happened in production at up to 19-way collisions on one dataset. build’s fetch loop instead advances the cursor with nextOffset = min(item[cursorField] on this page) + 1 (inclusive), which deliberately risks re-fetching a record it’s already seen, and then dedupes everything by --id-field, first occurrence wins. This is the one piece of domain logic in this tool that isn’t just “generic compression plumbing” — it’s the actual reason a hand-rolled version of this script existed in the first place.

The tie-group limit (and how build refuses to lose data at it). The inclusive cursor recovers a tie group only while that group is smaller than the page size — small enough that a single fetch reaches past it into older records, so the cursor floor keeps receding. A tie group as large as, or larger than, one page is a different problem: every fetch re-returns the same top rows of that one cursor value, all of them deduplicate, the floor never moves, and a naive loop would read the resulting all-duplicate page as “done” — silently dropping the rest of the tie group and everything older than it. This is not hypothetical for this pipeline: Squarespace caps a page at 20 records, and seed-squarespace bulk-imports with sequential per-second timestamps, so a single publish-second can easily exceed 20 records. Production’s worst tie was 19-way, one short of triggering it.

build detects the stuck state precisely — a full page whose items are all one cursor value and all already seen, i.e. a floor that has not receded past the previous offset boundary — and distinguishes it from a genuine end-of-data (a short page, or a page carrying more than one cursor value, cannot be hiding tied rows behind a boundary). On detecting it, build pages within the tie group by row offset, using an optional extension to the fetcher contract:

• If the fetcher is tieBreak-capable, build calls it with tieBreak = { cursorValue, skip }, the fetcher returns the records at exactly that cursor value skipping the first skip of them, and the group is drained fully before pagination resumes on strictly-older records. Recovery is transparent; all records are captured.

• If the fetcher is not tieBreak-capable (any existing fetcher that ignores the argument and doesn’t return tieBreakApplied: true), build aborts with a loud, actionable error — naming the cursor value and the tie-group size floor, and pointing at the two mitigations (supply a tieBreak-capable fetcher, or harvest the pages yourself and feed them to build --harvest). It will not finish a truncated index and pretend it’s complete.

As a belt-and-suspenders smoke signal, build also prints a warning whenever any full fetched page is entirely one cursor value — an early heads-up that you’re near or inside a tie group, even on the runs where recovery then succeeds silently.

Fetch / auth caveat

The live source for build is normally an authenticated browser session or at least a site-specific endpoint shape this tool can't assume. Two ways to feed it data:

1. --harvest <file.json> — hand it a pre-fetched file yourself, from wherever you can reach an authed session (a browser devtools snippet, a scripted browser session, an export). Shape: { "items": [...], "html": [...] } (a single page) or an array of such pages — build merges and dedupes across all of them. html, if present, must be an array parallel to items (same index = same record's rendered card), not one joined string.

2. --fetcher <module.mjs> --source-url <url> — write a small ESM module that fetches one page however your session requires (cookie header, whatever), and let build own the pagination loop, inclusive-cursor math, and dedup. Contract:

   export default async function fetchPage({ sourceUrl, offset, cursorField, pageNum, tieBreak }) {
     if (tieBreak) {
       // Optional. Page WITHIN one tie group by row offset: return the records
       // whose cursorField === tieBreak.cursorValue, in the same deterministic
       // order, skipping the first tieBreak.skip of them. MUST set
       // tieBreakApplied: true so build knows the request was honored.
       return { items: [...], html: [...], tieBreakApplied: true };
     }
     // Normal page: records with cursorField < offset, newest first
     // (or the newest page overall when offset is undefined). Deterministic.
     return { items: [...], html: [...] /* optional */, done: false };
   }

   The tieBreak branch is optional and backward compatible — a fetcher that omits it (any existing single-argument-object fetcher) keeps working unchanged, and build only ever calls with tieBreak when it hits an oversized tie group (see “The tie-group limit” above). Implement it only if your endpoint can page by row offset within a single cursor value; if it can’t, leave it out and build will abort loudly rather than truncate when it needs it.

   See examples/fetch-page-context.mjs for a worked example against Squarespace’s ?format=page-context endpoint, including where a session cookie plugs in, why that endpoint alone won’t give you rendered card html (it’s JSON-only; card markup is normally rendered client-side by your plugin/theme template), and why its timestamp-only offset genuinely cannot implement tieBreak — so an oversized single-second tie on that endpoint is a --harvest-it-yourself situation, not a bug. If you need pixel-identical pre-rendered cards in the index, scrape html from the live DOM separately and feed the result to --harvest instead.

Either way, compress/decompress/verify/patch are pure Node with no network or browser dependency — only build's live-fetch path touches the network, and only if you opt into --fetcher.

compress / decompress

index-squarespace compress -i payload.json -o index.woff2 [--prefix LZString]
index-squarespace decompress -i index.woff2 -o payload.json [--pretty]

Thin, generic wrappers around the LZString round-trip — useful standalone if you've assembled a payload some other way, or want to inspect/diff an existing index file.

verify

index-squarespace verify -i index.woff2 [--checks checks.json]

Always runs: items is a non-empty array, <article> count equals item count (tag name and field names configurable, checks individually skippable), no item has an empty title, no duplicate ids. Add --checks <file.json> for data-specific assertions — an array of:

{ "type": "...", "expect": <value | ">=N" | "<=N" | "!=N" | "==N">, "description": "optional" }

| type | fields | checks | |---|---|---| | itemCount | | items.length | | articleCount | articleTag | <tag> count (defaults expect to item count) | | noEmptyField | field, idField | items where field is empty | | noDuplicateIds | idField | duplicate id values | | htmlOccurrences | needle | substring count across html | | regexCount | pattern, flags | regex match count across html | | fieldValueCount | field, value | items whose (array or scalar) field includes/equals value | | itemFieldEquals | id, idField, field | one item's field against expect |

expect takes a bare value for exact equality, or an operator-prefixed number (">=214", "!=0") for a bound/regression check. See examples/sample-checks.json for a full example (item-count floor, a typo-regression guard, a curly-punctuation regression guard, a field-value-eliminated check, and a spot-check on one record). Exits 1 if anything fails.

patch

index-squarespace patch -i index.woff2 -p patch.json -o index-v2.woff2 --collection-path partner-directory

Applies a JSON list of per-record edits without a full re-harvest — the mechanism behind surgical index fixes. Every html edit is scoped to just that one record's <article>...</article> segment (located by href="/<collection-path>/<urlId>"), and every edit is fail-loud: if the expected old value isn't found exactly where it's supposed to be, patch throws rather than silently no-op'ing. That's deliberate — it's what keeps items[] and the pre-rendered html from quietly drifting apart across repeated patches. --dry-run reports what would change without writing.

Patch file — an array of entries:

[
  {
    "urlId": "2025-childrens-nutrition-peas",
    "items": [
      { "field": "title", "type": "set", "value": "PEAS" },
      { "field": "categories", "type": "replaceElement",
        "oldValue": "Pgr: Children's Nutrition",
        "newValue": "Pgr: Children’s Nutrition" }
    ],
    "html": [
      { "oldValue": "Peas", "newValue": "PEAS" }
    ]
  }
]

items[].type: set (assign any field directly), replaceElement (array field; replace the element that's === oldValue), replaceSubstring (array field; find the element that contains oldValue and do a string replace within it). html[] entries are scoped string replacements (oldValue → newValue, all occurrences within that record's segment); add "required": false to make one a no-op instead of a throw when absent. See examples/sample-patch.json.

Field-set and html-trim genericness — design notes

What's safe to generalize vs. what stays a per-site option:

• Item field set (--fields) — fully generic; defaults to the 7-field set (id,title,fullUrl,urlId,publishOn,categories,tags) that a Squarespace Custom Filter card needs, but any list works.
• Collection id/typeName (--collection-id, --collection-type) — fully generic, no defaults assumed.
• Cursor field (--cursor-field, default publishOn) and id field (--id-field, default id) — generic, but the defaults assume a Squarespace-shaped record; override for other sources.
• Whitespace collapsing — safe to always apply; html whitespace between tags is never meaningful. On by default, --no-collapse-whitespace to disable.
• Category-href stripping (--collection-path, on by default once that flag is set) — safe if your cards' category chips are display-only in the static index (which is the point of pre-rendering them — clicking one would need the plugin's live filter JS anyway, not a navigable link). Off unless you supply --collection-path, since without it there's nothing to scope the strip pattern to.
• The rendered card markup itself — deliberately not generalized. What an <article> card looks like is a function of your plugin/theme template, not something this tool should template-render. build treats card html as an opaque, pre-rendered string per record that you either scraped from the live DOM or supplied via --harvest.

Credits

LZString (lib/lz-string.min.js) is vendored, unmodified, from pieroxy/lz-string (WTFPL) — bundled so this tool needs zero npm dependencies.