@smart-cloud/publisher-exporter

@smart-cloud/publisher-exporter is the standalone Node.js CLI package behind WP Suite Static Publisher crawl, deploy, invalidate, and queue-runner workflows.

It is designed to run outside WordPress. The WordPress plugin only manages runtime config, queue state, and logs.

Install

Global install:

npm install -g @smart-cloud/publisher-exporter
publisher-exporter install-browsers

Without a global install:

npx @smart-cloud/publisher-exporter install-browsers

If multiple users may run jobs on the same host, prefer a shared Playwright browsers path:

export PLAYWRIGHT_BROWSERS_PATH=/var/lib/playwright-browsers
publisher-exporter install-browsers

If PLAYWRIGHT_BROWSERS_PATH points to a shared system directory, create that directory first and make it writable by the same OS user that will run publisher-exporter install-browsers. A one-time elevated setup step to create or re-own the directory is fine. The later cron job does not need elevated privileges to use an already installed shared browser cache, but it does need read and execute access to that directory tree.

Commands

• Normal usage:

PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl
PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl --crawl-mode incremental
PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl --resume-rewrite
PUBLISHER_CONFIG=./publisher.config.json publisher-exporter crawl --retry-timeouts
PUBLISHER_CONFIG=./publisher.config.json publisher-exporter deploy
PUBLISHER_CONFIG=./publisher.config.json publisher-exporter invalidate
publisher-exporter publish
publisher-exporter queue-runner --runtime-dir /srv/site/runtime --max-jobs 1

Equivalent npx usage:

PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter crawl
PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter crawl --crawl-mode incremental
PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter crawl --resume-rewrite
PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter deploy
PUBLISHER_CONFIG=./publisher.config.json npx @smart-cloud/publisher-exporter invalidate
npx @smart-cloud/publisher-exporter queue-runner --runtime-dir /srv/site/runtime --max-jobs 1

Crawl Modes And Repair Workflows

• crawl without extra flags performs a normal full crawl, asset download, and final text rewrite.
• crawl --crawl-mode incremental reuses the existing crawl manifest when possible and rewrites only the files affected by re-crawled pages, text assets, or changed asset-map entries.
• An incremental crawl now fails closed when the remote WP Suite subscription configuration cannot be loaded. It never silently falls back to a full crawl that clears the existing output tree.
• crawl --resume-rewrite skips discovery, rendering, and asset download, then reruns the final text rewrite over the existing output tree using the current rewrite rules.
• crawl --retry-timeouts retries timed-out URLs from the latest archived full crawl or publish log snapshot.

Use --resume-rewrite when the already exported output is otherwise valid but the rewrite logic changed or a previous crawl stopped during the rewrite phase. This is the fastest repair path for stale output caused by rewrite-only bugs such as escaped JSON replacement issues, protocol-relative asset URLs like //host/path.css, or an interrupted final rewrite.

If you need to repair existing exported HTML after a rewrite bug, prefer --resume-rewrite over a new incremental crawl. Incremental crawl only rewrites targeted files; --resume-rewrite reprocesses every text file in the current output.

During --resume-rewrite, the terminal can be quieter than a full crawl. Progress still updates in runtime/current-progress.json and the live crawl event snapshot under the configured log directory.

Crawl integrity and deployment safety

Every crawl creates an incomplete-crawl marker before it changes the output. The marker is removed only after final rewrite and deploy-plan generation complete. deploy refuses to upload while that marker exists, so stopping a crawl cannot fall back to scanning and uploading a partial or unre-written output tree.

If a crawl was stopped during final rewrite, run crawl --resume-rewrite; for an interrupted crawl at an earlier stage, run a new crawl. Do not delete the marker manually to bypass this protection.

Direct CLI With Runtime-Managed Config

When the WordPress plugin already wrote runtime/config.json into shared publisher storage, direct CLI invocation still needs both the config path and the runtime directory.

Example:

STATIC_PUBLISHER_RUNTIME_DIR=/mnt/site/runtime \
PUBLISHER_CONFIG=/mnt/site/runtime/config.json \
publisher-exporter crawl --resume-rewrite

PUBLISHER_CONFIG selects the main exporter config file. STATIC_PUBLISHER_RUNTIME_DIR anchors runtime artifacts such as current-progress.json, crawl-manifest.json, queue files, and storage-relative output/log paths.

Queue Runner

The queue runner reads the runtime JSON files generated by the WordPress plugin.

Example:

publisher-exporter queue-runner \
  --runtime-dir /var/www/site/wp-content/uploads/smartcloud-static-publisher/runtime \
  --max-jobs 1

Direct CLI invocation from cron is the recommended setup.

When the runtime config contains enabled scheduler rules and the active queue-runner policy allows scheduler auto-enqueue, queue-runner evaluates those rules once at startup before draining queued jobs.

• Scheduler rules are read from runtime/config.json and their last interval buckets are persisted in runtime/scheduler-state.json.
• Scheduler only auto-enqueues jobs into the runtime queue. It does not replace cron, systemd timers, or Windows Task Scheduler.
• A 1-minute external runner tick is the recommended cadence.
• Supported scheduled commands are publish, crawl, deploy, invalidate, retry-timeouts, and url.
• The scheduler timezone value is currently informational for operations context; interval matching is based on elapsed minute buckets checked at each queue-runner start.
• If an equivalent queued or running job already exists for the same command, crawl mode, deployment profile, and URL, that rule is skipped for the current interval bucket.

Same-host Linux cron example:

SHELL=/bin/bash
HOME=/home/<runner-user>
PATH=/home/<runner-user>/.nvm/versions/node/v24.15.0/bin:/usr/bin:/bin
PLAYWRIGHT_BROWSERS_PATH=/var/lib/playwright-browsers
RUNTIME_PATH=/var/www/site/wp-content/uploads/smartcloud-static-publisher/runtime
LOG_PATH=/var/www/site/wp-content/uploads/smartcloud-static-publisher/logs

* * * * * /usr/bin/flock -n /tmp/static-publisher.cron.lock publisher-exporter queue-runner --runtime-dir "$RUNTIME_PATH" --max-jobs 1 >> "$LOG_PATH/queue-runner-cron.log" 2>&1
17 3 * * * publisher-exporter prune-logs --runtime-dir "$RUNTIME_PATH" --older-than-days 30 >> "$LOG_PATH/prune-logs-cron.log" 2>&1

If WordPress and the runner are on different machines but share the same mounted publisher storage, keep outputDir and logDir storage-relative in WordPress admin and point cron at the crawler host's local mount path:

SHELL=/bin/bash
HOME=/home/<runner-user>
PATH=/home/<runner-user>/.nvm/versions/node/v24.15.0/bin:/usr/bin:/bin
PLAYWRIGHT_BROWSERS_PATH=/var/lib/playwright-browsers
RUNTIME_PATH=/mnt/site/runtime
LOG_PATH=/mnt/site/logs

* * * * * /usr/bin/flock -n /tmp/static-publisher.cron.lock publisher-exporter queue-runner --runtime-dir "$RUNTIME_PATH" --max-jobs 1 >> "$LOG_PATH/queue-runner-cron.log" 2>&1

If you do not want a version-pinned NVM path in crontab, create a stable user launcher in ~/bin and put that directory first in PATH:

mkdir -p "$HOME/bin"
cat > "$HOME/bin/publisher-exporter" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
export HOME=/home/<runner-user>
export NVM_DIR="$HOME/.nvm"
. "$NVM_DIR/nvm.sh"
nvm use default >/dev/null
exec "$(npm prefix -g)/bin/publisher-exporter" "$@"
EOF
chmod +x "$HOME/bin/publisher-exporter"

If cron already sets the correct HOME, remove the explicit export HOME=... line from the wrapper. A plain symlink to ~/.nvm/versions/node/vX.Y.Z/bin/publisher-exporter will break after upgrades. Prefer this small launcher, or enable an NVM-managed current symlink and link against that stable path.

After each finished, failed, or stopped job, queue-runner copies the exporter-generated working logs into "<logDir>/archive/<timestamp-command-jobId-status>/" and includes a job.json summary plus the latest current-progress.json snapshot when available. The live root log files remain the current working set and may be overwritten by the next job.

Archived files are gzip-compressed per file and listed in job.json. WordPress Audit Log rows can download those archived artifacts through the authenticated REST API.

retry-timeouts uses the newest archived full crawl or publish job log snapshot as its timeout source. It prefers the manifest-backed archived errors.* artifact when available and falls back to older uncompressed archive layouts.

Prune old archive directories with publisher-exporter prune-logs --runtime-dir /srv/site/runtime --older-than-days 30. Add that command to daily cron or another retention scheduler that matches your log policy.

Troubleshooting

• If exported HTML still contains old rewritten URLs after a rewrite fix, run crawl --resume-rewrite. This reprocesses every text file in the current output tree and does not depend on earlier rewriteComplete: true entries in crawl-manifest.json.
• If --resume-rewrite appears to stop after the initial resume message, check runtime/current-progress.json and <logDir>/current-crawl-event.json before assuming it is stuck. That phase can stay mostly quiet on stdout while it rewrites files.
• If direct CLI execution seems to ignore the expected runtime state, verify that PUBLISHER_CONFIG and STATIC_PUBLISHER_RUNTIME_DIR both point at the same runtime tree. A mismatched config path and runtime dir can make the run look idle or incomplete.
• Prefer incremental crawl when the existing manifest is trusted and you only need a fast recrawl of changed pages or assets. Prefer a full crawl when discovery rules changed, the manifest is missing or suspect, or you want a clean end-to-end rebuild.
• JavaScript bundle discovery follows only explicit dynamic import() calls and fixed Webpack chunk-filename mappings such as runtime.u = () => "custom-block-parser.js". It does not infer dependencies from arbitrary quoted paths or application-specific loader calls; add those manually as extra assets when needed.
• In sdk-upload-delete mode, remote-only S3 objects are protected unless the crawl deploy plan contains an explicit manifest-backed deletion tombstone. Protected keys are listed in deploy-diff.json and the deploy log. This intentionally favors availability over automatically removing an unverified remote-only object.
• queue-runner-heartbeat.json reports both the exporter package version and the installed @smart-cloud/wpsuite-core version. Crawl logs also report the runtime virtualAssetBaseUrl, remote config loader status, and resolved subscription type without logging the runtime token. Legacy runtime input using uploadUrl is accepted during migration but normalized to virtualAssetBaseUrl.
• For generated 404 capture, set generated404RequestPath to a path that really returns HTTP 404 from the origin. The exporter validates that response before reusing it as the generated 404 page.

WordPress Integration

The companion WordPress plugin can optionally store an External exporter dir setting. Point it at this package root when you want PHP-side diagnostics to verify the local CLI install.

Typical values:

• /usr/local/lib/node_modules/@smart-cloud/publisher-exporter
• /opt/smartcloud/publisher-exporter/node_modules/@smart-cloud/publisher-exporter

Notes

• Playwright browser binaries are separate from the npm package and may need reinstalling after Playwright version upgrades.
• For internal/self-signed TLS origins, set ignoreHttpsErrors in publisher.config.json or through the WordPress admin UI.
• The package expects Node.js 20 or newer.