@growth-labs/monitoring

Alert delivery is durable as of 0.2.0 and WordPress Fleet Registry contracts ship as of 0.3.0. Consumers must apply the package migrations before deploying: incident actions are written to gl_alert_deliveries, transport failures remain retryable, monitor cycles can write gl_monitor_heartbeats, and fleet consumers can materialize revision/site/environment/tombstone/receipt state.

Failed deliveries remain pending until a transport confirms receipt. Retry delays use exponential backoff capped at 15 minutes; there is intentionally no terminal attempt count that could silently discard an undelivered page. An independent dead-man should page on an aged pending-delivery backlog.

Delivery is at-least-once. Each attempt holds a bounded D1 lease and may update its receipt only while it still owns that exact lease. Alert content and transport errors are redacted and truncated before persistence; consumers should still avoid putting customer data in alert messages.

As of 0.3.1, consumers that own a narrow alert action can use flushAlertDelivery() with its exact dedupKey; it claims only that action's due rows and does not reconcile or enqueue unrelated open incidents. Read the Pushover row with getAlertDelivery() and require status === 'sent' before recording a successful monitor heartbeat. Apply 0007_open_incident_dedup.sql before deploying code that can overlap: it enforces one open incident per surface.

Operational observability for Growth Labs Cloudflare Workers: synthetic probes, authenticated browser canary primitives, browser error capture, uptime/error schemas, alerting, and a small public status page.

Install

pnpm add @growth-labs/monitoring @growth-labs/notify @growth-labs/analytics

Peer: astro ^6.0.0 for the status-page app. Runtime deps: @growth-labs/notify, @growth-labs/analytics, drizzle-orm, zod.

Modules

| Subpath | Purpose | | --- | --- | | @growth-labs/monitoring | Umbrella exports for the main factories. | | @growth-labs/monitoring/canary | Authenticated browser canary schemas, HTTP asset runner, browser error normalization, privacy-safe beacon/persistence helpers, and synthetic traffic naming. | | @growth-labs/monitoring/prober | createProber(config) for Cron Workers that run GET, POST, and happy-path checks, including the strict result-bearing managed-fleet path. | | @growth-labs/monitoring/site-uptime | createSiteUptimeProber(config) — per-apex homepage + canary-article + sitemap uptime checks with a FulcrumInternal UA and a 2-fail/2-success critical incident lifecycle. | | @growth-labs/monitoring/freshness | createFreshnessProbe(config) — per-table MAX(updated_at/created_at) age checks that alert when pipeline-fed analytics tables go stale. | | @growth-labs/monitoring/coordinator-receipt | Bounded duplicate-aware JSON byte parsers, strict EdDSA JWS receipt/storage schemas, cross-family exact-head evidence, post-migration live-arrival ordering, and fenced watchdog/daily-digest claim and release commands for Hermes coordinator health. | | @growth-labs/monitoring/fleet-registry | fleetRevisionSchema, typed WordPress fleet revision/site/environment/tombstone/receipt schemas, strict wordpress-host-v1 host projection schemas, and validateFleetRevision(). | | @growth-labs/monitoring/tail | createTailWorker(config) for Cloudflare Tail Worker trace batches. | | @growth-labs/monitoring/alerting | Threshold, dedup, incident, escalation, and notify delegation helpers. | | @growth-labs/monitoring/status-page | Astro integration that injects the status page and JSON route. | | @growth-labs/monitoring/schemas | Drizzle schema matching the shipped D1 migrations, including fleet registry tables. |

Coordinator watchdog notification builders require nonblank content both before and after redaction, so they cannot persist a notification watermark without a drainable outbox row. Daily-digest finalization redacts delivery errors before binding them for durable storage, and runtime watchdog row gates accept only states reachable through the fenced database transitions.

Raw SQL migrations ship at:

node_modules/@growth-labs/monitoring/src/schemas/migrations
node_modules/@growth-labs/monitoring/dist/schemas/migrations

WordPress Fleet Registry

import {
	fleetRevisionSchema,
	validateFleetRevision,
} from '@growth-labs/monitoring/fleet-registry'

const candidate = fleetRevisionSchema.parse(rawRevision)
const result = validateFleetRevision(candidate, previousActiveRevision)

if (!result.ok) {
	// Keep the last-known-good revision active and alert on result.errors.
}

const monitoringTargets = result.ok ? result.effectiveDenominators.monitoring : []
const backupTargets = result.ok ? result.effectiveDenominators.backups : []

FleetRevision contains revisionId, createdAt, declared/materialized denominators, sites, tombstones, and signature metadata that points at the registry artifact plus a detached signature URI. The only approved signature algorithm is ed25519; the signed payload model is fleet-registry-artifact-sha256-v1, and its artifact hash must match the registry artifact hash.

validateFleetRevision() performs structural validation only. It does not cryptographically verify Ed25519 signatures or exact artifact bytes. Promoting consumers must verify the registry signature, logical-backup receipts, PBS-snapshot receipts, and staging scrub receipts against trusted keys and the exact artifact bytes before promotion. The package never accepts signing secrets, host credentials, key material, signed URLs, or provisioning credentials. Artifact pointer fields are bounded, must use an approved HTTPS, R2, or PBS URI with an authority, must omit fragments, and must remain credential-free. SHA-256 digests use canonical lowercase hexadecimal, and every detached-signature pointer must be distinct from its artifact pointer.

Each environment identifies production or staging, canonical host, node/LXC desired state, PHP policy, backup cadence, probe policy, analytics store ids, writable identity ids, desired Growth Labs WordPress platform release, and a strict hostProjection. Canonical hosts must be normalized bare lowercase hostnames with no scheme, path, port, trailing dot, or formatting alias. Writable filesystem, database, object/page cache, secret-set, cache namespace, logical backup namespace, and PBS namespace ids must be unique across the whole fleet.

The wordpress-host-v1 projection is source-only desired state plus non-secret pointers. It requires an anchored Debian 13 standard template name, the unprivileged/no-nesting/no-device container policy, Proxmox CT ids 100..999999999, explicit cluster/node/network/exposed-SSH scopes, normalized RFC1918 or ULA private address and distinct gateway in the same usable subnet, SSH ports 20000..60999, at least 2 cores / 2048 MiB RAM / 20 GiB disk, integer disk and task values, and swapMb <= memoryMb. Base environment authorities for LXC id, node, resource profile, backup cadence, platform release, and canonical environment identity must exactly match the host projection.

Staging desired reachability is always private and scrubGated: true. Registry authors cannot self-assert scrub-cleared. Observed Cloudflare Access or scoped-SSH eligibility is accepted only when a separately verified scrub receipt is current, successful, exit code 0, bound to the staging source environment, current source restore id, scrub profile, and validation clock. Fleet review policy requires reviewRequired: true.

validateFleetRevision() captures the production validation clock internally; callers cannot supply or backdate it. Deterministic tests freeze the system clock. The validator rejects semantic candidates that must not replace last-known-good: empty site or environment denominators, sites with no environments, duplicate site/environment ids, duplicate canonical hosts across all environments, shared writable or backup namespaces, unsafe host projections, unexplained shrink, phantom or duplicate environment tombstones, future removal times, grace shorter than 48 hours, invalid finalization receipt identity/timing, shortened carry-forward grace, expired or overlong PHP fallback exceptions, unsupported or malformed PHP policies, ownerless PHP fallbacks, non-canonical probe policies, signature algorithm/binding failures, and declared/materialized count mismatches.

Successful validation keeps memberIds as the current active site ids and also returns effectiveDenominators.monitoring and effectiveDenominators.backups. Those denominator entries are environment-scoped and include active environments plus every tombstone still present in the revision. graceUntil is the earliest permitted finalization time, not an expiry: time alone never stops monitoring or backups and never permits a tombstone to disappear. Grace must be at least 48 hours; defaultFleetRemovalGraceUntil() returns a seven-day window. Every tombstone must include both environmentId and environment. A whole-site removal is represented as one tombstone per removed environment, and active tombstones carry forward with the same id, environment identity, removedAt, and non-shortened grace. A later revision may omit the tombstone only at or after graceUntil and only when the prior tombstone contains both a successful, identity-matching, post-removal logical-backup receipt and PBS snapshot receipt. Until then, its monitoring and backup flags must remain true. The managed-host probe policy is exact: customer-path cadence is 5 minutes, deep WordPress cadence is 10 minutes, and both originBypass and siteAgent must be true. Canonical PHP policy is exactly { version: '8.5' } with no fallback fields. PHP 8.3 and 8.4 are the only fallbacks; each requires a site owner, a non-empty reason of at most 240 characters, and a positive integer expiry in the future no more than 90 days from validation time.

The Fleet Registry records desired monitoring policy. This package does not prove that alerting is active, scheduled, or delivering notifications; that evidence belongs to the monitoring runtime and its consumer deployment. No alert-proof field is accepted in the registry contract.

Prober

import { createProber } from '@growth-labs/monitoring/prober'

const prober = createProber({
	realmId: 'growth-labs',
	d1Binding: 'MONITORING_DB',
	notifyConfig: {
		channels: ['slack', 'email'],
		emailProvider: 'resend',
	},
	alertingConfig: { consecutiveFailuresToOpen: 2, minSeverity: 'warning' },
	surfaces: [
		{
			name: 'bodybuildingwithabrain.com:customer-path',
			kind: 'get',
			schedule: '*/5 * * * *',
			url: 'https://bodybuildingwithabrain.com/',
			assertions: { statusCode: 200, contentMarker: 'Bodybuilding With The Brain' },
			timeoutMs: 10_000,
		},
	],
})

export default { scheduled: prober.scheduledHandler }

The legacy scheduled path remains best-effort. The fleet-wide managed WordPress monitor uses the additive result-bearing path so it can inspect the first result and make exactly one 55–75-second confirmation decision:

import {
	createProber,
	managedFleetProbeContextSchema,
} from '@growth-labs/monitoring/prober'

const context = managedFleetProbeContextSchema.parse({
	runId: crypto.randomUUID(),
	site: 'bodybuildingwithabrain',
	environment: 'production',
	surfaceKind: 'customer-path',
	trafficClass: 'synthetic',
})

const result = await prober.checkManagedFleetSurface(
	'bodybuildingwithabrain.com:customer-path',
	env,
	context,
)

if (result.status !== 'pass') {
	// The central monitoring consumer may wait 55–75 seconds, create a new
	// unique run id, and invoke this same package path exactly once.
}

checkManagedFleetSurface() applies the package-owned GrowthLabs-Monitor/0.3.5 user agent and exact canonical X-Fulcrum-* operational headers. Its first argument is a configured surface name; the name must resolve to exactly one surface in createProber(config), so callers cannot substitute an executable surface object. Callers also cannot add per-run headers, credentials, arbitrary metadata, or a user-agent override. Its single D1 insert includes site, environment, surface_kind, traffic_class, and run_id; the returned result carries the same run id. Ordinary HTTP/assertion failures return a persisted fail/timeout. Missing D1, invalid input, execution, persistence, or alert-infrastructure failure rejects with a bounded ManagedFleetProbeError instead of returning an apparently usable in-memory result. The package still owns runners, persistence, incident thresholds, and durable alert delivery; the consumer owns only cadence and the one confirmation wait.

Managed surfaces use credential-free HTTPS URLs and reject secret-shaped query keys, static headers, and static POST bodies. Every code-signin URL must be the account host or one of its subdomains; redirect-derived callbacks must use the exact configured callback origin. Session cookies follow host-only or Domain, Path, and Secure scope before they are replayed. Non-timeout runtime, adapter, Gmail, and body-read exceptions reject as a redacted EXECUTION_FAILED; legacy scheduledHandler and checkSurface() execution remains best-effort.

auth-monitor must use emailProvider: 'resend' when email alerts are enabled. The monitor exists partly to detect Cloudflare auth delivery failures, so alert email must not depend on Cloudflare Email.

Tail Worker

import { createTailWorker } from '@growth-labs/monitoring/tail'

const tail = createTailWorker({
	realmId: 'fulcrum-labs',
	d1Binding: 'MONITORING_DB',
	waeBinding: 'MONITORING_ERRORS',
	notifyConfig: { channels: ['slack', 'email'], emailProvider: 'resend' },
	alertingConfig: {
		newErrorDedupWindowMs: 60 * 60 * 1000,
		rateSpikeThreshold: 10,
		rateSpikeWindowMs: 5 * 60 * 1000,
	},
	sampling: {
		exceptionsPct: 1,
		fivexxPct: 1,
		consoleErrorPct: 1,
		consoleWarnPct: 0.1,
		slowRequestsPct: 0.01,
		slowRequestThresholdMs: 1000,
	},
})

export default { tail: tail.tailHandler }

Fingerprints are computed before redaction. Every persisted error is redacted and truncated before D1/WAE writes.

Status Page

import { createStatusPageApp } from '@growth-labs/monitoring/status-page'

export default {
	integrations: [
		createStatusPageApp({
			realm: 'fulcrum-labs',
			d1Binding: 'MONITORING_DB',
			surfaces: [{ name: 'fronts.co:login' }],
		}),
	],
}

The status page is read-only. It injects / and /api/status.json by default and reads only the gl_uptime_checks, gl_uptime_incidents, and gl_errors tables.

Authenticated Browser Canary Primitives

import {
	createSyntheticTrafficIdentity,
	evaluateCanaryRun,
	videoPlaybackSurfaceSchema,
} from '@growth-labs/monitoring/canary'

const surface = videoPlaybackSurfaceSchema.parse({
	name: 'fronts:subscriber-video',
	surface: 'media.video-playback',
	runner: 'browser.playwright',
	site: 'fronts',
	environment: 'production',
	role: 'subscriber',
	targetUrl: 'https://fronts.co/video/golden-story',
	firstPartyHosts: ['fronts.co'],
	credentialBinding: 'SUBSCRIBER_CANARY_CREDENTIALS',
	selectors: {
		player: '[data-test=vidstack-player]',
		blockedMarkers: ['[data-test=paywall]'],
	},
})

const traffic = createSyntheticTrafficIdentity({
	tool: 'authenticated-browser-canary',
	version: '1.0.0',
	realm: 'fulcrum-labs',
	site: 'fronts',
	environment: 'production',
	surface: surface.surface,
	runId: crypto.randomUUID(),
})

const result = await evaluateCanaryRun(surface, {
	runId: traffic.runId,
	startedAt: Date.now(),
	finishedAt: Date.now() + 4_000,
	userAgent: traffic.userAgent,
	statusCode: 200,
	finalUrl: surface.targetUrl,
	matchedMarkers: ['[data-test=vidstack-player]'],
	blockedMarkersSeen: [],
	playerMounted: true,
	currentTimeSeconds: 3.4,
	tracks: [],
	errors: [],
	artifacts: [],
})

Package-owned scope:

• surface schemas, assertion/result evaluation, privacy-safe beacon defaults, browser error normalization, and D1 migration columns
• reusable asset.http runner plus runner interfaces for browser.puppeteer and browser.playwright

Runtime-owned scope:

• cron cadence, Browser Rendering/Playwright adapter wiring, credential provisioning, artifact storage, and alert/incident policy configuration

Canary surfaces take binding names, not secrets. Pass values like SUBSCRIBER_CANARY_CREDENTIALS or PUBLISHER_CANARY_CREDENTIALS in runtime config and let the consumer Worker resolve them. Never inline cookies, Authorization headers, magic-link URLs, session tokens, raw email contents, or full page HTML in package config or persisted artifacts.

Browser error beacons default to sampleRate: 0.1 and maxEventsPerMinute: 5 per page context. Both synthetic and real-user browser events flow through the same normalization and redaction path before persistence.

Site Agent contracts

• WordPress schema v1 is pinned to producer 2fc785dda227f694137635dd8544128473610525, including expanded object-cache fields.
• Cache-runtime v2 is live; v1 exists only for queued replay and never asserts object-domain health.
• Environment service, integrity, storage, backup, and restore facts are bounded operational claims with explicit unavailable/error states.
• The envelope signs the exact documented ASCII field sequence plus SHA-256 of exact decoded facts bytes.
• Facts contain no site/environment/revision identity; the authoritative envelope supplies it from Fleet Registry.
• Fleet Registry holds public keys (never private keys) plus the declared backup/restore policy; one active and one optional retiring key, with fleet uniqueness and overlap rotation.
• Fleet revisions declare exact signed logical/PBS retention and restore-drill profiles; policy ids bind immutable canonical bodies.
• Backup/restore receipt admission resolves the envelope's original signed promoted revision, independently verifies R2/PBS protection, and tombstone finalization requires monitoring-platform's admitted policy-compliant receipt ledger — a structurally succeeded receipt never finalizes.
• Envelope verification resolves identity/revision/release and present-day retirement/revocation through monitoring-platform's explicit trusted accessor over its cryptographically verified promoted trust ledger. A structural Fleet validation alone cannot mint trust; version skew and key material are never accepted as independently asserted values.
• Live telemetry has a 15-minute freshness window. Authenticated durable replay has a seven-day ceiling, returns a delayed-only admission class, and can never update current health or incident state; expired FIFO records are terminally dead-lettered and paged by the host consumer.
• The package verifies but does not sign, transport, persist replay state, create incidents, schedule work, or deliver notifications.
• Monitoring-platform #72 must atomically claim replayFingerprint and owns D1/admission/incident/receipt behavior.
• Infra #53 must replace its stale WordPress cache field set and reconcile its provisional validators/fixtures before activation.
• WordPress-cache #6 remains the producer; changes are upstream releases, never per-site patches.