@qemuguest/lake

First-party analytics that lives in your own Cloudflare account: event and session-recording ingest, a structured query engine over Workers Analytics Engine and R2, and a Grafana-style React dashboard. Extracted from finterm.xyz.

npm i @qemuguest/lake

| Entry point | Runs in | What it gives you | |---|---|---| | @qemuguest/lake | Worker | createLake(): ingest + read API as one fetch handler, plus the query engine | | @qemuguest/lake/client | Browser | createTracker(): events, heartbeat, session recorder | | @qemuguest/lake/dashboard | Browser (React 18+) | <LakeDashboard>, <Explore>, datasources, shipped dashboards |

1. Bindings

// wrangler.jsonc
"analytics_engine_datasets": [{ "binding": "ANALYTICS", "dataset": "my_events" }],
"r2_buckets": [{ "binding": "RECORDINGS", "bucket_name": "my-recordings" }]

Secrets (wrangler secret put): CF_ACCOUNT_ID, CF_ANALYTICS_TOKEN (an API token with Account Analytics: Read), and LAKE_KEY (any long random string, for dashboard reads).

2. Mount the lake

import { createLake, keyAuth } from '@qemuguest/lake';

export default {
  async fetch(req: Request, env: Env) {
    const lake = createLake({
      basePath: '/lake',                       // default
      authorize: keyAuth(env.LAKE_KEY),        // gates /query and /sql; ingest is public
      analytics: env.ANALYTICS,
      recordings: env.RECORDINGS,
      ae: { accountId: env.CF_ACCOUNT_ID, apiToken: env.CF_ANALYTICS_TOKEN, dataset: 'my_events' },
      ingest: {
        identify: async (req) => '',           // your app's user id, '' when anonymous
        exclude: (req) => false,               // e.g. staff cookie
        beforeIngest: async (req) => null,     // return a Response (429) to rate-limit
        redact: (s) => s.replace(/([?&]token=)[^&]*/g, '$1'),
        untrackedPaths: ['/admin'],
        cookieDomain: '.example.com',          // share the visitor id across subdomains
      },
    });
    return (await lake.fetch(req)) ?? yourApp(req, env);
  },
};

lake.fetch returns null for paths outside basePath, so it composes with any router. It only uses Request/Response, so it also works inside Next.js route handlers, Hono, etc.

| Route | Auth | | |---|---|---| | POST /lake/event | public | one Analytics Engine row per event | | POST /lake/rec | public | one immutable R2 object per recorder batch | | GET /lake/v?p=/path | public | page-load pixel, works without JS | | GET /lake/query?op=describe | authorize | field registry + live-discovered event types and props.* keys | | GET /lake/query?op=query&table=… | authorize | structured query (below) | | GET /lake/query?catalog=sessions&scope=… | authorize | days, sessions, session, summary, clicks (session browser) | | GET /lake/sql?q=SELECT… | authorize | raw Analytics Engine SQL; $__dataset expands to your dataset |

3. Track

import { createTracker } from '@qemuguest/lake/client';

const lake = createTracker({
  endpoint: '/lake',
  untrackedPaths: ['/admin'],
  consent: () => localStorage.getItem('analytics') !== 'off',
});

lake.track('page_view');
lake.track('signup', { plan: 'pro' });     // props are queryable as props.plan
const stopBeat = lake.startHeartbeat();    // 60s, while visible; marks input as active:1
const stopRec = lake.startRecorder();      // pointer trail, clicks, keys (no values), scrolls, navs

Nothing is sent from iframes, untracked paths, or when consent returns false. Every call is a no-op outside a browser and never throws.

4. Look

import {
  LakeDashboard, Explore, sqlDashboard,
  registerBuiltinDataSources, registerBuiltinTransforms,
} from '@qemuguest/lake/dashboard';

registerBuiltinDataSources({ baseUrl: '/lake', headers: () => ({ Authorization: `Bearer ${key}` }) });
registerBuiltinTransforms();

<LakeDashboard dashboard={sqlDashboard} exploreHref="/analytics/explore" />
<Explore dashboard={sqlDashboard} storageKey="lake.dashboard.lake-sql" dashboardHref="/analytics" />

Two dashboards ship: sqlDashboard (raw Analytics Engine SQL: DAU/WAU/MAU, retention cohorts, pixel coverage, sessions) and lakeDashboard (the structured query language only, so it also runs on the in-memory stores). Dashboards are plain JSON (DashboardModel); edits are saved per browser under storageKey, and bumping version merges new shipped panels into saved copies.

Datasources: lake-sql, lake, lake-sessions, static-json. Add your own with registerDataSource({ id, name, query }) and pivots with registerTransform(name, fn).

One hub, many apps

Run one lake (the hub) and feed it from every app you have. Each app is a source and every row carries its source id (source in queries, blob19 in raw SQL); the dashboard gets a source picker.

// hub worker
createLake({
  authorize: keyAuth(env.LAKE_KEY),
  analytics: env.ANALYTICS, recordings: env.RECORDINGS, ae: { /* … */ },
  sources: {
    finterm: { key: env.FINTERM_KEY, cookieName: '_fid', cookieDomain: '.finterm.xyz' },
    acme:    { origins: ['https://app.acme.com'] },
  },
});

An app sends data one of two ways:

• Server forwarder (recommended). The app proxies its own /lake/* to the hub with forwardToLake, authenticating with its source key. The visitor cookie stays first-party, ad-blockers see a same-origin request, and the app passes the visitor's real IP / country / ASN (x-lake-* headers, trusted only from keyed callers). Server-side events use sendLakeEvent.

  // in the app's Worker / route handler for /lake/*
  import { forwardToLake, sendLakeEvent } from '@qemuguest/lake';
  const hub = { hub: 'https://lake-hub.you.workers.dev', key: env.LAKE_SOURCE_KEY };
  return forwardToLake(req, '/event', { ...hub, cookieName: 'lake_id' });   // or '/rec', '/v'
  await sendLakeEvent(hub, { type: 'signup_completed', userId, visitorId, req });

  Only the visitor cookie is forwarded, never the app's session cookies, and read routes are never proxied.

• Direct from the browser. List the app's origin under origins and point the tracker at the hub: createTracker({ endpoint: 'https://lake-hub.you.workers.dev/lake', source: 'acme' }). It sends CORS simple requests and keeps its visitor id in localStorage, since third-party cookies are blocked.

Requests from anything else are refused. legacySource names the source that rows written before you configured sources belong to.

The query language

Panels and Explore store queries as lines: the table first, then key=value.

events
where=device:neq:bot
where=props.plan:eq:pro
group=country
bucket=1d
agg=uniq(identity):visitors,count:events,p95(vw)
sort=-visitors
limit=20
from=now-30d

• Tables: events (Analytics Engine), sessions, batches, actions (R2 recordings). op=describe lists every field, its type, ops and cost.
• Ops: eq neq contains ncontains starts ends in nin gt gte lt lte range empty nempty; in takes a,b,c, range takes lo..hi.
• Aggregates: count uniq sum avg min max p50 p95, each fn(field):alias.
• Time: from/to accept now-7d, now-3h, all, epoch ms/s, YYYY-MM-DD, ISO. bucket is 1h 1d 7d 30d (auto in panels follows the dashboard range).

Every response is { ok, frame: { columns, rows }, meta }. meta.plan says how it ran: sql (pushed down exactly), scan (newest 10,000 matching rows evaluated in the Worker, e.g. for props.*), listing or body (R2). meta.truncated and meta.reasons flag any cap that cut the result.

Event layout

Rows are written positionally; only ever append. index1 is the visitor id.

| Slot | Field | | Slot | Field | |---|---|---|---|---| | blob1 | type | | blob10 | document.referrer | | blob2 | path | | blob11 | user agent | | blob3 | user_id | | blob12 | device (desktop/mobile/tablet/bot) | | blob4 | country | | blob13 | sid (join key to recordings) | | blob5 | HTTP referer | | blob14 | lang | | blob6 | props JSON | | blob15 | tz | | blob7 | ip (PII) | | blob16 | props.type | | blob8 | ref (first-touch tag) | | blob17 | props.symbol / ticker / asset | | blob9 | anon_id | | blob18 | ASN | | | | | blob19 | source |

Doubles: 1, client time, viewport w, viewport h, devicePixelRatio. Recordings are stored as rec/<YYYY-MM-DD>/<sid>/<seq>-<ts>.jsonl with per-batch counts in R2 custom metadata, so the sessions table answers from listings without reading bodies.

Without Cloudflare

MemoryAnalytics and MemoryBucket implement the same interfaces in memory, for tests and local dev:

import { createLake, MemoryAnalytics, MemoryBucket } from '@qemuguest/lake';

const analytics = new MemoryAnalytics();
const lake = createLake({
  authorize: () => true,
  analytics,
  eventRows: () => analytics.rows(),   // queries always take the scan plan
  recordings: new MemoryBucket(),
});

Privacy notes

You are the data controller. The default pipeline stores IP address, user agent and coarse location; strip them in redact/beforeIngest or drop the columns from your queries if you don't need them, gate tracking on consent where the law requires it, and set an R2 lifecycle rule for recordings.

License

MIT