@escapeaihq/client-core

Framework-agnostic Seer client building blocks. Vanilla TypeScript, no React or Vue dependencies. Imported by both seer-agent's React web/ app (this monorepo) and the Vue Escape web-app (sibling repo).

Module format

ESM-only. The published package.json points exports[".".import] at ./dist/index.js and exports[".".types] at ./dist/index.d.ts — there's no CJS require path. Both current consumers are ESM-default (Next.js 15 + React, Nuxt 3 + Vue + Vite), so this is fine. If a future CJS consumer (older Jest config, Node CLI script) appears, add a CJS bundle to the build (tsup or similar dual-output) before it bites.

What's in here

| Module | What it does | |---|---| | audioProtocol | Binary audio frame codec (1-byte tag + 320 samples PCM16); type defs for phoneme/word/barge-in messages | | audio | MicrophoneRecorder, AudioFramePlayer, PlaybackTracker, AudioSessionClient, AudioPipeline. Web Audio API based. Clean barge-in via GainNode + tracked activeSources. Sentence-boundary viseme scheduling. | | visemeScheduler | Maps Inworld TTS 1.5 phoneme timing to mouth shape changes via requestAnimationFrame, locked to AudioContext.currentTime. Stops the loop when idle. | | visemeGeometry | Pure path math: derive inner-lip donut, mouth clip ellipse, teeth/tongue geometry from existing mouth_paths data | | websocket | ChatWebSocket — auto-reconnecting wrapper around native WebSocket with text/binary multiplexing. sendPlaybackState(state: PlaybackState) publishes { action: 'playback_state', videoId, currentTime, shotId?, paused, clientTs } for cross-device video↔chat sync (src/websocket.ts:493 sendPlaybackState + src/websocket.ts:69 PlaybackState). Fire-and-forget: drops silently when the socket isn't OPEN. See specs/2026-05-17-cross-device-playback-sync.md. | | api | ApiClient REST client + all backend response/request type definitions. Constructor takes baseURL and chatBaseUrl from the caller (no env-var coupling). Image-tool wrappers since 0.3.0: extractImage, regenerateImage, pollVisemeBundleUntilDone. Bundle resolution since 0.5.0: loadBundleManifest (handles the inline-vs-S3 body_svg split internally) + canonical VisemeBundleManifest type. Reactor/Helios wrappers: getReactorToken, getSeedImageBlob, upsampleReactorPrompt. See CHANGELOG.md. | | reactor/ | Pure builders for the /interactive page in seer-agent (and reusable by the Vue Web-App). buildRegenPrompt(input) produces deterministic Gemini Flash Image prompts with locked 16:9 + identity guarantees plus a creative-direction escape hatch. buildUpsamplePrompt(input) returns {system, user} message blocks for any LLM transport, encoding the Helios prompt-guide rules and weaving in entity/persona/previous-prompt context. aggregateEntityContext and aggregatePersonaContext adapt our existing ImageRecord / EnrichmentRecord / PersonaConfig types into the canonical context shapes. Phase 2: mapEnrichmentToTimeline(enrichment, opts) consumes a EnrichmentRecord and returns a TimelineBeat[] (chunk-pinned prompts) for the /interactive/director timeline UI. Defensively probes multiple Pegasus result shapes. Phase 3: heliosTools.ts — HeliosCommand (discriminated union), parseHeliosCommands(text) (extracts [SET_SCENE: "…"]-style inline markers from streamed chat output), stripHeliosCommands(text) (removes markers from the displayed transcript), and the future-use BEDROCK_TOOL_SCHEMAS Converse-tool-spec export. toolPromptAddon.ts — HELIOS_STEERING_ADDON_ID, getHeliosSteeringAddonText(), idempotent buildHeliosSteeringSystemPrompt(base) for personas with prompt_addon_ids: ['helios-world-steering']. entitySearch.ts — findEntityImageBySeedName(apiClient, videoId, name) resolves an entity name to its ImageRecord (searches extracted then character, case-insensitive). Per-(videoId, category) in-process cache so repeat SWAP_SEED commands skip the API round-trip; forceRefresh: true and clearEntitySearchCache() bypass it. heliosCommandDispatcher.ts — dispatchHeliosCommand(cmd, ctx) executes a HeliosCommand against a structural Reactor bridge (sendCommand + uploadFile); pure async function, no React. HeliosToolContext.signal?: AbortSignal cancels in-flight upsample + seed-image work on unmount. seedImageOrchestrator.ts — loadSeedImageFile(imageId, apiClient) + sendSeedFileToReactor(file, reactor, opts) + convenience loadAndApplySeedImage. Used by all three /interactive React pages AND by the SWAP_SEED branch of the dispatcher, so the load → upload → set_image sequence has exactly one implementation. No @reactor-team/js-sdk dependency — that stays in the consuming app. Spec: specs/interactive-helios-prototype.md. |

Layout

src/
  audioProtocol.ts
  audio.ts
  visemeScheduler.ts
  visemeGeometry.ts
  websocket.ts
  api.ts
  reactor/
    types.ts
    regenTemplate.ts
    upsamplePrompt.ts
    entityContext.ts
    scenesFromEnrichment.ts        ← Phase 2 timeline mapper
    heliosTools.ts                 ← Phase 3 marker grammar
    toolPromptAddon.ts             ← Phase 3 system-prompt addon
    toolPromptAddonId.ts           ← Phase 3 addon ID constant
    entitySearch.ts                ← Phase 3 SWAP_SEED lookup (cached)
    heliosCommandDispatcher.ts     ← Phase 3 HeliosCommand executor (abort-aware)
    seedImageOrchestrator.ts       ← Phase 3+ load → upload → set_image
    index.ts                       ← submodule re-exports
  index.ts                      ← top-level re-exports
  __tests__/
    api-pollers.test.ts
    audio.test.ts
    loadBundleManifest.test.ts
    visemeGeometry.test.ts
    visemeScheduler.test.ts
    reactor/
      api-reactor.test.ts
      entityContext.test.ts
      regenTemplate.test.ts
      upsamplePrompt.test.ts
      scenesFromEnrichment.test.ts
      heliosTools.test.ts
      toolPromptAddon.test.ts
      entitySearch.test.ts
      heliosCommandDispatcher.test.ts
      seedImageOrchestrator.test.ts

Consumption — locally in this repo

The repo root is an npm workspace covering packages/* (client-core's own dev tooling). web/ is not a workspace member — it depends on this package via "@escapeaihq/client-core": "file:../packages/client-core", which makes npm create a symlink in web/node_modules/@escapeaihq/client-core. Edits here show up in web/ immediately. Next.js compiles the TS source through transpilePackages: ['@escapeaihq/client-core'], so no build step is required during dev.

Web was deliberately moved out of the root workspace: npm's workspace lockfiles strict-filter platform-specific optional deps, and a lockfile generated on macOS would break next build on Vercel/Linux when it couldn't resolve lightningcss-linux-x64-gnu. The file: symlink gives the same dev experience without polluting web's lockfile.

Consumption — sibling repo (web-app)

For the sibling Vue web-app, @escapeaihq/client-core is a regular dep pulled from the public npmjs.com registry (MIT). No auth required. package.json declares "@escapeaihq/client-core": "0.5.0" (pinned exact while pre-1.0). Plain npm install / npm ci fetches the compiled dist/ artifact — Nuxt dev SSR and production builds both consume that.

Optional: local live edits via npm link, for when you want to iterate here and see changes in web-app without re-publishing:

# one-time setup, after both repos are cloned side-by-side
cd ~/projects/seer-agent/packages/client-core && npm link
cd ~/projects/web-app && npm link @escapeaihq/client-core

Unlink with npm unlink @escapeaihq/client-core to fall back to the registry version. The symlink path used to be the default historic workflow (when the package was on auth-required GitHub Packages); since the move to public npm, the registry path works directly and the symlink is only needed for active cross-repo development.

Where to run npm install when adding/upgrading deps

The repo deliberately runs three install lifecycles, one per consumer:

| Goal | Command | Lockfile that gets updated | |---|---|---| | Add/upgrade a dep used by @escapeaihq/client-core (jest, ts-jest, typescript, etc.) | npm install <pkg> --workspace=@escapeaihq/client-core (from repo root) | repo-root package-lock.json | | Add/upgrade a dep used by web/ (Next.js, React, Tailwind, etc.) | cd web && npm install <pkg> | web/package-lock.json | | Pull in remote changes after a branch update | npm ci from repo root, then cd web && npm ci | both — neither lockfile is rewritten |

web/ is not a workspace member (the cross-platform optional-deps issue described above), so npm install at the repo root will not refresh web/node_modules. Always run installs in web/ separately.

Build & publish

The repo-checked-in package.json points main/types/exports at ./src/index.ts so in-monorepo consumers (web/) can import TypeScript source directly through Next.js's transpilePackages. The published artifact needs to point at ./dist/index.js instead — the publish workflow rewrites these in place via npm pkg set before running npm publish. (npm does not honor publishConfig.main/.types/.exports overrides the way pnpm/yarn-berry do, hence the rewrite.) The on-disk file is never committed with the rewritten paths.

A second post-build step (scripts/add-ext.mjs) appends .js to extension-less relative imports in dist/*.js. TypeScript's emitter doesn't add them, and Node strict ESM (used by Nuxt dev SSR in the sibling repo) refuses to resolve from './foo' without an explicit extension.

# from the package directory
npm run typecheck   # tsc -p tsconfig.json --noEmit
npm test            # jest (jsdom)
npm run build       # tsc -p tsconfig.build.json && node scripts/add-ext.mjs → dist/

Direct npm publish from a dev machine is not part of the flow — releases go through CI; see below.

Cutting a release

CI publishes to public npm (npmjs.com) on every push of a client-core-v* tag (.github/workflows/publish-client-core.yml). The package ships as MIT, installable without auth. The flow:

# 1. bump version in packages/client-core/package.json (semver)
# 2. commit the bump on main
git commit -am "client-core: release v0.4.2"
# 3. tag and push — the tag triggers the publish workflow
git tag client-core-v0.4.2
git push origin main client-core-v0.4.2

The workflow runs npm ci at the repo root, then npm --workspace=@escapeaihq/client-core run typecheck, test, and build, then npm publish --workspace=@escapeaihq/client-core. Auth uses an NPM_TOKEN secret — a granular access token with bypass 2FA enabled, scoped to the @escapeaihq org's packages. Provenance attestations (--provenance) are deliberately disabled: npm rejects them for packages whose source repo is private (seer-agent is private by design), and the rejection happens after the attestation is signed to the public sigstore log, which would leak commit references on every release. Supply-chain guarantees are instead SHA-pinned action versions + npm audit --audit-level=high gating publish.

Semver policy (manual bumps):

| Bump | When | |---|---| | Patch (0.x.Z) | Bug fix in an existing function; new internal helper; documentation; tests; non-API-visible refactor. | | Minor (0.Y.0) | New exported function/class/type; new optional parameter; new module under src/. | | Major (X.0.0 — pre-1.0 still bumps the leading 0.) | Renamed/removed exported symbol; required argument added or order changed (e.g. ApiClient constructor in 1a.1); changed return type or thrown errors that consumers branch on. |

While this is 0.x, the package is pre-stable: any release may include breaking changes in the constructor signatures of ApiClient, AudioPipeline, etc. — consumers should pin to an exact version ("@escapeaihq/client-core": "0.1.0", no caret) until 1.0.

What does NOT belong here

• React or Vue components, hooks, or composables — those live with their consumer
• App-specific config (env vars, base URLs) — pass them in via constructor args
• Any DOM rendering beyond what's needed for Web Audio / Web Animations / WebSocket APIs

Testing

Jest with ts-jest/presets/default-esm and jsdom environment. Tests live next to source under src/__tests__/. Run from this directory:

npm test
# or from monorepo root
npm run test:client-core