@streamoji/aitwin

Embeddable React component that renders an AI twin face (canvas + viseme diff lipsync) and exposes speakText() for parent-controlled speech.

Installation

npm install @streamoji/aitwin

Peer dependencies:

npm install react react-dom

Usage

import { useRef } from "react";
import { AiTwin, type AiTwinHandle } from "@streamoji/aitwin";

function Demo() {
  const twinRef = useRef<AiTwinHandle>(null);

  return (
    <>
      <AiTwin
        ref={twinRef}
        id="olivia"
        authToken={optionalBearerToken}
        onReady={() => console.log("face ready")}
        onStatusChange={(s) => console.log("status", s)}
        onError={(msg) => console.error(msg)}
      />
      <button
        type="button"
        onClick={() => void twinRef.current?.speakText("Hi, how are you?")}
      >
        Speak
      </button>
    </>
  );
}

Props

Provide id (cloud twin) or assets (fixed URLs). One is required.

| Prop | Description | |------|-------------| | id | Twin id for getAiTwin (e.g. olivia) | | assets | { twinBase, binBase, encrypted } — skip getAiTwin (lab / custom CDN) | | authToken | Bearer for TTS + encrypted assets; omitted → dev getAuthToken | | ttsEngineId | TTS engine when using assets (default Cartesia) | | voiceId | Override TTS voice | | speakingRate | Default 0.85 | | showErrorOverlay | Canvas error overlay (default true) | | onReady | Assets loaded and canvas ready | | onStatusChange | TTS status: idle, loading, speaking, done, error | | onDisplayStatus | Compositor label (viseme / idle / transition) | | onError | Load or runtime errors |

Use stable useCallback handlers for onReady / onError / onDisplayStatus in parent components.

Ref handle

| Method | Description | |--------|-------------| | speakText(text, options?) | TTS + lipsync; optional per-call tts / voiceId | | stop() | Stop playback and return toward idle | | setTtsProvider(provider) | Switch Google / Inworld / Cartesia | | renderViseme(to, options?) | Manual viseme transition (lab) | | isReady() | Whether face assets are loaded |

Architecture (aitwin monorepo)

| Path | Role | |------|------| | packages/aitwin | Source of truth — canvas renderer, worker, TTS lipsync, AiTwin | | frontend | Lab app: /viseme-diff-preview uses <AiTwin assets={…} />, /aitwin-demo uses id | | frontend/src/components/AiTwin | Legacy Talking Lady still-image widget only (re-exports TTS from package) | | frontend/src/lib/twinPreviewConfig.ts | Vite env → blondeladyPreviewAssets() for the diff preview page |

Do not duplicate renderer code under frontend/src/lib; extend the package instead.

Worker CDN

The viseme diff Web Worker is hosted on R2 and loaded at runtime via fetch + blob URL (cross-origin new Worker(cdnUrl) is blocked by browsers).

The worker URL is baked into each release from package.json version (config/defaults.ts) and uploaded to the matching versioned R2 path via npm run upload:worker (see below). This avoids CDN cache mismatches between the npm bundle and the worker script.

Example URL for 0.1.4: https://pub-607ad1fc22e2400eb57d17240aab857c.r2.dev/aitwin-workers/v0.1.4/visemeDiffPreview.worker.js

Upload worker after build

export R2_ACCESS_KEY_ID=...
export R2_SECRET_ACCESS_KEY=...
npm run build
npm run upload:worker

R2 CORS (required for browser fetch)

Allow GET and HEAD from your app origins (https://aitwin.me, http://localhost:3000). Use Cloudflare dashboard → R2 → aitwin bucket → Settings → CORS, or:

npx wrangler r2 bucket cors set aitwin --file scripts/r2-cors.json

(scripts/r2-cors.json is included; needs a Cloudflare API token with R2 Admin permissions.)

Publishing

From packages/aitwin:

npm run build
npm version patch
npm publish --access public

prepublishOnly runs build and upload:worker automatically.

The published tarball includes only dist/ (bundled JS + .d.ts + worker chunk). No src/ and no source maps.

Scoped packages need --access public on the free npm plan.