@streamoji/aitwin

Embeddable React component that renders an AI twin face (canvas + viseme diff lipsync) and exposes speakText() for parent-controlled speech.

Installation

npm install @streamoji/aitwin

Peer dependencies:

npm install react react-dom

Usage

import { useRef } from "react";
import { AiTwin, type AiTwinHandle } from "@streamoji/aitwin";

function Demo() {
  const twinRef = useRef<AiTwinHandle>(null);

  return (
    <>
      <AiTwin
        ref={twinRef}
        id="olivia"
        authToken={optionalBearerToken}
        onReady={() => console.log("face ready")}
        onStatusChange={(s) => console.log("status", s)}
        onError={(msg) => console.error(msg)}
      />
      <button
        type="button"
        onClick={() => void twinRef.current?.speakText("Hi, how are you?")}
      >
        Speak
      </button>
    </>
  );
}

Props

Provide id (cloud twin), avatarId/faceId (R2 custom face), or assets (fixed URLs). One is required.

| Prop | Description | |------|-------------| | id | Twin id for getAiTwin (e.g. olivia) | | avatarId | 64-char hex pipeline id → R2 custom-faces/{avatarId}/ (skips getAiTwin) | | faceId | Alias for avatarId (same CDN path) | | assets | { twinBase, binBase, encrypted } — skip getAiTwin (lab / custom CDN) | | authToken | Bearer for TTS + encrypted assets; omitted → dev getAuthToken | | ttsEngineId | TTS engine when using assets (default Cartesia) | | voiceId | Override TTS voice | | speakingRate | Default 0.85 | | showErrorOverlay | Canvas error overlay (default true) | | onReady | Assets loaded and canvas ready | | onStatusChange | TTS status: idle, loading, speaking, done, error | | onDisplayStatus | Compositor label (viseme / idle / transition) | | onError | Load or runtime errors | | onUserTranscript | Voice STT transcript (partial + final) | | onBotOutput | Bot LLM text from voice session | | onRealtimeLipsyncDebug | Lipsync debug snapshots (lab) |

Use stable useCallback handlers for onReady / onError / onDisplayStatus in parent components.

Ref handle

| Method | Description | |--------|-------------| | speakText(text, options?) | SSE TTS + lipsync; optional per-call tts / voiceId | | connect(options) | Mic + WebSocket voice session (/ws/voice) | | disconnect() | End voice session | | isConnected() | Whether voice WebSocket is active | | stop() | Stop playback and return toward idle | | setTtsEngineId(engineId) | Switch Google / Inworld / Cartesia | | renderViseme(to, options?) | Manual viseme transition (lab) | | isReady() | Whether face assets are loaded | | getStatus() | Current lipsync status |

Thumbnail URL

For avatar lists or previews, use getAiTwinThumbnailUrl — no <AiTwin> mount, no worker, no encrypted asset fetch:

import { getAiTwinThumbnailUrl } from "@streamoji/aitwin";

const thumb = getAiTwinThumbnailUrl(faceId); // faceId = pipeline avatarId (64-char hex)
// <img src={thumb} alt="" />

Optional custom CDN base (e.g. VITE_R2_PUBLIC_BASE + "/custom-faces"):

getAiTwinThumbnailUrl(faceId, customCdnBase);

Realtime voice (connect)

Call from a user gesture (button click) so the browser unlocks mic + audio:

import { useRef } from "react";
import { AiTwin, type AiTwinHandle } from "@streamoji/aitwin";

function VoiceDemo() {
  const twinRef = useRef<AiTwinHandle>(null);

  const onConnect = async () => {
    const authJwt = await getAuthToken(); // client_… — same as SSE
    await twinRef.current?.connect({
      authToken: authJwt,
      voiceId: optionalCartesiaUuid,
      speakingRate: 0.85,
    });
  };

  return (
    <>
      <AiTwin
        ref={twinRef}
        id="olivia"
        onUserTranscript={(text, final) => final && console.log("you:", text)}
        onBotOutput={(text) => console.log("bot:", text)}
      />
      <button type="button" onClick={() => void onConnect()}>
        Connect mic
      </button>
    </>
  );
}

WebSocket URL: wss://<api-host>/ws/voice?<query>. The SDK builds it from connect() options. Browsers cannot set Authorization on WebSocket easily — pass the same JWT as authToken query param.

| Query param | Required | Source | |-------------|----------|--------| | authToken | Yes | Same JWT as SSE Authorization: Bearer (client_… from getAuthToken) | | tenant | Yes | Always aiTwin | | voiceId | No | Cartesia UUID override (from getAiTwin or connect({ voiceId })) | | personaId | No | getAiTwin knowledgeContextId when present | | speaking_rate | No | 0.5–1.5, default 0.85 | | tts_stream | No | Default true (server TTS + avatar_audio_chunk) |

speakText() (SSE) and connect() (voice) share the same billing token. Pass authToken on <AiTwin> or in connect({ authToken }); omitted → dev getAuthToken is fetched automatically.

Low-level API: createRealtimeVoiceLipsyncController, buildVoiceWebSocketUrl, fetchDevAuthToken.

Architecture (aitwin monorepo)

| Path | Role | |------|------| | packages/aitwin | Source of truth — canvas renderer, worker, TTS lipsync, AiTwin | | frontend | Lab app: /viseme-diff-preview uses <AiTwin assets={…} />, /aitwin-demo uses id | | frontend/src/components/AiTwin | Legacy Talking Lady still-image widget only (re-exports TTS from package) | | frontend/src/lib/twinPreviewConfig.ts | Vite env → blondeladyPreviewAssets() for the diff preview page |

Do not duplicate renderer code under frontend/src/lib; extend the package instead.

Worker CDN

The viseme diff Web Worker is hosted on R2 and loaded at runtime via fetch + blob URL (cross-origin new Worker(cdnUrl) is blocked by browsers).

The worker URL is baked into each release from package.json version (config/defaults.ts) and uploaded to the matching versioned R2 path via npm run upload:worker (see below). This avoids CDN cache mismatches between the npm bundle and the worker script.

Example URL for 0.1.4: https://aitwin.bubu.social/aitwin-workers/v0.1.4/visemeDiffPreview.worker.js

Upload worker after build

export R2_ACCESS_KEY_ID=...
export R2_SECRET_ACCESS_KEY=...
npm run build
npm run upload:worker

R2 CORS (required for browser fetch)

Allow GET and HEAD from your app origins (https://aitwin.me, http://localhost:3000). Use Cloudflare dashboard → R2 → aitwin bucket → Settings → CORS, or:

npx wrangler r2 bucket cors set aitwin --file scripts/r2-cors.json

(scripts/r2-cors.json is included; needs a Cloudflare API token with R2 Admin permissions.)

Publishing

From packages/aitwin:

npm run build
npm version patch
npm publish --access public

prepublishOnly runs build and upload:worker automatically.

The published tarball includes only dist/ (bundled JS + .d.ts + worker chunk). No src/ and no source maps.

Scoped packages need --access public on the free npm plan.