@monoverse/voicebot-react-native

The VoiceBot store assistant — voice + text — native in your React Native app. The same AI assistant that runs as the VoiceBot web widget: your customer taps a launcher, talks or types, and the bot searches your catalog, navigates, filters, and adds to cart through handlers you register.

The SDK is headless — you own the UI and the tool handlers. Voice runs 100% server-side on the VoiceBot backend (Gemini Live); the SDK only streams microphone PCM up and plays the bot's audio back. No AI model ships on the device.

Features

• Voice + text, one session. Realtime mic streaming (PCM16) and text turns over a single WebSocket.
• Headless core, your UI. A typed VoicebotClient / VoicebotSession state machine and event stream — drop it behind any component tree. No DOM, no forced widgets.
• Optional prebuilt UI kit. A themeable <VoicebotLauncher> (floating FAB + chat panel) under the @monoverse/voicebot-react-native/ui subpath — pure JS/RN, tree-shakeable, fully replaceable. Skip it and build on the session API, or drop it once at your root and you're done.
• Native audio with hardware AEC. iOS AVAudioEngine + .voiceChat; Android AudioRecord (VOICE_COMMUNICATION) + AcousticEchoCanceler. Barge-in flush, route/interruption handling.
• Tool handlers + deep links. Map the bot's UI actions (add_to_cart, open_product, apply_filter, …) to your cart API or app routes. Unregistered actions degrade safely — no crash.
• Context enrichment (zero-PII). Tell the bot what the user is looking at and what's in the cart so it answers sharper — updateContext() / sendCartEvent().
• Resilient transport. Exponential backoff, ~25s heartbeat, ≤3 reconnect attempts, and a 60s server reattach window so a backgrounded app resumes the same conversation.
• Minimize ≠ close. Collapsing the UI keeps the session alive; only an explicit close ends it.
• TypeScript-first. Strict types, zero any in the public API, typed event payloads and errors.

Platform support

| | | |---|---| | React Native | 0.76+ (New Architecture) · floor 0.73+ | | Architecture | New Architecture only (Nitro Modules; Turbo fallback) | | iOS | 13+ (audio path 16+) | | Android | API 21+ (audio path API 24+) | | Expo | Dev builds + expo-dev-client + the bundled config plugin — not Expo Go |

Install

npm install @monoverse/voicebot-react-native react-native-nitro-modules
# iOS:
cd ios && pod install

The native audio module uses the New Architecture — make sure it's enabled (default on RN 0.76+).

iOS — bare React Native (non-Expo)

The Expo config plugin injects the mic-permission string automatically. Bare RN apps must add it themselves, or iOS terminates the app the first time startVoice() requests the microphone. Add this to ios/<YourApp>/Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>Allow $(PRODUCT_NAME) to use the microphone so you can talk to the store assistant.</string>

(Reword the string for your brand — Apple shows it in the permission prompt.) See docs/STORE_SUBMISSION.md for the privacy-manifest and 5.1.2 disclosure steps.

Android — microphone permission

RECORD_AUDIO is declared by the library manifest and merges into your app automatically (the Expo plugin also ensures it). You do not need to call PermissionsAndroid yourself — session.startVoice() requests the runtime permission through the current Activity and resolves to a typed result. On a fresh install the user sees the system mic dialog on the first startVoice(); a denial surfaces as AudioError('permission_denied') and an error event with code: 'mic_denied', while the text path keeps working.

Expo

This SDK ships a native module, so it does not run in Expo Go. Use a dev build:

npx expo install @monoverse/voicebot-react-native react-native-nitro-modules expo-dev-client

Add the config plugin to app.json, then build a dev client:

{ "expo": { "plugins": ["@monoverse/voicebot-react-native/app.plugin"] } }

Reference the plugin by the /app.plugin subpath, not the bare package name. The package's exports map (dual CJS/ESM) shadows the bare-name plugin lookup, so "@monoverse/voicebot-react-native" alone fails prebuild with does not contain a valid config plugin.

npx expo run:ios      # or: npx expo run:android

The plugin wires NSMicrophoneUsageDescription (iOS) and RECORD_AUDIO (Android). Text chat works without the native module; voice does not.

Quickstart

import { VoicebotClient } from '@monoverse/voicebot-react-native';

// 1. Init (synchronous, no network). Pick an auth mode — see "Auth & API key".
const client = VoicebotClient.init({
  baseUrl: 'https://api.monoverse.tech',
  auth: { mode: 'publishableKey', apiKey: 'vb_pk_…', appId: 'com.yourco.app' },
});

// 2. Map the bot's UI actions to your app.
client.registerToolHandler('add_to_cart', async ({ id, qty }) => {
  await myCart.add(String(id), Number(qty));
  return { success: true, cartItemCount: myCart.count };
});
client.registerDeepLinkMap('open_product', ({ id }) => `myapp://product/${String(id)}`);

// 3. Start a session and listen.
const session = await client.startVoiceSession({ lang: 'uk' });
session.on('transcript', (t) => console.log(t.role, t.text)); // works on the text path alone
session.on('status', (s) => console.log(s.status));
session.on('error', (e) => console.warn(e.code, e.message));

// 4. Optional: supplement the bot with zero-PII app context.
session.updateContext({ screen: 'Home', locale: 'uk', loggedIn: false });

// 5. Voice (needs a real device + the native module). Failure is non-fatal — text keeps working.
await session.startVoice();

// Turns & control:
session.sendText('скільки коштує?'); // text turn
session.setMuted(true);              // mute the mic
session.minimize();                  // collapse UI — session STAYS ALIVE (no end sent)
session.restore();                   // un-collapse
await session.endSession();          // user closed the chat → real end + teardown

UI kit (optional)

The SDK is headless by default — you own the UI. But if you want the assistant on screen in five minutes, the package ships an optional, themeable prebuilt UI kit under the /ui subpath. It's pure JS/React Native (no extra native dependencies), tree-shakeable, and fully replaceable — ignore it entirely and build on the VoicebotSession event API if you prefer.

Import from the subpath: @monoverse/voicebot-react-native/ui. The headless core (@monoverse/voicebot-react-native) never imports the kit, so headless installs stay lean.

Turnkey: <VoicebotLauncher>

Drop it once at your app root — a floating launcher button that floats over every screen and owns the whole assistant (the VoicebotClient/session lifecycle + the chat panel):

import { VoicebotClient } from '@monoverse/voicebot-react-native';
import { VoicebotLauncher } from '@monoverse/voicebot-react-native/ui';

const client = VoicebotClient.init({
  baseUrl: 'https://api.monoverse.tech',
  auth: { mode: 'publishableKey', apiKey: 'vb_pk_…', appId: 'com.yourco.app' },
});
client.registerDeepLinkMap('open_product', ({ id }) => `myapp://product/${String(id)}`);

export default function App() {
  return (
    <>
      <YourAppNavigator />
      <VoicebotLauncher
        client={client}                 // or pass `config={{ baseUrl, auth }}` and it builds one
        startOptions={{ lang: 'uk' }}
        theme={{ colors: { primary: '#2563eb' } }}
        onSession={(s) => s.updateContext({ screen: 'Home', locale: 'uk', loggedIn: false })}
      />
    </>
  );
}

Behavior matches the lifecycle: tapping the button opens the panel; minimize collapses back to the button and the session STAYS ALIVE (no end); the X closes the session (endSession() → real teardown). If the bot or a fatal close ends the session, the kit drops back to the idle button.

Standalone components (custom layouts)

VoiceButton and ChatPanel are also exported on their own if you want to place them yourself. Use the useVoicebotSession(session) hook to get a reactive view (status, audioState, amplitude, transcript, lastError) from a session you manage:

import { VoiceButton, ChatPanel, useVoicebotSession } from '@monoverse/voicebot-react-native/ui';

function Assistant({ session, open, setOpen }) {
  const view = useVoicebotSession(session);   // subscribes to the session event streams
  return (
    <>
      <VoiceButton status={view.status} audioState={view.audioState} onPress={() => setOpen(true)} />
      <ChatPanel
        visible={open}
        status={view.status}
        audioState={view.audioState}
        transcript={view.transcript}
        onSend={(t) => session.sendText(t)}
        onMinimize={() => { session.minimize(); setOpen(false); }}   // session stays alive
        onClose={() => { session.endSession(); setOpen(false); }}    // real end
        onToggleMic={(next) => (next === 'start' ? session.startVoice() : session.setMuted(next === 'mute'))}
      />
    </>
  );
}

• VoiceButton — floating circular FAB. Reflects state: idle, connecting (spinner), listening (animated pulse ring), bot-speaking (accent color). Themeable size / position (corner + margin) / colors / elevation / icon (renderIcon).
• ChatPanel — the assistant window. Header (title + minimize + close), scrollable transcript with live streaming partials, footer (text input + send + mic toggle). Slide/fade animation, keyboard- and safe-area-aware. All chrome strings are overridable via labels (localization).

Theming

A single VoicebotTheme object (colors, radii, spacing, typography) with sensible defaults. Pass a partial theme to any kit component (or wrap a subtree in <ThemeProvider theme={…}>) and it is deep-merged over the defaults — omit what you don't want to change:

import { ThemeProvider, defaultTheme } from '@monoverse/voicebot-react-native/ui';

<ThemeProvider theme={{ colors: { primary: '#7c3aed', speaking: '#22c55e' }, radii: { panel: 28 } }}>
  {/* VoicebotLauncher / VoiceButton / ChatPanel inside read this */}
</ThemeProvider>

The same component + theme model is shared 1:1 with the Flutter SDK, so the cross-platform docs describe one model. It's all optional — the headless VoicebotClient/VoicebotSession API stays the primary surface and is fully usable without importing /ui.

Auth & API key

Authentication mirrors the web widget exactly. A tenant has one VoiceBot key = a paired tenantId plus a shared secret (created when you pair the plugin). The same key authorizes both your website widget (origin-bound) and your mobile app (bundle-id-bound).

Billing kill-switch. The backend mints session tokens only while the connection is active. If a subscription lapses or a limit is hit, the connection status flips and every token request fails and active sessions are revoked within ~30s. This is the "account alive → block the key" lever — no extra wiring on your side.

The SDK never embeds the shared secret in the app binary. Pick one of three modes via auth:

Mode A — server-minted token (recommended; secret stays server-side)

Your backend (the one already serving the app its products) holds the shared secret and mints the session token, exactly like the web. Implement a tiny TokenProvider that calls your endpoint, and pass it as custom:

const client = VoicebotClient.init({
  baseUrl: 'https://api.monoverse.tech',
  auth: {
    mode: 'custom',
    tokenProvider: {
      async issueToken(lang) {
        const r = await fetch('https://shop.example.com/voicebot/token', {
          method: 'POST',
          body: JSON.stringify({ lang }),
        });
        return r.json(); // { sessionToken, expiresAt, refreshAfter }
      },
    },
  },
});

Your backend signs the upstream POST /api/v1/mobile/issue-token request with the ingest HMAC (see docs/PROTOCOL.md §1 for the exact preimage). The SDK also ships an HmacTokenProvider that performs this signing itself — use it only where holding the shared secret is acceptable (a trusted server runtime or local development), never in a shipped binary.

Mode B — publishable key (convenience; no backend)

Embed a publishable key vb_pk_… — an identifier, not the secret, safe to ship. The SDK calls the public POST /api/v1/mobile/issue-token-public with your key + app_id (your bundle id / package, which must be allow-listed for the connection). Optional App Attest / Play Integrity attestation can be layered on as hardening.

auth: { mode: 'publishableKey', apiKey: 'vb_pk_…', appId: 'com.yourco.app' }

Session tokens are short-lived JWTs (TTL ~1h, refresh ~30m), stored securely and never logged.

Lifecycle (identical to the web widget)

| You call | What happens | |---|---| | client.startVoiceSession() | issue token → open WS → handshake (ready) | | session.minimize() | UI-only collapse. WS + session stay alive. No end is sent. | | session.restore() | un-collapse (nothing on the wire) | | app backgrounded | WS may drop → 60s reattach window; return <60s resumes the same conversation_id, >60s starts fresh with a recap | | session.endSession() | user closed the chat (X) → sends {type:"end"} → real termination + teardown | | bot ends / fatal close | SDK emits ended, tears down | | session.destroy() | hard teardown on screen unmount (no end sent) |

The headline rule: minimize ≠ close. In custom UI, wire your "collapse" control to minimize() and your "X" to endSession().

Tool handlers & deep links

The bot executes catalog lookups server-side and just speaks the result. For UI actions it pushes an action to the SDK, which runs your handler (or fires a deep link) and acknowledges the backend automatically.

// Native callback — e.g. mutate the cart, apply a filter:
client.registerToolHandler('apply_filter', (args) => {
  productList.setFilter(args);
  return { success: true };
});

// Deep link — navigate via your router:
client.registerDeepLinkMap('open_product', ({ id }) => `myapp://product/${String(id)}`);
client.registerDeepLinkMap('view_cart', () => 'myapp://cart');

Deep links open through React Native Linking by default (override with deepLinkOpener). An action with no registered handler is auto-acknowledged { success: false, error: "no_handler" } — the bot adapts and your app never crashes. See docs/PROTOCOL.md §9 for the full action → handler map.

Context enrichment (zero-PII)

Give the bot live, client-side context so it doesn't have to ask. Both methods reuse existing backend frames — no backend change required.

// On screen changes — current screen / product / category / locale + custom k/v:
session.updateContext({
  screen: 'ProductDetails',
  currentProductId: 'sku-42',
  category: 'phones',
  locale: 'uk',
  loggedIn: true,
  custom: { promoActive: true },
});

// Cart snapshot on start, then deltas:
session.sendCartEvent({
  type: 'snapshot',
  currency: 'UAH',
  items: [{ productId: 'sku-42', quantity: 1, price: 14999 }],
});
session.sendCartEvent({ type: 'add', items: [{ productId: 'sku-7', quantity: 2 }] });

Zero-PII contract. Never put a buyer's name, email, phone, address, payment data, or any personally-identifying order id in AppContext / CartEvent — the whole payload reaches the model. Keep it to catalog/UI coordinates and a logged-in boolean.

Voice & privacy

Voice is processed on the VoiceBot backend (Gemini Live), server-side — the SDK only captures microphone audio as PCM16 (16 kHz up) and plays the bot's PCM (24 kHz down). No AI/LLM SDK runs on the device. The Expo plugin wires the mic permission and the bundled ios/PrivacyInfo.xcprivacy declares Audio Data (App Functionality; no required-reason APIs). Foreground-only — no background audio.

Store submission: because voice is sent to a third-party AI (Gemini), Apple Guideline 5.1.2 and Play's GenAI/Data-safety policies require you to disclose it. Follow docs/STORE_SUBMISSION.md — a copy-paste iOS + Android checklist.

Voice readiness / tested platforms

The native audio pipeline is feature-complete: runtime mic permission (iOS + Android), PCM16 capture at 16 kHz with hardware AEC, 24 kHz playback with a ~100 ms jitter buffer + underrun handling, barge-in flush, audio-focus / interruption (incoming call, .shouldResume) and route-change handling (headset / Bluetooth), and software resampling if a device can't capture natively at 16 kHz.

| Surface | Verification | |---|---| | JS core (transport, codec, session, auth, tools, permission gate) | unit tests (Jest) — CI | | iOS Swift module compiles | xcodebuild on a simulator — CI (ios job) | | Android Kotlin module compiles | ./gradlew assembleDebug (NDK + CMake) — CI (android job) | | ≤40 ms capture→WS-frame latency on a real device (VOIC-45 AC) | requires a physical device — not yet run |

Both native modules now compile in CI on every push. The remaining acceptance criterion — measured capture→frame latency ≤40 ms and clean duplex audio on real iOS 16+/Android API 24+ hardware — needs a physical-device run and is the only open item before voice GA.

Status

🚧 Pre-1.0. Headless core + text path + auth (Mode A/B) + context enrichment + the optional prebuilt UI kit (/ui) are implemented and tested. The native audio pipeline is feature-complete and CI-compiled on both platforms (see Voice readiness above); the only open voice item is a real-device latency measurement. Packaging/publish hardening and the hosted docs site follow. Track the wire contract in docs/PROTOCOL.md.

Docs

• docs/PROTOCOL.md — the WebSocket wire contract (ground truth; build against this).
• docs/HANDOFF.md — the implementation contract (scope, decisions, lifecycle).
• docs/impl-notes.md — implementation log.
• docs/STORE_SUBMISSION.md — App Store + Play store-compliance checklist for merchants.
• example/ — a runnable app (voice + text) against a configurable backend.
• CONTRIBUTING.md — local dev, CI, and the release flow (npm OIDC Trusted Publishing — one-time setup + tag-to-publish).

License

MIT