hear-my-site

Headless, framework-agnostic live browser audio transcription for agents such as Codex through WebMCP. The package mixes browser audio tracks locally, sends the selected provider either mono PCM16 or one mixed MediaStreamTrack, and registers exactly three adopter-named tool roles on document.modelContext. Providers are Gemini 3.5 Transcribe Live, OpenAI GPT-Live-Transcribe, and Chrome Web Speech.

The package is browser-only ESM and requires Chrome 150+ with the current WebMCP imperative API. It has no components, CSS, framework hooks, WebMCP polyfill, deprecated alias, or no-WebMCP mode.

Install

pnpm add hear-my-site

Minimal integration

import { createHearMySite } from 'hear-my-site';

const hear = await createHearMySite({ credential: { type: 'api-key', value: userSuppliedKey } });
hear.addAudioSource(existingMeetingStream);

The constructor waits up to five seconds for document.modelContext. If the browser does not expose it, creation rejects with WEBMCP_UNAVAILABLE.

Complete API

const hear = await createHearMySite({
  credential,
  options: {
    provider: 'gemini',
    languageCode: '',
    mode: 'VERBATIM',
    customVocabulary: [],
  },
  webMcp: {
    tools: {
      start: { name: 'join_meeting', title: 'Join meeting', description: 'Join as the local meeting assistant.' },
      get: { name: 'follow_meeting', title: 'Follow meeting', description: 'Wait for the next meeting segment.' },
      stop: { name: 'leave_meeting', title: 'Leave meeting', description: 'Leave and release local assistant resources.' },
    },
    allowCredentialInput: true,
    transcriptWaitMs: 20_000,
  },
});

hear.setCredential(credential);
hear.clearCredential();

const sourceId = hear.addAudioSource(streamOrTrack, {
  id: 'remote-alice',
  gain: 1,
  owned: false,
});
hear.removeAudioSource(sourceId);

const unsubscribe = hear.subscribe((state) => render(state));
hear.getState();

await hear.start({ mode: 'SMART' });
const page = hear.getTranscript({
  afterSegmentId: 0,
  maxChars: 50_000,
  includeInterim: true,
});
await hear.stop();

unsubscribe();
await hear.destroy();

start(), getTranscript(), and stop() return a shared CommandResult shape. Expected conditions such as a missing credential, no active audio, a required user gesture, or an already stopped session are represented by stable codes rather than hidden fallbacks.

Stopping retains the completed transcript for later reads. Starting again creates a new session. destroy() stops the session, clears the in-memory credential, aborts all three WebMCP registrations, closes the audio graph, and releases every package-owned resource. A destroyed instance cannot be reused.

Credentials

Gemini and OpenAI accept a directly supplied browser API key or an ephemeral token. Chrome Web Speech requires no credential:

const hear = await createHearMySite({ options: { provider: 'web-speech' } });
hear.addAudioSource(existingMeetingStream);
await hear.start();

Select the provider in the transcription options; the package never silently switches providers.

const hear = await createHearMySite({
  credential: { type: 'api-key', value: userSuppliedOpenAiKey },
  options: { provider: 'openai' },
});

hear.setCredential({ type: 'api-key', value: anotherUserSuppliedKey });
hear.setCredential({ type: 'ephemeral-token', value: token });

An async provider is also available when an adopting site wants to retrieve fresh credentials from its own backend:

const hear = await createHearMySite({
  credential: async ({ provider, reason, connection }) => {
    const response = await fetch(`/api/${provider}-token`, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ reason, connection }),
    });
    const { token } = await response.json();
    return { type: 'ephemeral-token', value: token };
  },
});

For Gemini and OpenAI, the credential provider is called for the initial connection and again during automatic nine-minute connection rotation. The package does not write credentials to storage or include their values in surfaced errors.

Audio ownership and mixing

addAudioSource() accepts a MediaStream or live audio MediaStreamTrack. Multiple local and remote sources are mixed through one Web Audio graph, and sources may be added or removed while transcription is running.

Sources are borrowed by default. Borrowed WebRTC tracks remain live after stop() and can continue powering the meeting. Set owned: true only when the package should stop and remove the track on stop, as the Hear My Browser extension does for its tab-capture stream.

Each source accepts a gain from 0 to 4. Duplicate IDs, invalid gains, and inputs without a live audio track fail explicitly. Gemini receives 16 kHz PCM16 in 100 ms chunks, OpenAI receives 24 kHz PCM16 through WebRTC, and Chrome Web Speech receives one mixed MediaStreamTrack. Optional Web Speech language, punctuation, and vocabulary hints are applied when supported and otherwise ignored.

The package ships its AudioWorklet as a separate same-origin JavaScript asset instead of a data: URL, so applications can keep a strict script-src 'self' Content Security Policy.

Transcript cursors

getTranscript() defaults to 50,000 finalized characters and includes the current interim utterance. Use its cursor incrementally:

let cursor = 0;

const result = hear.getTranscript({ afterSegmentId: cursor });
if (result.ok) {
  consume(result.data.segments);
  cursor = result.data.cursor;
  if (result.data.hasMore) readAgainImmediately();
}

historyTruncated signals that the requested cursor predates retained local history. Transcript results carry untrustedContentHint: true in WebMCP because spoken material is data, not an instruction to Codex or another agent.

WebMCP tools

Creation always registers three roles—start, get, and stop—but the adopting site owns their public identity. Omit webMcp.tools to use the transcription-oriented defaults:

• start_browser_transcription
• get_browser_transcript
• stop_browser_transcription

When customizing, provide all three names, titles, and descriptions. Names must be unique and contain at most 64 letters, numbers, underscores, or hyphens. Complete identity sets prevent a half-renamed surface whose tool names and descriptions disagree.

allowCredentialInput is false by default. Enabling it adds an optional API-key or ephemeral-token credential to the start role. The executor validates the credential, stores it only on the in-page session, and never includes it in results or errors. Use this only when Codex or another calling agent has its own approved secret source.

transcriptWaitMs enables cancellable long reads for the get role. When the cursor is current, the call waits for a finalized segment or session-state change, then returns immediately; the maximum is 30 seconds. A timeout still returns the latest interim text. This keeps a meeting assistant responsive without a fixed five-second polling loop.

The WebMCP executors call the same session object as the manual methods. All three honor the executor’s lifecycle signal; destroying the instance aborts registration and unregisters every tool.

When Hear My Browser is also installed, the native site registration owns the page. The extension automatically unregisters its default tools and yields to the site's configured tool set, so Codex and other agents never see duplicate tools or a session different from the site's state.

Defaults

| Setting | Default | | --- | --- | | Provider | gemini | | Language | Automatic detection ("") | | Mode | VERBATIM | | Vocabulary | [] | | Transcript page | 50_000 finalized characters | | Interim text | Included |

Security boundary

Raw audio travels directly from the adopting page to the explicitly selected transcription provider. Gemini and OpenAI accept direct browser API keys or ephemeral tokens; Chrome Web Speech requires no credential and Chrome determines whether recognition is local or remote. This package has no analytics, telemetry, remote logging, persistence, signaling, or developer-operated service. A website that issues ephemeral tokens is responsible for authenticating its own token endpoint and minting appropriately constrained tokens.

License

MIT