@nia-teams/multi-speaker-agent-demo
v0.1.0
Published
Browser-audio multi-speaker transcription, speaker attribution, and live Agent suggestions demo
Downloads
183
Readme
Multi-Speaker Agent Demo
English | 简体中文
A local workbench for browser audio capture, multi-speaker transcription, speaker attribution, and live Agent suggestions.
npm install @nia-teams/multi-speaker-agent-demoAI coding assistants can load node_modules/@nia-teams/multi-speaker-agent-demo/skills/multi-speaker-agent-demo/SKILL.md (skills/multi-speaker-agent-demo/SKILL.md).
Quick start
Node.js >= 20.12. Run from your project directory:
npx @nia-teams/multi-speaker-agent-demo setup
# Set ASR_API_KEY and LLM_API_KEY in .env.
npx @nia-teams/multi-speaker-agent-demo doctor
npx @nia-teams/multi-speaker-agent-demo startOpen http://127.0.0.1:18765, select an input device and capture mode, and click 开始收音 (Start Capture).
The CLI reads .env and models/eres2net_zh.onnx from the current working directory. setup creates the configuration from .env.example and downloads the SHA-256-verified speaker model; mono attribution uses this model, while stereo attribution follows physical channels.
Repository checkouts also provide npm run setup, npm run doctor, and npm start.
Capture and speaker modes
| Mode | Input | Speaker attribution |
| --- | --- | --- |
| Mono (mono, default) | Built-in laptop microphone, USB conference speakerphone, or USB omnidirectional microphone | ERes2Net voiceprint clustering (limit of 2 or 6 speakers); the first speaker becomes SPK_A (人 A) |
| Stereo (stereo) | Dual-channel wireless lavaliers or stereo-routed dual-beam USB microphones | Left channel → SPK_A (人 A); right channel → SPK_B (人 B) |
The default suggestion policy advises SPK_A (人 A): in mono mode, have the assisted participant speak first; in stereo mode, assign them the left-channel microphone.
Stereo mode suppresses crosstalk between channels. Selecting a mono microphone as a stereo input produces persistently near-identical L/R waveforms; when detected, the server retains left-channel transcription and displays a warning in the UI.
Voiceprints attribute speech within a session and are not an authentication mechanism.
Architecture
Browser WebAudio (16 kHz PCM16LE) ──WS /ws/audio──▶ Qwen Realtime ASR ──▶ Speaker attribution (ERes2Net / L/R channels) ──▶ @nia-teams/agent-core ──▶ Live captions and Agent suggestions- Capture and framing: The browser resamples to 16 kHz PCM16LE and sends roughly 100 ms frames: 3200 bytes for mono or 6400 bytes for interleaved L/R stereo. Mono enables echo cancellation, noise suppression, and gain control; stereo disables these processing options to preserve independent channels.
- Transcription and attribution: The server maintains persistent ASR connections and sends in-progress text as
partialand completed speech asutterance. Mono extracts timestamp-aligned speech from a 120-second PCM ring buffer for voiceprint computation and session-scoped speaker assignment; stereo transcribes each channel separately. - Session and connection lifecycle: One process holds a single in-memory session shared across open browser tabs. Before switching inputs or clearing history, the browser sends
stopand waits for trailing speech to commit before receivingstopped;reconnect=1reattaches to the active ASR streams within 15 seconds. Browser and upstream connections retry up to five times with bounded send queues; once retries are exhausted, the browser stops capture and mutes the session via/api/mute.
Configuration
The server listens only on local loopback at http://127.0.0.1:<PORT> (default 18765), without built-in user authentication.
| Variable | Default | Purpose |
| --- | --- | --- |
| PORT | 18765 | HTTP and WebSocket port |
| ASR_BASE_URL | wss://dashscope.aliyuncs.com/api-ws/v1/realtime | Realtime ASR WebSocket endpoint |
| ASR_MODEL | qwen3-asr-flash-realtime | ASR model |
| ASR_API_KEY | empty | ASR credential; missing disables live capture |
| ASR_LANGUAGE | zh | Recognition language; leave empty to omit the upstream language parameter |
| ASR_PROMPT | empty | Optional ASR vocabulary hint; omitted when empty |
| LLM_PROTOCOL | openai-completions | openai-completions, openai-responses, anthropic, or gemini |
| LLM_BASE_URL | https://dashscope.aliyuncs.com/compatible-mode/v1 (openai-completions) | Model endpoint URL; other protocols use their SDK default when empty |
| LLM_MODEL | qwen-flash | Model for live Agent suggestions |
| LLM_API_KEY | empty | Model credential; missing disables live Agent suggestions |
GET /api/state exposes session state; POST /api/mute, /api/capture_mode, /api/speaker_mode, and /api/prompt control mute, capture mode, speaker limit, and Agent suggestion policy. POST /api/clear resets conversation and voiceprint state and rotates session_id.
Development
npm run typecheck
npm test
npm run doctor
npm pack --dry-runPrivacy and licensing
Transcripts, conversation history, and voiceprints remain in process memory until the session is cleared or the process exits. Audio goes to the configured ASR provider and conversation text goes to the configured model provider; obtain participant consent before capture.
Released under the MIT License. Model and dependency notices are in THIRD_PARTY_NOTICES.md.
