@deepseek-ai/dsh-voice-text-mainline

Host side of the voice text mainline: the voice configuration schema and the STT/TTS provider registry. Browser-side recording, session-event bridging, and the voice UI live in the paired Client package @deepseek-ai/dsh-client-voice-text-mainline.

What it does

• Defines the voice.* configuration (sttProvider, ttsProvider, language, autoPlay, silenceThresholdMs) with deployment defaults.
• Provides the voiceProviders registry service: STT/TTS implementations register under stable ids and resolve by configuration.
• Ships the Doubao (Volcengine SAUC) cloud STT provider: a binary WebSocket client for bigmodel_nostream that registers under the doubao id when DOUBAO_API_KEY is present in the Host environment.
• Exposes the voice Remote API (ctx.remote.voice.transcribe on the Client): the browser sends re-encoded wav bytes, the Host runs the cloud adapter where credentials live.
• Exposes the shared voice-domain types (VoiceConfig, VoiceStatus, VoiceSttProvider/VoiceTtsProvider, VoiceControl) that the Client package consumes through a workspace dependency.

Voice is an enhancement, not the mainline: when no STT/TTS provider is configured or available, the text mainline keeps working unchanged. No session event is added and agent-loop is untouched.

Config

| Key | Default | Meaning | | --- | --- | --- | | sttProvider | mock | STT provider id resolved through voiceProviders | | ttsProvider | mock | TTS provider id resolved through voiceProviders | | language | zh-CN | Language passed to TTS | | autoPlay | true | Whether assistant replies auto-play | | silenceThresholdMs | 500 | Silence threshold (recorded; auto-stop is a later phase) |

Usage

- id: voice-text-mainline
  name: '@deepseek-ai/dsh-voice-text-mainline'
  config:
    sttProvider: mock
    ttsProvider: mock
    language: zh-CN
    autoPlay: true

Cloud ASR (Doubao SAUC)

Set sttProvider: doubao and export the API key in the Host environment (the browser never sees it):

- id: voice-text-mainline
  name: '@deepseek-ai/dsh-voice-text-mainline'
  config:
    sttProvider: doubao
    ttsProvider: mock

export DOUBAO_API_KEY=<your-volcengine-x-api-key>

The doubao provider registers only when the key is present; otherwise the client-side resolver falls back to the text-only mainline. The recognition endpoint (wss://openspeech.bytedance.com/api/v3/sauc/bigmodel_nostream) and resource id (volc.bigasr.sauc.duration) are the documented defaults.

Config loading order

voice.* follows the standard harness layering; each later layer replaces whatever the earlier layer produced for the keys it sets:

1. Plugin defaults — the schemastery Config in src/index.ts supplies mock / mock / zh-CN / true / 500.
2. cordis.yml / profile rows — values on the voice-text-mainline row override the defaults per key.
3. --patch overlays — a patch replaces the targeted row's whole config, so an overlay restates every key it owns (same rule as cordis.patch.yml).
4. Environment variables — only where the config file explicitly reads them, e.g. sttProvider: !!js process.env.VOICE_STT_PROVIDER.

A deployment with no voice config keeps the defaults; an empty sttProvider or ttsProvider disables voice (isVoiceEnabled false) and the Client falls back to the text-only mainline. Values outside the schema bounds fail loud at load.

Host-to-Browser control surface

The shared VoiceControl contract (in src/types.ts) defines the control verbs the browser half implements: startRecording, stopRecording, cancel, and getStatus. The Host package declares the contract and the voiceProviders service carries the resolved configuration; the implementation lives in the Client package (@deepseek-ai/dsh-client-voice-text-mainline), which owns recording, playback, and the state machine. Phase 1 adds no RPC channel — configuration crosses as the shared Config schema, and control stays inside the browser.

Text mainline safety

Voice never touches agent-loop, SessionEventMap, or the user/message / assistant/chunk / assistant/message contract: input enters as ordinary text via session.prompt(content, 'queue'), and output is observed from the conversation snapshot. With voice disabled the Host provides an empty registry and the Client simply does not mount its UI.

Model Experience

None, as the model sees no voice-specific request context; the voice input is converted to ordinary user/message text before reaching any model request.

KV Cache effect

No package-owned change to any model request prefix; voice input arrives as ordinary text messages.

Known Limitations and Deferred Work

• Cloud TTS not implemented — ttsProvider stays mock (tone playback); real synthesis belongs to a later change.
• Doubao resource grant is deployment-specific — the default resource id must be granted to the key's console account; a requested resource not granted error means the console resource needs activating.
• silenceThresholdMs is recorded but not enforced — automatic stop on silence belongs to a later phase.