valajs

A low-latency, fully in-browser voice/text conversational agent. Speech-to-text, an LLM, and text-to-speech each run in their own Web Worker — nothing is uploaded anywhere. The mic listens continuously, including while the agent is thinking or talking, so you can interrupt it mid-sentence and it stops immediately, like a real conversation instead of a phone tree.

Install

npm i valajs@latest

That's it for most setups — see "Serving the workers" below if your bundler needs a nudge.

What's actually in here

• STT: Whisper (whisper-tiny.en) via transformers.js, in a Web Worker.
• LLM: any WebLLM-prebuilt model (small Qwen2.5 instruct models by default), in its own Web Worker.
• TTS: Kokoro via kokoro-js, in a Web Worker.
• No React, no framework dependency. ValaAgent is a plain class with callback options — use it from anything.

Framework support

ValaAgent is framework-agnostic — it's a plain class, so it works in any frontend stack. The worker auto-resolution (see "Serving the workers" below) is tested and works out of the box with:

• Next.js (App Router or Pages Router)
• React (Vite, Create React App, or any webpack 5 setup)
• Nuxt.js (Vue)
• NukeJS (React)

Since ValaAgent touches the microphone and Worker/AudioContext APIs, it only runs in the browser — instantiate it client-side:

• Next.js: create/use the agent inside a "use client" component (e.g. in useEffect), not in a Server Component.
• Nuxt.js: create/use the agent inside onMounted(), or guard with import.meta.client / process.client, so it never runs during SSR.
• React (Vite/CRA, no SSR): safe to use directly in useEffect.

Minimal React/Next.js example:

"use client"; // only needed in Next.js App Router

import { useEffect, useRef } from "react";
import { ValaAgent } from "valajs";

export function VoiceAgent() {
    const agentRef = useRef<ValaAgent | null>(null);

    useEffect(() => {
        const agent = new ValaAgent({ systemPrompt: "You are a friendly, concise assistant." });
        agentRef.current = agent;
        agent.warmup().then(() => agent.startListening());
        return () => agent.destroy();
    }, []);

    return <div>Listening…</div>;
}

Quick start

import { ValaAgent } from "valajs";

const agent = new ValaAgent({
    systemPrompt: "You are a friendly, concise assistant. Keep replies to 1-2 sentences.",
    onStageChange: (stage) => console.log("stage:", stage),
    onModelLoadProgress: ({ source, fraction }) => console.log(source, Math.round(fraction * 100) + "%"),
    onPartialTranscript: (text) => console.log("hearing:", text),
    onUserTurn: (text) => console.log("user said:", text),
    onAssistantSentence: (sentence) => console.log("agent:", sentence),
    onError: (err, context) => console.error(context, err),
});

await agent.warmup();         // downloads/loads STT + LLM + TTS
await agent.startListening(); // opens the mic, always-listening from here on

// or, text input instead of / alongside voice:
agent.sendText("What's the capital of France?");

// stop the agent's current reply and let the person talk over it:
agent.interrupt();

// later:
agent.destroy();

Serving the workers

Each model runs in its own Web Worker. By default, ValaAgent resolves its three pre-bundled worker files (shipped inside the package, already built — see dist/workers/) automatically, using the new Worker(new URL("./workers/x-worker.js", import.meta.url), { type: "module" }) convention. Vite, webpack 5, Next.js, and most modern bundlers special-case exactly this pattern to bundle/copy the worker file for you — so on any of those, there is nothing else to configure. Just:

const agent = new ValaAgent({ systemPrompt: "..." });
await agent.warmup();

If your bundler doesn't support that convention

(e.g. you're calling esbuild directly, or another bundler without that special-case.) Copy the three files from node_modules/valajs/dist/workers/ into whatever directory your app serves as static assets, and point the agent at them explicitly:

cp node_modules/valajs/dist/workers/*.js public/

new ValaAgent({
    systemPrompt: "...",
    workerUrls: {
        llm: "/llm-worker.js",
        stt: "/stt-worker.js",
        tts: "/tts-worker.js",
    },
});

Any workerUrls field you provide overrides the auto-resolved default for just that worker — you don't need to set all three if only one needs it.

API

new ValaAgent(options)

See src/types.ts for the full ValaAgentOptions reference. The only required field is systemPrompt. Notable options:

| Option | Default | Notes | |---|---|---| | modelId | Qwen2.5-0.5B-Instruct-q4f16_1-MLC | Any WebLLM-prebuilt model id. Smaller = lower latency. | | maxTokens | 100 | Per-reply cap. The single biggest lever on worst-case reply latency. | | enableVoiceInput / enableVoiceOutput | true / true | Set either false for text-only in/out. | | voice | af_heart | Kokoro voice id. | | workerUrls | auto-resolved | Only needed to override on a bundler that doesn't support new Worker(new URL(...)) — see "Serving the workers" above. |

Methods

• warmup(): Promise<void> — loads whichever models are actually needed. Call once, before startListening()/sendText() (they'll still work without it, they'll just eat the load time on the first turn instead).
• startListening(): Promise<void> — opens the mic, begins the always-listening loop.
• stopListening(): void — closes the mic. Any turn already in flight keeps running.
• sendText(text: string): void — submits a text turn; interrupts whatever the agent was doing first.
• interrupt(): void — stops the current reply (LLM + TTS) and returns to listening. Also called internally the instant voice barge-in is detected.
• getStage(): AgentStage / getHistory(): ChatMessage[]
• destroy(): void — tears down the mic, playback, and both worker pools. Not reusable after this.

Why it's low-latency

A few deliberate design choices, in order of impact:

1. The mic never stops. Most voice-agent implementations only listen between turns. Here, a new mic capture opens the instant the previous one ends — before transcription, the LLM call, or TTS playback for that turn have even started — so it's already recording by the time the person starts talking again, whether that's a normal next turn or an interruption of the agent's current reply.
2. Replies are spoken sentence-by-sentence, streamed. The LLM response is split into complete sentences as it streams in; each sentence is sent to TTS and enqueued for playback the moment it's ready, rather than waiting for the whole reply to finish generating. The person hears the first sentence while the model is still writing the third.
3. Speech is transcribed incrementally while you're still talking. A partial transcription re-runs every ~900ms during a recording. If nothing meaningful was said in the last moment before you stopped, the cached partial is reused instead of re-transcribing the full clip from scratch after the fact — the STT work is mostly already done by the time you finish your sentence.
4. Adaptive silence detection, not a fixed timer — long enough to survive a normal mid-sentence breath, short enough not to feel laggy on a quick reply (see SILENCE_MS/SILENCE_GRACE_PER_SPEECH_MS in src/audio-io.ts).
5. Small model, small token budget by default. maxTokens: 100 and a 0.5B-parameter model are both deliberate — this package optimizes for a snappy back-and-forth over long, detailed replies. Raise either if your use case needs more.

Browser requirements

• WebGPU (falls back to WASM automatically for STT if unavailable — TTS and the LLM engine currently require WebGPU-capable browsers via WebLLM/kokoro-js).
• Microphone permission (HTTPS or localhost).
• Everything downloads on first use and is cached by the browser afterward (WebLLM uses Cache Storage; transformers.js/kokoro-js use the HTTP cache), so a second warmup() — even after a reload — is near-instant.

License

MIT