jervis-voice-control

jervis-voice-control is a framework-agnostic voice automation engine for web applications with Arabic/English command parsing, semantic actions, DOM automation, routing, safety guards, sequences, Vue integration, runtime events, cancellation, async UI waits, and optional feature plugins such as Point Cloud.

Documentation

Full documentation: https://jervis-voice-control.surge.sh

Install

npm install jervis-voice-control

Vue integration is optional:

import { createJervisVoicePlugin } from 'jervis-voice-control/vue'

Point Cloud integration is optional:

import { installPointCloudVoicePlugin } from 'jervis-voice-control/point-cloud'

Core setup

import {
  DOMExecutor,
  RiskGuard,
  SequenceParser,
  WebSpeechProvider,
  createDomMutationWaitStrategy,
  createVueRouterExecutor,
  createVoiceControl,
} from 'jervis-voice-control'

const voice = createVoiceControl({
  speech: new WebSpeechProvider({ language: 'ar-SA' }),
  sequenceParser: new SequenceParser(),
  waitStrategy: createDomMutationWaitStrategy(),
  guard: new RiskGuard({
    confirmFrom: 'destructive',
    confirm: async (intent) => window.confirm(`Execute ${intent.target ?? intent.action}?`),
  }),
  executors: [
    createVueRouterExecutor(router),
    new DOMExecutor(),
  ],
})

voice.start()

The engine supports commands such as:

افتح المستخدمين
اضغط على حفظ
اكتب أحمد في الاسم
اختار Admin من الصلاحيات
انزل
ارجع

open users
click save
type Ahmed in name
select Admin in role

Runtime lifecycle

Typical states are:

idle -> processing -> executing -> idle
listening -> processing -> executing -> listening

Blocked operations remain in blocked so the application can present a confirmation or recovery path. Runtime failures emit error events and recover to a stable idle or listening state.

A newer transcript cancels the previous active execution by default. This follows a latest-command-wins policy.

voice.cancel()

Executors may cooperate with cancellation:

const executor = {
  async execute(intent, transcript, context) {
    if (context?.signal.aborted) return false
    await doAsyncWork({ signal: context?.signal })
    return true
  },
}

Registered semantic actions can also receive signal and executionId.

Runtime events

voice.events.on('execution:start', ({ executionId, transcript }) => {})
voice.events.on('execution:success', ({ executionId, intent }) => {})
voice.events.on('execution:incomplete', ({ executionId, reason }) => {})
voice.events.on('execution:error', ({ executionId, error }) => {})
voice.events.on('execution:cancelled', ({ executionId }) => {})
voice.events.on('sequence:complete', (report) => {})

execution:success is reserved for commands that were actually handled. Unknown commands, unhandled intents, and incomplete sequences emit execution:incomplete with a reason instead of being counted as success.

Status, transcript, intent, blocked, unhandled, and error events remain available as well.

Semantic actions

Registered actions are preferred over raw DOM automation:

voice.action({
  id: 'users.create',
  aliases: ['إضافة مستخدم', 'ضيف مستخدم', 'create user'],
  risk: 'interaction',
  handler: ({ signal }) => {
    if (signal?.aborted) return
    openCreateUserDialog()
  },
})

Vue Router discovery

Routes can expose voice aliases directly through metadata:

{
  path: '/users',
  name: 'users',
  component: UsersPage,
  meta: {
    voice: {
      aliases: ['المستخدمين', 'إدارة المستخدمين', 'users'],
      label: 'Users',
    },
  },
}

createVueRouterExecutor(router) discovers route names, paths, labels, and aliases automatically.

Vue plugin

import { createVoiceControl, SequenceParser } from 'jervis-voice-control'
import { createJervisVoicePlugin } from 'jervis-voice-control/vue'

const voice = createVoiceControl({ sequenceParser: new SequenceParser() })

app.use(createJervisVoicePlugin({
  voice,
  router,
  useDOM: true,
  useVuetify: true,
  autoStart: false,
}))

v-voice

<v-btn
  v-voice="{
    id: 'users.create',
    label: 'إضافة مستخدم',
    aliases: ['ضيف مستخدم', 'create user'],
    handler: openCreateUser
  }"
>
  إضافة
</v-btn>

Context-aware automation

import { VoiceContextRegistry } from 'jervis-voice-control'

const contexts = new VoiceContextRegistry()

contexts.register({
  id: 'create-user-dialog',
  priority: 10,
  root: () => document.querySelector('#create-user-dialog'),
})

contexts.activate('create-user-dialog')

Async UI waits

import {
  composeWaitStrategies,
  createDomMutationWaitStrategy,
  createTargetWaitStrategy,
} from 'jervis-voice-control'

const voice = createVoiceControl({
  sequenceParser: new SequenceParser(),
  waitStrategy: composeWaitStrategies(
    createTargetWaitStrategy({ timeout: 5000 }),
    createDomMutationWaitStrategy({ timeout: 1500, quietPeriod: 80 }),
  ),
})

Wait results are explicit:

success
timeout
aborted
error

A wait timeout stops the current sequence by default. Set waitFailureBehavior: 'continue' only when continuing after a missing/late UI condition is intentional.

Command sequences

Example:

افتح المستخدمين وبعدين اضغط إضافة مستخدم ثم اكتب أحمد في الاسم

Each parsed step is executed in order. By default, the sequence stops when a step cannot be handled or when a wait times out.

The sequence intent preserves the highest risk among its steps. A sequence completion report contains per-step states such as:

handled
unhandled
blocked
wait-timeout
wait-error
cancelled

Safety

Risk levels:

safe
interaction
write
commit
destructive
restricted

Use RiskGuard to require confirmation or block sensitive operations before any executor or semantic action runs.

Web Speech lifecycle

new WebSpeechProvider({
  language: 'ar-SA',
  restartOnEnd: true,
  restartDelayMs: 250,
  maxRestartDelayMs: 4000,
})

Restarts use bounded backoff. Fatal permission/microphone errors such as not-allowed, service-not-allowed, and audio-capture stop automatic restart instead of entering a restart loop.

Point Cloud plugin

import { createVoiceControl } from 'jervis-voice-control'
import { installPointCloudVoicePlugin } from 'jervis-voice-control/point-cloud'

const voice = createVoiceControl()

installPointCloudVoicePlugin(voice, {
  preset: (preset) => pointCloud.setPreset(preset),
  fit: () => pointCloud.fitCamera(),
  camera: (delta) => pointCloud.moveCamera(delta),
  timeline: (action) => action === 'play'
    ? pointCloud.play()
    : pointCloud.pause(),
})

Validation

npm run typecheck
npm test
npm run build
npm run e2e:browser
npm run test:consumer

Or run the full validation pipeline:

npm run validate

The browser E2E demo verifies an Arabic multi-step command through routing and DOM automation, checks the saved result, requires a successful sequence report, requires an execution-success event, and verifies that the final runtime state returns to idle. It also covers an expected failure path and verifies execution:incomplete without a false success event.

npm run test:consumer creates the real npm tarball, installs it into an isolated consumer project, and checks every public entry point plus TypeScript declarations with real framework types. The temporary consumer is removed automatically after the test.

Architecture

Speech Provider
      ↓
Parser Chain
      ↓
Structured Intent
      ↓
Risk Guard
      ↓
Semantic Actions / Executor Chain
      ↓
Wait / Sequence Runtime
      ↓
Lifecycle Events + Stable State

AI parser infrastructure exists in the codebase, but further AI integration is intentionally deferred while the deterministic runtime and browser execution path are stabilized.

Current status

Implemented:

• Web Speech API provider with restart/backoff handling
• Arabic normalization
• Arabic/English deterministic parser
• Dynamic parser chain
• Semantic action registry
• Dynamic executor chain
• Vue Router route discovery
• DOM target resolver and automation
• Context registry
• Vuetify executor foundation
• Smart target resolver / ambiguity foundations
• Multi-step sequence parser
• Async UI wait strategies
• Sequence execution reports
• Runtime state/event bus
• Latest-command cancellation and cooperative AbortSignal
• Central runtime error recovery
• Risk/confirmation guard and highest-risk sequence aggregation
• Vue plugin/composables/directive
• Point Cloud plugin migration
• Unit/runtime/Web Speech tests
• Browser E2E success and failure scenarios
• Framework-neutral installer, React, Svelte, Angular helpers, and Web Components
• Component adapters, numbered hints, discovery panel, and dynamic routes
• Packaged consumer smoke test for all public exports
• TypeScript declarations and ESM build
• GitHub Actions validation pipeline
• MIT license and changelog

Deferred / next:

• richer Vuetify autocomplete/dialog support
• final public API compatibility pass before 1.0
• AI integration expansion after core stabilization