@studio-dev/vsdk-plugin-ai-captions

AI auto-caption plugin for @studio-dev/vsdk. Extracts audio from any video/audio clip in the browser with MediaBunny, posts it to a TranscriptionAdapter, and attaches the result back to the source clip's own style.transcription. The SDK's video renderer reads that field and overlays captions automatically — no separate caption track item is created, so move/trim/split the clip and the captions follow.

Language is auto-detected. Audio clips store the transcription as data (no visual surface — export to SRT / VTT / JSON or render elsewhere).

Architecture

This package ships both halves of the feature in one install — browser plugin (default entry) and Node-only server kit (/server subpath). Browser bundlers only resolve .; server runtimes resolve /server/*.

| Subpath | Environment | What's inside | |---|---|---| | @studio-dev/vsdk-plugin-ai-captions | Browser | React plugin, MediaBunny audio extraction, HTTP adapter, timeline integration | | @studio-dev/vsdk-plugin-ai-captions/server | Node | createTranscriptionHandler, toNodeListener, types, errors | | @studio-dev/vsdk-plugin-ai-captions/server/assemblyai | Node | AssemblyAI Universal-2 provider | | @studio-dev/vsdk-plugin-ai-captions/server/openai | Node | OpenAI Whisper API provider | | @studio-dev/vsdk-plugin-ai-captions/server/elevenlabs | Node | ElevenLabs Scribe provider |

Surfaces

• Inspector tab on video and audio items
• Timeline topbar button (popover that captions every selected clip)

Install

# Plus one or more provider SDKs — each is an optional peer dependency.
pnpm add @studio-dev/vsdk-plugin-ai-captions \
  assemblyai openai @elevenlabs/elevenlabs-js

MediaBunny is bundled — no separate install required.

Browser — register the plugin

import {
  aiCaptionsPlugin,
  httpTranscriptionAdapter,
} from '@studio-dev/vsdk-plugin-ai-captions'

const plugins = [
  // ...
  aiCaptionsPlugin({
    adapter: httpTranscriptionAdapter({ endpoint: '/api/transcribe' }),
    defaultLanguage: 'en',
  }),
]

Server — mount the handler

Next.js Route Handler

import { createTranscriptionHandler } from '@studio-dev/vsdk-plugin-ai-captions/server'
import { assemblyAIProvider } from '@studio-dev/vsdk-plugin-ai-captions/server/assemblyai'
import { openAIProvider } from '@studio-dev/vsdk-plugin-ai-captions/server/openai'
import { elevenLabsProvider } from '@studio-dev/vsdk-plugin-ai-captions/server/elevenlabs'

const handler = createTranscriptionHandler({
  providers: [
    assemblyAIProvider({ apiKey: process.env.ASSEMBLYAI_API_KEY! }),
    openAIProvider({ apiKey: process.env.OPENAI_API_KEY! }),
    elevenLabsProvider({ apiKey: process.env.ELEVENLABS_API_KEY! }),
  ],
  defaultProvider: 'assemblyai',
  basePath: '/api/transcribe',
})

export const runtime = 'nodejs'
export const POST = handler
export const GET = handler
export const OPTIONS = handler

Also add the provider SDKs you use to serverExternalPackages in next.config.ts:

const nextConfig = {
  serverExternalPackages: [
    'assemblyai',
    'openai',
    '@elevenlabs/elevenlabs-js',
  ],
}

Express

import express from 'express'
import {
  createTranscriptionHandler,
  toNodeListener,
} from '@studio-dev/vsdk-plugin-ai-captions/server'
import { assemblyAIProvider } from '@studio-dev/vsdk-plugin-ai-captions/server/assemblyai'

const handler = createTranscriptionHandler({
  providers: assemblyAIProvider({ apiKey: process.env.ASSEMBLYAI_API_KEY! }),
  basePath: '/api/transcribe',
})

const app = express()
// IMPORTANT: mount BEFORE express.json() — handler reads multipart itself.
app.use('/api/transcribe', toNodeListener(handler))
app.use(express.json())
app.listen(4000)

Custom adapter (e.g. on-device Whisper.wasm)

import type { TranscriptionAdapter } from '@studio-dev/vsdk/contracts'

const myAdapter: TranscriptionAdapter = {
  async transcribe(audio, request) {
    // audio is a Blob (WAV or MP3 from MediaBunny)
    // return TranscribeResult with ms timing
  },
}

Plugin options

| Option | Type | Default | Description | |---|---|---|---| | adapter | TranscriptionAdapter | — | Required. Where to send audio. | | defaultLanguage | string | undefined | ISO-639-1 (e.g. 'en'). Omit for auto-detection. | | defaultProvider | string | undefined | Server-kit provider id. | | defaultCaptionStyle | Partial<CaptionsStyle> | clean preset | Style applied to created caption items. | | attachToVideo | boolean | true | Attach raw transcription to source style. | | createCaptionItem | boolean | true | Create a caption track + item. | | uploadFormat | 'wav' \| 'mp3' | 'wav' | MediaBunny output format. | | order | number | 5 | Inspector tab + topbar order. | | hideTopbarButton | boolean | false | Hide the topbar popover. | | hideInspectorTab | boolean | false | Hide the inspector tab. | | defaultGlossary | string[] | undefined | Custom vocabulary terms. |

Programmatic API

import { transcribeItem } from '@studio-dev/vsdk-plugin-ai-captions'
import { useStudioStoreApi } from '@studio-dev/vsdk/ui'

const store = useStudioStoreApi()
const result = await transcribeItem(store, itemId, {
  language: 'en',
  provider: 'openai', // optional override
  onProgress: (p, stage) => console.log(stage, p),
})
console.log(result.transcription.words.length)

Pro / paid gating

aiCaptionsPlugin({
  adapter,
  access: {
    useIsAllowed: () => useAuth().user.plan === 'pro',
    label: 'Pro feature',
    description: 'AI auto-captions are part of the Pro plan.',
    ctaLabel: 'Upgrade to Pro',
    onUpgrade: () => openPricingModal(),
    // Or fully custom UI per surface:
    components: {
      sidebar: MyUpsellCard,
      inspector: MyInspectorBanner,
      topbar: MyTopbarBadge,
    },
  },
})

Every surface (sidebar, inspector, topbar) renders the locked UI when useIsAllowed() returns false. See docs/plugins/ai-captions/access.

Custom style presets

aiCaptionsPlugin({
  adapter,
  customPresets: [
    {
      id: 'brand-yellow',
      label: 'Brand Yellow',
      description: 'Editorial caption style',
      category: 'brand',
      style: {
        fontFamily: 'Inter, sans-serif',
        fontSize: 64,
        fontColor: '#FACC15',
        backgroundColor: 'transparent',
        position: 'center',
        typography: { fontWeight: 800, textTransform: 'uppercase', strokeColor: '#000000', strokeWidth: 8 },
        highlight: { mode: 'color-swap', color: '#FFFFFF' },
      },
    },
  ],
  // replaceBuiltInPresets: true,  // optional — hide the built-in gallery
})

See docs/plugins/ai-captions/custom-presets.