@omelhorsite/video-sdk
v0.7.0
Published
The engine of the omelhorsite video editor: the .omsv project model, placement math, effects, karaoke captions, compilation to ffmpeg arguments, and the omelhorsite cloud client. Isolate-safe: no node builtins, no environment access, no stdout.
Maintainers
Readme
@omelhorsite/video-sdk
The engine of the omelhorsite video editor, as a library: the project model,
the math that puts every clip where it belongs, the effects, the karaoke
captions, the compiler that turns all of it into ffmpeg arguments, and the
client for the omelhorsite cloud.
bun add @omelhorsite/video-sdkIt is pure. Nothing here reads a file or spawns a process: files come in and
go out as Blob or Uint8Array, and rendering is a function that returns the
arguments for ffmpeg. That is what lets it run in Bun, Node 18+ and the
browser alike.
The project
A Project is { width, height, fps, assets[], tracks[] }. Tracks render
bottom to top in array order; a clip is { start, duration, in } in seconds on
the project timeline, and a visual clip carries a transform whose x/y are
the centre in normalised coordinates and whose scale multiplies the size the
asset gets when fitted inside the frame.
An .omsv file is a ZIP with that object and all of its media inside,
portable and self-contained.
import { emptyProject, packProject, unpackProject, isZipData } from "@omelhorsite/video-sdk";
const project = emptyProject();
project.assets.push({ id: "a1", kind: "video", name: "clip.mp4", path: "clip.mp4", duration: 12 });
project.tracks[1].clips.push({
id: "c1", kind: "video", assetId: "a1", start: 0, duration: 4, in: 2,
transform: { x: 0.5, y: 0.5, scale: 1, opacity: 1 }, fadeIn: 0, fadeOut: 0, muted: false, volume: 1,
});
const bytes = packProject({ project, assetBytes: new Map([["a1", clipBytes]]), nowIso: new Date().toISOString() });
const { project: again, assetBytes } = unpackProject(bytes);Editing in source time
People quote times from the raw files ("cut 1:09 to 1:11"), not from the edited timeline. These helpers speak that language:
parseTime("1:09.2"),formatTime(69.2)andparseRange("1:09-1:11").sourceAt(project, t)says which asset and which second of it plays at timelinet;timelineAt(project, assetId, s)goes the other way and returns every place that moment plays (a moment can repeat), andnearestKeptbrackets a moment the edit removed.removeSource(project, [{ assetId, from, to }])removes source ranges: clips of that asset are split around the hole and every track closes up behind it. Music and overlays that span the hole keep playing and end sooner; a flash follows its cut.buildFromSource(project, ranges)lays an edit list back to back on the track that plays those assets; everything on the other tracks stays pinned to the source moment it sat on.retrimClip(project, clipId, from, to)moves one clip's source window and ripples what follows.speechSpans(project)is what counts as recorded speech: unmuted video clips with their own sound, or the audio extracted from them. Free audio (music) and clips at a bed volume (SPEECH_MIN_VOLUME) never do.
Captions follow all of this on their own: the words live in source time.
Rendering
compile returns the ffmpeg argument list for the whole project in one
pass: layers, transforms, fades, text, effects and captions. Text arrives
rasterised, as PNGs the caller provides, so what was drawn is what gets
composited.
import { compile, projectDuration } from "@omelhorsite/video-sdk";
const { args } = compile(project, { output: "out.mp4", preset: "slow", crf: 17, textImages });
// spawn("ffmpeg", args)placement.ts is the one place that decides where a clip goes (fittedSize,
clipRect, visibleAt); a preview drawn with the same functions matches the
export by construction.
renderPlan resolves the same render into numbers, for a compositor that is
not ffmpeg: every layer's rectangle and draw position, its inclusive frame
window, the clip frame shown on each output frame, fades in whole frames,
blur and shadow sizes, and each flash frame's constants. The numbers are the
ones compile writes into the filter graph, including ffmpeg's rounding (fades
quantised to frames, overlay positions floored to even, the extra frame a clip
carries when it ends off the frame grid). What a plan renderer cannot draw is
listed in unsupported, so the caller can fall back to compile.
import { compileAudio, renderPlan } from "@omelhorsite/video-sdk";
const audio = compileAudio(project, { output: "mix.m4a" }); // null when silent
// spawn("ffmpeg", audio.args)
const plan = renderPlan(project, {
textImages,
captionOverlay,
audio: "mix.m4a",
probes: { [assetId]: { rotation: 0, frameRate: 25, transfer: "bt709", range: "tv" } },
});
if (plan.unsupported.length) {
// render with compile() instead
}compileAudio is the audio half of compile on its own: the same inputs and
filters, so the decoded PCM is identical.
Effects and captions
makeFlashClip(id, cutAt)builds the flash transition that hides a cut;DEFAULT_FLASHholds its parameters.- Captions live in the project in source time (
project.captions[].words,{ t0, t1, text }), so cutting or reordering never invalidates them.timelineWords(project)brings the words onto the timeline;groupWordsandcaptionStatesdo the karaoke grouping. A word belongs to the piece that holds its centre: a word said before a cut has it on one side, one said after has it on the other, and a few hundredths of drift in a speech model's timings cannot change that. sourceCaptions(project, words)is the way back, for captioning an edit rather than the raw material: give it words in timeline time and it returns them in source time, grouped by asset; when two voices overlap, a word goes to the louder one.audibleSpans(project)lists the pieces whose recorded speech is in the mix — a silent overlay or a music bed carries no words.project.captionDictholds spelling rules applied when captions are drawn (applyCaptionDict); the transcript itself is not changed. A key is a word or phrase matched ignoring case and accents, or a pattern with groups:{ "merda": "m**da", "portugu[êe]s(a|es|as)?": "portuguez$1", "dar te": "dar-te" }. Clitics a transcriber splits off ("dar" "-te") are joined anyway.- Words may carry a
speaker(andspeakerLockedfor a manual label), and the project lists them inspeakerswith an optionalmainSpeaker. A word with no speaker belongs to the main one;groupWords(words, style, main)never puts two speakers in one group, andspeakerColors(project, style)gives each one its highlight.assignSpeaker,mergeSpeakers,stampSpeakersandcarrySpeakersedit and propagate labels. buildReview,parseReviewReplyandapplyReviewreview captions against a model's answer without any network: phrases in timeline time, a JSON schema (REVIEW_SCHEMA) for the reply, andalignWords, which puts corrected text onto the existing words keeping the timing of the ones that match.censorWord/censorWordsmask profanity asc**alho.sliceProject(project, from, to)cuts a window out of a project for a fast partial render.- Faces and objects to hide live in
project.redactions(per asset, in source time, as tracks of boxes) andproject.redactsays which are hidden and how (blurorpixelate,strength,pad).redactBoxAtresolves a track at any instant, bridging gaps up toREDACT_BRIDGEand holding the boxREDACT_TAILseconds past the ends, moving and growing: it errs on the side of hiding too much.clipRedactionmaps that onto a clip frame by frame;renderPlanandcompileboth draw it, a face as a feathered ellipse around the head and an object as its padded box.redactAppearanceslists where each track shows on the timeline. - A track with
shape: "poly"is an outline per frame instead (a sheet of paper segmented on every frame):redactPolysAtunites the outlines withinREDACT_UNIONseconds, so one missed frame does not flash, and nothing is held or grown. A key'skeepOuthalf-planes keep the outlines borrowed from its neighbours off the faces of that frame. The plan carries the outlines (frames[].p);compilehides them through a grey mask per clip (redactMaskJobsdescribes it,redactMaskspasses it), or by their bounds without one. Held object boxes are cut back off the faces inavoidand in the face tracks of the same asset (keepOffFaces).
Sound, colour and review notes
- Loudness. A clip may carry
loudness(target LUFS, EBU R128) and an audio track may carryduck: { depth, attack?, release? }(dB the music drops while someone speaks). Both need the sources' momentary-loudness curves (LoudnessCurve: one 400 ms block every 100 ms), which the host measures once per file and passes asaudioAnalysistocompileandcompileAudio;loudnessAssets(project)lists the assets that need one. The integrated loudness of any trim comes out of the same curve (integratedLoudness), so trimming never needs a new measurement. The voices are the video clips' own sound plus audio tracks markedvoice;voiceIntervalsfinds where they speak andduckGainis the resulting envelope. An audio clip or track may instead carrybelowVoice(dB under the voice as heard,voiceLevel), so music and ambience follow the voice when its target changes;loudnessTargetresolves either kind. A clip withclean: truegetsVOICE_CLEAN_FILTERS(high-pass, gentle denoise, two small EQ moves, soft de-esser) before its gain, without touching the file. Every mix ends in a true-peak limiter, oversampled 4x, atMIX_TRUE_PEAK_DB(-1.5 dBTP), and comes out at 48 kHz. - Colour.
VisualClip.color(exposurein stops,contrast,saturation,warmth, all neutral at 0) is one affine map on Y'CbCr code values (colorAffine);compileapplies it withlutyuvand the render plan carries it per layer, so a plan renderer applies exactly the same map. - Notes.
project.notesholds change requests:{ id, t?, source?: { assetId, t }, text, status: "open" | "done", author?, createdAt?, doneAt? }. A note pinned tosourcefollows its clip through cuts and reorders;noteTimessays where it lands on the timeline (nowhere when that part was cut),openNoteslists what is left to do,parseTimecodereads"1:05.5".
The omelhorsite cloud
Cloud signs in with the device grant, keeps whole projects in the account's
storage, and runs transcription and captions on the server. It takes a
TokenStore for the tokens and, when the host needs it, a fetch.
import { Cloud, memoryTokenStore, describeCloudError } from "@omelhorsite/video-sdk";
const cloud = new Cloud({ store: memoryTokenStore() });
await cloud.login({ onPrompt: (p) => show(p.userCode, p.url) });
const saved = await cloud.uploadProject({ data: bytes, name: "promo" }, { onProgress: report });
const projects = await cloud.listProjects();
const { data } = await cloud.downloadProject(saved.id);
const tx = await cloud.transcribe({ data: wav, name: "voice.wav" }, { language: "pt" });
const captioned = await cloud.captions({ data: mp4, name: "render.mp4" }, { words: timelineWords(project) });stringTokenStore adapts anything that can keep a string (a file, a
keychain) into a TokenStore. describeCloudError turns any failure into a
sentence fit for a screen. Progress arrives per finished transfer, or per byte
where the runtime has XMLHttpRequest and no custom fetch was given.
Jobs can be started in one process and awaited in another:
transcribeStart/transcribeWait and captionsStartTranscribe/
captionsWaitTranscribe split the upload from the wait, and the wait polls at
a fixed interval (pollIntervalMs, 1 s by default), reporting each reading to
onStatus (state, percent, elapsed time, the machine and queue position when
the API says). When several processes share one token store, pass
refreshLock: the refresh runs inside it, re-reads the store first, and reuses
tokens another process already refreshed instead of spending a rotated
refresh token twice.
Developing
bun test # renders real mp4 files with the system ffmpeg
bun run typecheck
bun run check:isolate
bun run build