@jtdigital/renderbox-sdk

Typed TypeScript SDK for renderbox video processing. You describe a pipeline as a graph of sort-typed streams; renderbox's GPU/CPU workers run it. The SDK builds the graph and the job message — it does not process video itself.

npm install @jtdigital/renderbox-sdk

Building a pipeline

Open inputs, chain operations on the returned streams, attach outputs:

import { graph } from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('in.mp4');               // [VideoStream, AudioStream]

const out = v.scale({ w: 1280, h: 720 })       // ops are methods → autocomplete
             .fade({ type: 'in', d: 1 });

g.write('out.mp4', out, a);

Streams are sort-typed: VideoStream exposes video ops, AudioStream audio ops, DetectionStream tracking and filtering. A sort mismatch (piping audio into a video op) is a compile error, not a runtime one.

AI: perception as streams

A model's output is a stream you keep composing, not a JSON file you post-process. Detection returns a DetectionStream, which tracking refines and redaction consumes:

const g = graph();
const [v, a] = g.open('street.mp4');

const faces = v
  .detect({ model: 'retinaface_mv2' })    // VideoStream → DetectionStream
  .track({ algorithm: 'bytetrack' });     // stable ids across frames

g.write('redacted.mp4', v.redact(faces, { mode: 'black' }), a);

Each modality has its own sort, and they compose: segmentation feeds a masked blur, depth feeds a depth-aware blur, pose feeds a skeleton overlay.

const g = graph();
const [v, a] = g.open('clip.mp4');

const people  = v.detect({ model: 'dfine_x', filter_classes: 'person' });
const matte   = v.segment({ model: 'yolo11x_seg' });        // → SegmentationStream
const depth   = v.estimateDepth();                          // → DepthStream
const skeleton = v.estimatePose();                          // → PoseStream

const out = v
  .maskBlur(matte, { sigma: 30 })      // blur only what the mask covers
  .depthBlur(depth, { maxSigma: 12 })  // and fake a shallow depth of field
  .drawSkeleton(skeleton);
g.write('out.mp4', out, a);

The graph decodes once and forks: every branch above reads the same frames, so a second model costs inference, not another decode. That is the whole reason to describe perception as a graph rather than as a sequence of CLI passes.

Detect on the ORIGINAL frames, not on a burned-in one — a box found on a frame that already has a black rectangle over it is a box found on your own edit:

const faces  = v.detect({ model: 'retinaface_mv2' }).track({ algorithm: 'bytetrack' });
const plates = v.detect({ model: 'plate_detector' }).track({ algorithm: 'bytetrack' });
g.write('scrubbed.mp4', v.redact(faces, { mode: 'black' }).redact(plates, { mode: 'black' }), a);

redactPasses packages exactly that shape and owns the invariant for you.

Text and speech are streams too. recognizeText gives a TextStream you can redact by pattern, and transcribe gives a TranscriptStream you can write out:

g.write('no-pii.mp4', v.redactText(v.recognizeText(), { pattern: '\\d{3}-\\d{2}-\\d{4}' }), a);
g.writeTranscript('words.jsonl', a.transcribe(), 'jsonl');   // 'jsonl' keeps word timings

The registry types your call sites

Op options are generated from the engine's Rust op registry, so its rules show up as TypeScript errors instead of failed jobs:

v.redact(faces, { mode: 'blur', radius: 21 });  // ✓ blur requires a radius
v.redact(faces, { mode: 'black', radius: 21 });
//                               ~~~~~~ compile error — `radius` is the
//                               blur/pixelate strength; black forbids it

v.pad({ w: 1920, h: 1080, align: 'center' });   // ✓ layout sugar
v.pad({ w: 1920, h: 1080, align: 'center', x: 0 });
//                                         ~ compile error — `align` and
//                                         `x`/`y` are mutually exclusive

v.detect({ model: 'dfine_x', threshold: 1.5 });
// throws RangeError while building: detect: threshold=1.5 must be in [0, 1]

Optional params also accept an explicit undefined (stripped before the wire), so computed options stay flat — { filter_classes: classes || undefined }, no conditional spreads.

Composing pipelines — pipe and fold

The fluent chain covers pipelines whose shape is fixed in code. When the shape comes from data — user config, a DB row — use the combinators every stream carries. pipe applies transforms left to right; fold sequences a pipeline from an array, replacing let out; for (…) out = out.… loops:

const g = graph();
const [v, a] = g.open('cctv.mp4');

// kinds: e.g. ['face', 'plate', 'object'] — from the case record, not the code
const out = v.fold(kinds, (s, kind) =>
  s.redact(
    v.detect(detectOptsFor(kind)).track({ algorithm: 'bytetrack' }),  // detect on ORIGINAL frames
    treatmentFor(kind),                                               // burn accumulates on `s`
  ));

g.write('redacted.mp4', out, a);

For the privacy motif specifically, redactPasses packages that shape as pure data — and owns the detect-on-original-frames invariant:

import { redactPasses } from '@jtdigital/renderbox-sdk';

const out = redactPasses(v, kinds.map((kind) => ({
  detect: detectOptsFor(kind),
  redact: treatmentFor(kind),   // default: { mode: 'black' }, tracked with ByteTrack
})));

Multiple inputs

Combine ops are methods on the first stream; open more inputs on the same graph:

const g = graph();
const [v, a] = g.open('clip.mp4');
const [logo]  = g.open('logo.png');

g.write('branded.mp4', v.overlay(logo, { x: 20, y: 20 }), a);

open() takes source options, spelled the way the compiler reads them — which is not how ffmpeg's CLI spells any of them. loop is a boolean (a number emits nothing), the cut is duration (not t), and a still needs imageInput:

const [bed] = g.open('music.mp3', { loop: true, duration: 118.5 });  // -stream_loop -1 -t 118.5
const [still] = g.open('card.jpg', { loop: true, imageInput: true, duration: 4 });

Detections to a data sink

Not every output is a video — write structured results too:

const g = graph();
const [v] = g.open('traffic.mp4');
const cars = v.detect({ model: 'yolo11x', classes: 'car' }).track({ algorithm: 'bytetrack' });
g.writeDetections('cars.jsonl', cars);

A transcript is the same idea, and the format decides whether you keep the words. 'srt'/'vtt' flatten the transcript to sentence cues; 'jsonl' keeps the model's per-word tokens, which is what a karaoke sweep, a progressive caption reveal or a script-to-audio alignment is made of:

const [, a] = g.open('read.wav');
g.writeTranscript('words.jsonl', a.transcribe(), 'jsonl');

transcribe runs whole-file rather than streaming, so the artifact is one line holding { segments: [{ start, end, text, tokens: [{ text, start, end }] }] } — not one line per word. Read segments[].tokens[].

Recipes

One-liners for common pipelines. faceRedact, depthBlurPipeline, and makeSlideshow return a ready Graph; letterbox, snip, and redactPasses are stream transforms:

import { faceRedact, snip } from '@jtdigital/renderbox-sdk';

const g = faceRedact('in.mp4', 'out.mp4', { redactMode: 'blur' });

// or as a step in your own graph:
const [v] = graph().open('long.mp4');
const clip = snip(v, 30, 45);   // seconds 30–45

Presets

Opinionated building blocks with production defaults — smaller than a recipe, bigger than an op. Audio: masterAudio (podcast / music / broadcast / youtube mastering chains), noiseGate, limiter, radioVoice, telephone, ducking, audioMix, channelMapPreset, and synthesized notification / ambience / drone. Text: title, caption, textWatermark, lowerThird on top of a positioned text atom. Video: xfadeChain (automatic crossfade offset math), concatChain, fadeBookend, and kenBurns / kenBurnsChain motion:

import { graph, masterAudio, lowerThird, xfadeChain } from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('interview.mp4');

const video = lowerThird(v, 'Ada Lovelace', 'Analyst', { startTime: 2, endTime: 8 });
const audio = masterAudio(a, 'podcast');   // speechnorm → deesser → compressor → -16 LUFS → 48 kHz

g.write('out.mp4', video, audio);

Captions: fonts, documents and safe areas

renderPillAss turns transcript words into an ASS document; the rest of the caption surface is what a generator of your own needs — measured against the render hosts' own font files, so a size fitted here is the size libass draws:

import {
  graph, captions, face, assFontSize, measureText, TIKTOK, INSTAGRAM_REELS, safeInner,
  assDocument, assText, assColor, wrapWords, place, pillLayers,
} from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('ad.mp4');
const poppins = face('poppins-medium');            // on the fleet; your own: stageFace(g, s3Key, measuredJson)
const fs = assFontSize(poppins, 60);               // \fs for 60 px glyphs — the usWin box, not the em
const wide = measureText(poppins, 'Meet gypsy pages', 60) > safeInner(TIKTOK.safe, 1080, 1920).w;

const doc = assDocument({ playRes: [1080, 1920], wrapStyle: 2 })
  .style({ name: 'Pill', fontname: poppins.fontname, fontsize: fs, borderStyle: 3, outlineWidth: 18, outline: '#00FCA5', primary: '#000000', alignment: 7 });
const { lines, widths } = wrapWords(poppins, 'Meet gypsy pages 40 days', { maxWidthPx: 823, emPx: 60, maxLines: 2 });
const at = place({ playRes: [1080, 1920], safe: INSTAGRAM_REELS.safe, anchor: 'bottom', face: poppins, emPx: 60, widths, leading: 1.1, pad: 18 });
lines.forEach((line, i) => doc.events(pillLayers({
  start: 0.3, end: 2.4, style: 'Pill',
  text: assText(line).words((w) => /\d/.test(w), `\1c${assColor('#9900FC')}`, `\1c${assColor('#000000')}`),
  // the anchor goes in the text as {\an7\pos(x,y)}; see place()
})));
g.write('out.mp4', captions(v, doc, { playRes: [1080, 1920] }), a);   // validates PlayRes + faces, attaches the sidecar

const { inputs, sidecars } = g.io();               // upload each sidecar's content to its s3_key first

FACES carries 62 measured faces and FLEET_FAMILIES the 43 families every render host mirrors from one manifest (Anton, Bebas Neue, Montserrat, Poppins, Oswald, Inter, Roboto, Open Sans, Playfair Display, …; every variable font shipped as static weights), with Noto behind them for any other script; PLATFORMS carries each platform's geometry, duration cap and the safe area its UI covers, measured from the platforms' overlay templates.

Composing a video

Two ways to produce a finished cut, and real consumers use both.

The quick path: a named pipeline

Thirty server-side pipelines already do the common shapes — slideshow, montage, blur-fill, auto-subtitle, face-redact. pipelineArgs types the args against the exact structs the Rust builders parse, so you submit { pipeline_id, args } without hand-rolling a graph. This is how jt-cut renders every project:

import { pipelineArgs } from '@jtdigital/renderbox-sdk/pipelines';

const job = {
  job_id: myApp.newJobId(),
  tenant_id: 'jt-cut',
  ...pipelineArgs('slideshow', { slides, music: 'bed.mp3', audio_mastering: { target_lufs: -14 } }),
  input_files,
  output: { bucket, key: outputKey, format: 'mp4' },
};

The composed path: build the cut yourself

When the edit is the product — per-shot timing, transitions you choose, captions that move with the copy — build the graph. This is what everyframe-composer does to turn one script into many ads. The shape, reduced to its spine:

const g = graph();

// One open per DISTINCT key. `open` does not deduplicate, so opening the same
// object twice puts two decoders on one input and FFmpeg exits 254.
const opened = new Map<string, [VideoStream, AudioStream]>();
const open = (key: string) => opened.get(key) ?? (opened.set(key, g.open(key)), opened.get(key)!);

// Each shot: its slice of its source, conformed to one master geometry so the
// segments are joinable at all (xfade and concat both demand identical frames).
const segments = shots.map((shot) =>
  open(shot.key)[0]
    .trim({ start: shot.in, end: shot.in + shot.len })
    .setpts({ expr: 'PTS-STARTPTS' })          // never optional after a trim
    .fps({ fps: 30 })
    .scale({ w: 1080, h: 1920, force_original_aspect_ratio: 'increase' })
    .crop({ w: 1080, h: 1920 })
    .setsar({ sar: '1' }));

// Join them. All blends → `xfadeChain`; all hard cuts → `concatChain`.
const video = xfadeChain(segments.map((stream, i) => ({ stream, duration: shots[i].len })), 'fade', 0.4);

// Silence is the base of the audio bed, not the voice: `amix(duration: 'first')`
// then takes its length from a stream whose length YOU chose, so a missing
// voiceover shows through as silence instead of truncating the cut.
const bed = g.anullsrc({ d: totalSec });
const audio = bed
  .amix([voice, music.volume({ volume: 0.18 })], { duration: 'first', normalize: false })
  .loudnorm({ I: -14, TP: -1.5, LRA: 11 })
  .aresample({ rate: 48000 });     // loudnorm emits 192 kHz; AAC clamps to 96

// One sink per job. Crop to the format FIRST, then burn captions — a caption
// drawn on the master is cropped off the frame it was laid out for. And
// `format('yuv420p')` before every sink: xfade negotiates 4:4:4, which Safari
// and iOS will not play.
const framed = video.scale({ w: 1080, h: 1350 }).crop({ w: 1080, h: 1350 });
g.write(outputKey, captions(framed, doc).format({ pix_fmts: 'yuv420p' }), audio);

Mixing cuts and blends is the case neither recipe covers, and it has a rule. Feeding a concat output into an xfade fails on the fleet with exit 234 (FADE,CUT, FADE,FADE and all-cuts each succeed; CUT,FADE does not). So fold each blended run into its own xfadeChain, then join the chains with ONE concat at the end — no xfade ever takes a concat as input. Each chain times from its own zero, because an xfade offset is relative to the stream it is cutting into, not to the whole timeline.

Every comment above is a fleet failure someone already paid for. The four that cost the most: dedupe your opens, never feed a concat into an xfade, yuv420p before a container sink, and aresample after loudnorm.

Motion: compositions on a clock

@jtdigital/renderbox-sdk/motion writes a timed composition, such as a looping display video, as one graph. A composition is a display, a clock and a table of scenes. Layers are values: text sized by its cap height, shapes, images, clips and stills. show() says when a layer shows and how it moves. Every animated value is a track. build() samples each track once a frame at build time. It lowers the whole composition to stock FFmpeg filters and ASS documents, which libass draws. The program never reads a clock, and it has no per-frame code.

import {
  DISPLAYS, DISSOLVE, EASE, MOVE, asset, build, clip, color, composition, displayLoop, move, rect, shape, text, tween,
} from '@jtdigital/renderbox-sdk/motion';

const ink = color('#F1F4F9');
const stand = composition({
  display: DISPLAYS.stand34, // 3440 × 1440, and the least cap heights legible on it
  fps: 30,
  duration: 12,
  loop: true, // the last frame runs on into the first
  ground: color('#0A0C10'),
  scenes: { intro: { from: 0 }, demo: { from: 6, enter: DISSOLVE.dip } },
});

stand.scene.intro.add(
  text('Every frame, checked.', { type: { face: 'anton', caps: true }, capPx: 190, color: ink, x: 160, capTop: 120, role: 'headline' })
    .show({ from: 0.4, enter: move(MOVE.rise, { dy: 40 }) }),
  shape.rect({ rect: rect(160, 400, 1200, 8), w: tween(0, 1200, { at: 0.8, dur: 1.2, ease: EASE.outCubic }), fill: ink }),
);
stand.scene.demo.add(
  clip(asset('renders/demo.mp4', { size: [1920, 1080] }), { rect: rect(1720, 120, 1560, 878), range: [0, 6], start: 6, radius: 24 }),
);

const built = build(stand);
for (const issue of built.lint) console.warn(issue.check, issue.message);
displayLoop(built, { key: 'renders/stand.mp4' }); // H.264 for a player that loops the file
// built.graph is a Graph like any other. Submit built.graph.json() with built.graph.io().

Layers and timing. text, shape (rect, circle, line, path, arc), image, clip and stills each make a layer at rest. A layer shows for its whole scene unless show() says otherwise. from and until bound it. enter and exit bring it in and take it out, with a move, a wipe or a dissolve. pose moves it in between, and origin is the point a scale works about. group() makes layers that show and move as one.

Tracks. key() lists keyframes, each with its own ease. tween() runs from one value to another, series() gives one value a frame (a detector's count, say), and repeat() loops a track on a period. passes() and lit() turn a track into the moment it passes a value, or the span while a part plays. mix() blends two colours on a track. counter() shows a number track as text, formatted for a locale by numberFormat().

Type. A text is sized by capPx, the height of a capital. It is anchored by its cap top, its baseline, or where its first glyph's ink starts. The fleet's subtitles filter sets text without kerning, so the emitter writes each kerning pair itself. tracking is CSS's letter-spacing, and figures: 'tabular' is CSS's tabular-nums. A run() gives part of a line its own colour, turns it, or sets it in a box of its own.

Clips. A clip plays range from start. before and after say what shows around it. crop names the source pixels that show, and pan moves the crop on the clip's own frames to follow a target. fit, focus, radius, fill, shadow and window place the picture as CSS would. when() and project() carry the clip's own time and pixels onto the composition.

Components and brand. defineBrand() turns hex colours and type styles into typed tokens, so a colour name that does not exist fails to compile. typeBlock(), eyebrow(), progress() and tag() are the recurring parts of a card. trackOverlay() draws a tracker's record over the clip it tracked, with a magnified view that follows the target.

Output and checks. displayLoop() encodes H.264 High for a player that loops the file. It puts a keyframe every 2 s and none on scene cuts, tags the colour BT.709, and states the level that fits the size and rate unless you name one. contactSheet() tiles chosen frames, and build(c, { only: [...] }) builds just those frames. built.lint checks the loop's seam, each text's cap height against the least its role needs on the display, and every character against its face.

The Dan inovativnosti stand video (72 s at 3440 × 1440) is one such program, in ts-sdk/examples/stand-a of the renderbox-sdk repository. Its checks render it and compare every beat with the hand-built original.

Running your pipeline on renderbox

A Graph is a description. To actually render it, submit it as a job to renderbox infrastructure — the orchestrator compiles it and dispatches it to a GPU or CPU worker, which streams back progress and a completion result.

You reference inputs and outputs by S3 key. Two calls give you everything the worker needs: g.json() is the graph_ir, and g.io() is the S3 ⇄ storage binding (which object to download for each input, where to upload each output). The SDK assigns no identity — the job id, tenant, and user come from whoever submits (your app/auth, or the managed client).

import { graph, type ProgressMessage, type CompletionMessage } from '@jtdigital/renderbox-sdk';

const g = graph();
const [v, a] = g.open('uploads/raw.mp4');           // reference inputs by S3 key
g.write('renders/out.mp4', v.scale({ w: 1920, h: 1080 }), a);

const io = g.io();
// io.inputs  → [{ s3_key: 'uploads/raw.mp4', filename: 'raw.mp4', content_type: 'video/mp4' }]
// io.outputs → [{ s3_key: 'renders/out.mp4', filename: 'out.mp4' }]

// Assemble the job with identity from YOUR context, then publish over AMQP.
// (The message omits pipeline_id so the orchestrator routes on graph_ir.)
const job = {
  job_id: myApp.newJobId(),      // assigned by your app/DB — not the SDK
  tenant_id: ctx.tenantId,       // from auth
  graph_ir: g.json(),
  input_files: io.inputs,
  output: { bucket: MY_BUCKET, key: io.outputs[0].s3_key, format: 'mp4' },
};
await channel.publish(EXCHANGE, ROUTING_KEY, Buffer.from(JSON.stringify(job)));

output is not optional in practice. The envelope carries exactly ONE OutputSpec, and a job without it "promises nothing: only diagnostics artifacts" — it renders and uploads no file, reporting success. So a graph carries one container sink per job: io.outputs is your check, not a list to iterate. Several formats means several jobs, each with its own output.

You never write filenames by hand: open('uploads/raw.mp4') gives the graph a sandbox-safe basename (raw.mp4) and records the S3 key for you — the worker rejects slashes in the sandbox, which is why the graph and the binding differ.

Then consume results off your result queue. Progress arrives as the render proceeds; completion carries the output location(s):

function onProgress(msg: ProgressMessage) {
  // percent is null until it can be computed (null ≠ 0%)
  console.log(`${msg.job_id}: ${msg.percent ?? '—'}% @ ${msg.fps} fps`);
}

function onCompleted(msg: CompletionMessage) {
  if (msg.status === 'success') {
    for (const out of msg.outputs) console.log(out.kind, out.s3_key, out.bytes);
  }
}

Prefer a managed client?

A managed client (@renderbox/client) wraps the renderbox SaaS HTTP API — API-key auth, asset upload, job submission, SSE progress streaming and result download — so you can hand it a Graph instead of running your own transport. It is not published yet, and its working copy still pins this SDK at >=0.3.0, so build against the AMQP path above until that changes.

Surface

• graph() → Graph — inputs: open (opts: loop / imageInput / duration / seekTo) / openVideo / openAudio / rtsp / hls / mqtt / camera / screenCapture / slot, plus synthetic sources. Outputs: write / writeVideo / writeAudio / writeDetections / writeTranscript ('srt' | 'vtt' | 'jsonl' — only 'jsonl' keeps per-word timings) / writeMqtt / writeHttp / outputSlot. Sidecars: attach / attachContent. Terminals: json() (graph_ir) and io() (S3 ⇄ sandbox bindings, sidecars to upload).
• Stream methods — 210+ typed operations, one per engine op, on the sort it applies to (scale, crop, fade, detect, track, transcribe, overlay, redact, depthBlur, …), plus the pipe/fold composition combinators.
• Presets — audio dynamics/mastering/synthesis (masterAudio, noiseGate, ducking, notification, …), text styles (title, caption, lowerThird), video sequencing (xfadeChain, kenBurnsChain, fadeBookend).
• Recipes — letterbox, snip, faceRedact, redactPasses, depthBlurPipeline, makeSlideshow.
• Captions — renderPillAss (words in), captions (document in, validated), the authoring layer (assDocument, assText, place, pillLayers, wrapWords, fitSize, wordWindows), the ASS primitives (assColor, assStyleLine, assDialogueLine, entranceTags, isEmphasised, groupWords), fonts (FACES, assFontSize, measureText) and presets/platform (PLATFORMS, safeInner, checkDuration).
• Motion (/motion) — composition with its scenes. Layers: text, shape, image, clip, stills, group. Tracks: key, tween, series, repeat, passes, lit, mix. Transitions: wipe, dissolve, DISSOLVE. Layout: row, columns, capStack, space. Components: typeBlock, eyebrow, progress, tag, counter, trackOverlay and defineBrand. Then build, and the sinks displayLoop, repeated and contactSheet.
• Named pipelines — typed args for all 30 server-side pipelines (pipelineArgs('face-redact', { … }) → the { pipeline_id, args } slice of a JobMessage), generated from the exact serde structs the Rust builders parse.
• Wire types — GraphJSON, OpNode, JobMessage, ProgressMessage, CompletionMessage, FailureMessage.

License

Apache-2.0 © JT Digital d.o.o.