@unisim/media

MP4/ISO-BMFF containers and an on-device media pipeline for the Universal Apps. MIT. No ffmpeg, no GPL, no wasm download, no third-party CDN, no COOP/COEP.

npm install @unisim/media

Why it exists

Every current browser already contains an H.264 encoder and an AAC encoder, and exposes both through WebCodecs. What no browser gives you is a container: WebCodecs deals in frames, so nothing in the platform will hand you an EncodedVideoChunk out of an MP4, and nothing will turn chunks back into one.

That gap — not the codec — is what used to force ffmpeg.wasm into a project, and with it a GPL core, a 30.7 MiB .wasm that Cloudflare Pages will not even accept (25 MiB per-file limit), and SharedArrayBuffer cross-origin isolation. Filling the gap ourselves is ~1,300 lines and costs none of that.

The argument in full is §10.1, §10.2 and §10.6 of Docs_UNI_SIM/next-products.md. The one-line version: the licensing question and the code-sharing question are the same question — a GPL engine is only a problem because we want to share it, and sharing is exactly how one app's licence becomes four apps' licence.

What it does

| | | |---|---| | Read | MP4, M4V, MOV — non-fragmented ISO-BMFF. Stills through createImageBitmap | | Write | MP4 (H.264 + AAC) · M4A (AAC) | | One file in, one out | trim (keyframe-aligned), resolution cap, quality tier, drop audio | | Many files in, one out | a multi-clip timeline: cuts, gaps, stacked tracks, fades, crossfades, image cards, and a summed audio mix | | Many files out | a STORED .zip, reproducible byte-for-byte, no compression library — built in memory or streamed to a showSaveFilePicker() handle, Zip64 past 4 GB | | Refuses | MKV, WebM, AVI, WMV, fragmented MP4 — each by name, on drop, with a sentence | | Browsers | Chrome, Edge, Safari 16.4+. Not Firefox — it has no WebCodecs H.264 encoder; videoSupported() says so rather than failing mid-conversion |

Shape

| Module | What it is | |---|---| | mp4read | ISO-BMFF box walker; resolves stts/stsc/stsz/stco/stss/ctts into a flat sample list | | mp4mux | Two-track movie writer (vide + soun) | | mp4 | Audio-only M4A writer, plus the mp4a/esds sample entry both writers share | | box | The [size][type][payload] primitives | | aac | AAC frames from the browser's own AudioEncoder | | framesize | The sizing arithmetic — bits-per-pixel-per-frame, even dimensions for H.264 macroblocks | | video | The SINGLE-input pipeline: demux → decode → scale → encode → mux | | timeline | The edit document — types only, the contract between an editor UI and the renderer | | compose | The composition arithmetic: what is on screen when, at what alpha, for how long | | render | The MULTI-input pipeline: many clips → one MP4, picture and sound composed together | | probe | Header-only read: metadata off a 4 GB file without loading it | | plan | The memory ceiling — predict the output, arm or refuse the button before it is pressed | | trim | Clock parsing, trim windows, file naming | | zip | The STORED archive "download all" comes down as — hand-rolled, no dependency. createZip/zipBytes build it whole; openZip writes it entry by entry into a sink |

Everything except video and render is DOM-free and exercised by scripts/selftest.mjs without a browser. Those two are exercised by scripts/rendertest.mjs, which drives a real Chromium — see below.

The timeline

timeline.ts is the contract, and it turns on one decision: a clip carries its own audio. There is no separate audio clip sitting under a video clip, because two objects describing one piece of footage is how a cut ends up splitting the picture and not the sound. A cut splits one Clip into two, so the audio is cut at the same instant by construction rather than by remembering to.

import { renderTimeline, planTimelineRender, DEFAULT_TIMELINE_SETTINGS } from '@unisim/media'

const timeline = {
  width: 1920, height: 1080, fps: 30,
  sources: [
    { id: 'card', kind: 'image', name: 'intro.png', durationSec: 2, width: 1920, height: 1080, hasAudio: false },
    { id: 'a',    kind: 'video', name: 'walk.mp4',  durationSec: 40, width: 1920, height: 1080, hasAudio: true  },
  ],
  clips: [
    { id: 'c1', sourceId: 'card', inSec: 0, outSec: 2, startSec: 0, track: 0,
      audio: { enabled: false, gain: 1 } },
    { id: 'c2', sourceId: 'a', inSec: 12, outSec: 20, startSec: 1.5, track: 1,
      transitionIn: { kind: 'crossfade', durationSec: 0.5 },
      audio: { enabled: true, gain: 1 } },
  ],
}

const blob = await renderTimeline(timeline, new Map([['card', pngFile], ['a', mp4File]]))

What the renderer guarantees, and what the tests assert:

• Gaps render black and silent, for their own length. The movie is as long as the furthest a clip reaches — never the sum of the clips, because they overlap and sit past gaps.
• Higher track covers lower. A crossfade is two clips overlapping on different tracks, which is why "slide one video on top of another" and "cross dissolve" are the same gesture.
• durationSec is clamped to half the shorter participating clip. A 2 s crossfade between two 1 s clips is otherwise a transition with no clip left to show, and clamping is friendlier than rejecting.
• A crossfade with nothing to dissolve into becomes a fade from black. The intent is legible; refusing it mid-export would be worse than rendering it.
• A kind: 'image' source trims, stacks and takes transitions like footage. It contributes one still repeated for its span — that is the whole difference.
• The audio is cross-faded with the picture, from the same clamped numbers, so a dissolve cannot look like one thing and sound like another. Overlapping clips are summed and clamped rather than allowed to wrap.

⚠️ The alphas are not the opacities. Drawing two clips at 0.5 each, one over the other, gives the lower one a weight of 0.25 and loses a quarter of the frame to the black underneath — a dissolve that dips dark through its middle. drawPlanAt() solves for the alphas that reproduce the intended weights. This was measured, not reasoned about: rgb(67,23,115) at the midpoint of a red-to-blue crossfade, where rgb(125,30,125) was intended.

Refuse before, never crash after

An out-of-memory kill fires no onerror and rejects no promise. There is no recovery path, so the only defence is the refusal made before any work starts:

import { probeVideoFile, planConversion, convertVideo, DEFAULT_VIDEO_SETTINGS } from '@unisim/media'

const probe = await probeVideoFile(file)          // reads the header, not the file
const plan  = planConversion(probe, DEFAULT_VIDEO_SETTINGS)

if (plan.verdict === 'refuse') {
  show(plan.detail)                                // always names a setting that works
} else {
  await convertVideo(file, DEFAULT_VIDEO_SETTINGS, undefined, (p) => {
    show(`${p.framesDone} / ${p.framesTotal} frames · ${p.bytesOut} bytes so far`)
  })
}

What binds is the output, not the input. A short 4K clip re-encoded at Best quality can produce more bytes than the long file it came from. planConversion budgets input + 2 × output against a device budget — see the header of plan.ts for why both terms are what they are, and for three corrections to §10.4 found by reading the shipped code.

A timeline render is heavier, in ways a single conversion never is. Every video source is resident at once (a clip can start anywhere, so nothing can be dropped part way), every source's audio is decoded to float PCM and all of it is alive together while the mix renders, and there is one decoder per simultaneously-visible clip. planTimelineRender() budgets all four terms and timelineCost() reports them separately — because "which of these is the problem" is the only thing that tells a user what to change, and the two refusals are different sentences: a smaller frame fixes an oversized output, and nothing in the settings fixes too much footage imported.

Testing

npm test          # the arithmetic, in plain Node, in milliseconds
npm run test:render   # a real Chromium: compose, encode, read the MP4 back

scripts/rendertest.mjs exists because a renderer that compiles proves nothing. It renders four real MP4s — two clips butt-joined, two with a crossfade, an image intro/outro around a clip, and a clip cut in half — then hands the bytes back to the browser's own demuxer: <video> for the picture and decodeAudioData for the sound. It reads the middle pixel half way through the dissolve, because a cut and a dissolve produce files of identical length and only the colour can tell them apart. It found the alpha bug above.

Consumers

• Universal Converter — the video tab, and the M4A output on the audio tab.
• Universal Video — opensource.unisim.co.uk/video, the standalone front door for "compress a video without uploading it", and the editor that drives renderTimeline().
• Universal Compress and Universal PDF — for zip alone. Both bag up already-compressed files, which is the whole reason the writer stores rather than deflates.

zip arrived the same way the MP3 encoders did: four apps had grown four copies. Three were near-identical; PDF's was a separate implementation with a different signature and a DOS mod-date of 0 — day 0 of month 0, not a date. Both call shapes survive (createZip for Blobs, zipBytes for a contiguous buffer), because the Blob path hands the constructor un-concatenated parts and forcing one allocation the size of the archive would cost real gigabytes on a video batch.

openZip (0.6.0) is the third shape, and the one that changes what is possible. It takes a sink — a FileSystemWritableFileStream from showSaveFilePicker(), in practice — and writes one entry at a time, so the caller can release each source the moment it has been written:

const handle = await showSaveFilePicker({ suggestedName: 'pieces.zip' })
const writable = await handle.createWritable()
const zip = openZip({ write: (c) => writable.write(c), close: () => writable.close() })
for (const piece of pieces) await zip.add(piece.name, piece.blob)  // released after each
await zip.close()

Three things about it are load-bearing:

• It writes the same bytes createZip does, and the selftest asserts it. That is why an entry is read twice — once for its CRC, which the local header needs, and once for the data — rather than taking the usual streaming shortcut of a data descriptor. The same batch exported on Chrome and on Safari must not be two different files.
• Closing early is legal, and produces a valid archive of exactly the entries that landed. Universal Video relies on this: a batch whose fifth piece fails keeps the four before it instead of discarding them.
• Zip64 is written per entry, only where it is needed — an entry at or past 4 GB, an offset past 4 GB, or more than 65,535 entries. Below all three the output is the classic 32-bit archive this writer has always emitted, so nothing that reads today's zips stops being able to. Streaming is what made this necessary: an in-memory archive is bounded by the caller's memory budget long before 4 GB, and a stream to disk is not.

The files here moved out of Universal Converter (which shipped them in commit 2e0fabd); they were not rewritten. That was deliberate: this code carries fixes a clean-room rewrite would quietly drop — the AudioEncoder.isConfigSupported() lie for AAC (it reports true and then fails at exactly 32 kbps per channel, so support is established by encoding one real frame), the AVCC-not-Annex-B format choice, and the two-pass stco write.

Releasing

auto-release.yml covers this package: a version bump in packages/media/package.json landing on main tags media-v<version> and publishes it. The standing rule still applies — never npm publish by hand, it collides with the auto-release run.

Do not

• Add ffmpeg.wasm. §10.1 and §10.6 are the argument; the 25 MiB Cloudflare Pages per-file limit is the practical wall.
• Load an engine from a third-party CDN. The privacy claim is a trust claim, and it is falsifiable by anyone with the network tab open.
• Set COOP/COEP. require-corp blocks the SDK navbar's cross-origin org logos — a paying customer's branding silently vanishing — and the WebCodecs path is already off the main thread.

Licence

MIT © 2026 James Markey / Universal Simulation Ltd.