@eventual-ai/mcap-viewer

A TypeScript browser SDK for playing native H.264/H.265 video and timestamped samples from streamed MCAP recordings. /core owns playback and independent renderers; /data-loader handles the MCAP Delivery API (MDA). The package root exports both.

npm install @eventual-ai/mcap-viewer@alpha

import {
  McapRecording,
  Player,
  ImageRenderer,
} from "@eventual-ai/mcap-viewer/core";

const recording = await McapRecording.open(url, { signal });
const player = new Player(recording);
const renderer = new ImageRenderer(canvas, { imageTopic: "head_left" });
player.subscribe(renderer);
await player.seek(recording.start);
await player.play();

open reads files, URLs and custom streams progressively. Messages must be ordered by logTime; unordered input fails. Do not await recording.loaded before playback. Observe loading and late errors with recording.subscribe. When unmounting, call player.close(), then renderer.destroy(), then recording.close().

A listener attached during buffering receives the current frame snapshot when playback leaves buffering. Late subscribers report snapshot failures through the player error handler; their asynchronous decoder flush does not pause existing listeners. Seeking waits for all participating listeners to finish their snapshot.

The core delivers every topic at its native timestamp. Video topics use eventual.mda.v1.CompressedVideo: native H.264/HEVC packets with their decode order, in MCAP with embedded Protobuf schemas and, for spans over 20 MiB, ZSTD-compressed chunks. Recordings can contain annotations or other samples without video. Times remain bigint nanoseconds.

import { DataLoader } from "@eventual-ai/mcap-viewer/data-loader";

const loader = new DataLoader({
  baseUrl: MDA_URL,
  headers: { Authorization: `Bearer ${token}` },
});
const topics = await loader.discover(datasource);
const [result] = await loader.manifest({
  datasource,
  topics: ["head_left"],
  spans: [{ producer, start: 10, end: 20 }],
});
const recording = await loader.openSpan(result, { signal });

Each manifest supplies one streamed MCAP URL. Payload requests omit control credentials. Playback covers the requested span, which the MCAP records in its mda metadata, so McapRecording.open on a saved file starts at the same time (bounds: null keeps the keyframe preroll). discover maps topic names to canonical schemas and actual wire encodings:

{
  "head_left": {
    "schema": "eventual.mda.v1.CompressedVideo",
    "schema_encoding": "protobuf",
    "message_encoding": "protobuf"
  }
}

capabilities lists canonical schemas with status: "live".

Video requires WebCodecs over HTTPS or localhost; available codecs depend on the browser and device. Unsupported codecs report an error through renderer.onError.

Using offline or in a sandboxed page

The core plays plain MCAP files without a backend. Open a Blob or File, a URL, or a custom McapStreamSource:

// A file the user picks or drops, or any Blob, such as one bundled with the page.
input.addEventListener("change", async () => {
  const recording = await McapRecording.open(input.files![0]);
});

// A file next to the page, when the host serves .mcap files.
const recording = await McapRecording.open(new URL("clip.mcap", location.href));

A URL is fetched without cookies or credentials, unless you open new HttpSource(url, { credentials: "include" }). In a sandboxed iframe without allow-same-origin, the page has an opaque origin, so even a same-origin URL needs CORS headers; open a Blob instead when the host does not send them.

For a host that serves only text files, store the MCAP as base64 parts and open them with Base64Source, which decodes each part as it arrives:

base64 -i clip.mcap | split -b 1000000 - clip.mcap.b64.

import { Base64Source } from "@eventual-ai/mcap-viewer/core";

const recording = await McapRecording.open(
  new Base64Source(["clip.mcap.b64.aa", "clip.mcap.b64.ab"]),
);

Base64Source fetches the parts with credentials: "same-origin", so a host that serves a page's own files only with its session still serves them. Pass { credentials } as its second argument to change that.

Opening makes no other network request: the ZSTD decompressor is WebAssembly embedded in the package and compiled from its bytes, so a Blob opens under connect-src 'none'. A Content Security Policy must allow WebAssembly compilation and, while Protobuf decoding generates code at runtime, eval:

Content-Security-Policy: script-src 'self' 'wasm-unsafe-eval' 'unsafe-eval'

Pages without a bundler can import dist/mcap-viewer.standalone.js, which holds /core and its dependencies in one ES module: copy it next to the page and import { McapRecording, Player, ImageRenderer } from "./mcap-viewer.standalone.js".

A file exported by MDA plays the requested span, recorded in its mda metadata as start_ns and an exclusive end_ns; the file starts at the preceding keyframe, which is decoded but not shown. bounds replaces that span, and bounds: null plays every message:

const span = await McapRecording.open(file); // the span in the mda metadata
const all = await McapRecording.open(file, { bounds: null }); // every message
const part = await McapRecording.open(file, {
  bounds: { start: span.start, end: span.start + 999_999_999n }, // the first second
});

TimeBounds.end is inclusive, so a span with metadata end_ns has end === end_ns - 1n. A file without mda metadata plays from its first to its last message, and its end grows while it loads.

Camera overlays and native samples

Player also delivers nonvideo topics at their native MCAP timestamps. Seek snapshots include their history over the selected video decode interval, plus the preceding sample, so overlays can follow the displayed frame through seeks and reordered video. Video dependency buffering remains unchanged.

ImageRenderer draws only video. ImageAnnotationsRenderer independently draws eventual.mda.v1.ImageAnnotations on its own canvas. Applications choose the canvas dimensions in annotation pixel coordinates and subscribe it to the desired topics. No video channel, camera metadata, or WebCodecs decoder is required:

import { ImageAnnotationsRenderer } from "@eventual-ai/mcap-viewer/core";

annotationCanvas.width = imageWidth;
annotationCanvas.height = imageHeight;
const annotations = new ImageAnnotationsRenderer(annotationCanvas);
player.subscribe(annotations, { topics: [annotationTopic] });
await player.seek(recording.start);

The annotation renderer selects the latest complete timestamp group at or before the player time. Empty messages clear the preceding annotations. maxAgeNs optionally limits how long a sample remains visible. Call destroy() when unmounting, after unsubscribing or closing the player.

Applications own composition: place the canvases side by side, stack a transparent annotation canvas over video, or show annotations alone. For a stacked view, forward annotation messages and snapshot callbacks without forwarding player state, then call annotations.draw(time) from the video's onDraw(time) hook. Match the annotation canvas dimensions to the video canvas before drawing. This aligns annotations to the frame actually drawn, including reordered video and seeks. Use the video's onClear hook to clear the annotation canvas when no frame is shown. Neither renderer owns the other; this coordination lives in the application, as shown in examples/viewer.js.

ImageAnnotationsRenderer.render(context, message) also draws primitives into an application-owned canvas without selecting samples or clearing it. This supports explicit shared-canvas composition. It supports styled dots (type: 1) and independent line segments (type: 4). The client does no projection or hand-specific geometry. Exported HandPose and HandPose2D types describe native messages for application views.

The example requests the selected video topic and only the comma-separated annotation topics entered in the form. Its episode field accepts a full named producer key as JSON, as well as an opaque string producer. DataLoader passes the key through unchanged.

The example form can be prefilled with url, datasource, recording, topic, annotations, start, and end URL query parameters. recording may contain a JSON producer key. Prefilling never reads a token parameter and does not load automatically.

CameraCalibration and FrameTransform are also exported decoded-message types. Calibration matrices retain the protobuf MatrixRow[] shape (row.values). Missing pose positions/orientations and unprojectable 2D positions remain null; optional 2D depth remains absent when unset. Schema-name constants are exported alongside these types for application dispatch without topic-name assumptions.

Text intervals

TextLabelView independently subscribes to eventual.mda.v1.TextLabel topics. Its activeLabels and onChange(labels) expose every active label with its topic, value, and complete bigint start/end bounds. The application owns rendering and layout. Activity is start <= time && time < end; overlapping and equal-time labels remain distinct. The wire TextLabel keeps start_time and end_time as decimal strings. No interval is clipped to the requested playback span.

const labels = new TextLabelView();
labels.onChange = active => renderLabels(active);
player.subscribe(labels, { topics: ["subtasks/caption", "subtasks/state_changes"] });

Seeks reconstruct all overlapping text intervals, including long intervals whose start precedes later labels. Playback expires labels at their exclusive end even when no new message arrives. reset() clears a view for reuse; destroy() ignores later callbacks. Text labels are independent of video and annotation renderers.

The example accepts explicit comma-separated label topics in its labels field or URL query parameter, defaulting to subtasks/caption. With video selected, labels appear in a separate subtitle overlay. The Subtitles button toggles its visibility without fetching or decoding again. Clear the optional video topic for labels-only playback; the separate label list uses the same player clock and controls.

Standalone annotation playback requires an explicit annotation image width and height in native pixel coordinates. Those fields appear only when annotation topics are selected without video; annotation-width and annotation-height query parameters can prefill them. The example hides the unused video canvas. Video overlays continue to size from decoded video frames, without using these standalone dimensions or fetching calibration data.

Training transport metadata

Manifest results can include content_id and expires_at (Unix seconds). These fields are optional in TypeScript for compatibility with older MDA deployments. capabilities().limits reports the maximum spans, topics, span-topic pairs and span duration when supported.

Use content_id for exact-recording cache identity, not the signed URL. It is not a checksum of the downloaded bytes. Request the same manifest again to replace an expired URL. The server resolves the current data; if content_id changes, download and cache the new recording under that ID.

HttpError, exported from both /core and /data-loader, exposes status, body and retryAfterMs on failed control or payload HTTP requests. The retry delay supports both seconds and HTTP-date Retry-After headers. The client does not retry automatically; callers can apply their own bounded backoff with jitter and cancellation. Payload requests continue to omit control credentials.

License

MIT, except the embedded Zstandard decompressor, which is BSD-3-Clause; see LICENSE.