SpatialJS

SpatialJS is the WebAssembly version of SpatialStudio for creating and editing Spatials (4D videos) from JavaScript/TypeScript environments (e.g. browsers, Node.js).

For more info check out https://true3d.com/.

Installation + Getting Started

Here's how to install SpatialJS:

npm install spatial-player

Here's how to import it into your project to use the web components:

import 'spatial-player/index.js';

Here's a barebones example using the API to create your first spatial:

import SPLV from 'spatial-player/index.js';

// set up virtual filesystem (indexedDB):
SPLV.FS.mkdir('/persistent');
SPLV.FS.mount(SPLV.IDBFS, {}, '/persistent');

await new Promise((res, rej) =>
	SPLV.FS.syncfs(true, (err) => (err ? rej(err) : res()))
);

// create encoder:
const width = 128, height = 128, depth = 128;
const encoder = new SPLV.Encoder(
    width, height, depth, 30.0, "my_spatial.splv"
);

// generate frames (moving red cube):
for(var i = 0; i < 100; i++)
{
    const frame = new SPLV.Frame(width, height, depth);

    frame.fill(
        i     , i     , i     ,
        i + 20, i + 20, i + 20,
        { r: 255, g: 0, b: 0}
    )

    encoder.encode(frame);
}

// finish encoding:
encoder.finish()

// download file as blob:
const data = SPLV.FS.readFile("my_spatial.splv");
const blob = new Blob([data], { type: 'application/octet-stream' });

const a = document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = "output.splv";
a.click();

Web Components

<splv-player>

A custom HTML element for rendering and interacting with .splv files.

Attributes

| Attribute | Type | Description | | ---------------- | -------- | -------------------------------------------------------------------------- | | src | string | URL to a .splv file (binary). Automatically fetched and loaded when set. | | video-controls | string | Controls video UI display: "show", "hide", or "hover". | | bbox-width | string | Bounding box line width. | | top-color | string | RGBA string (e.g. "255 255 255 255") for top gradient color. | | bot-color | string | RGBA string for bottom gradient color. | | camera | string | Camera type: "default", "preview", or "boomerang". |

Methods

get_metadata() → object | null

Returns metadata for the currently displayed spatial frame.

get_camera() → object | null

Returns current camera parameters.

spatial_set(spatial: ArrayBuffer)

Replaces current spatial sequence with a new .splv.

spatial_queue(spatial: ArrayBuffer)

Adds a new spatial segment to the playback queue.

set_volume(vol: number)

Sets audio playback volume (range: 0.0 to 1.0).

audio_context_running()

Returns whether the audio context is currently running.

resume_audio_context()

Attempts to resume the audio context, if it is suspended.

set_playing(value: boolean)

Plays or pauses playback.

set_scrubbing(value: boolean)

Enables or disables scrubbing mode.

set_scrubber_position(position: number)

Updates the scrubber UI and playback position.

set_progress(progress: number)

Sets the current playback progress ([0, 1]).

set_camera(type: string, options: object)

Updates the active camera type and parameters.

set_callback_pause_play(callback: Function)

Sets a callback for when play/pause state changes.

set_callback_render(callback: Function)

Sets a callback triggered after each frame render.

set_callback_frame_decoded(callback: Function)

Sets a callback triggered when a new frame is decoded.

set_callback_dropped_frames(callback: Function)

Sets a callback triggered when frames are dropped.

set_callback_camera_moved(callback: Function)

Sets a callback for camera movement events.

set_callback_spatial_finished(callback: Function)

Sets a callback when a spatial segment finishes playback.

<vv-player>

A lightweight viewer for rendering single .vv frames.

Attributes

| Attribute | Type | Description | | ------------ | -------- | ---------------------------------------------------------------------------- | | src | string | URL to a .vv frame (binary). Automatically fetched and displayed when set. | | bbox-width | string | Bounding box line width. | | top-color | string | RGBA string (e.g. "255 255 255 255") for top gradient color. | | bot-color | string | RGBA string for bottom gradient color. | | camera | string | Camera type: "default", "preview", "boomerang", or "portal". | | volume | Number | The volume of any audio, must be in [0, 1]

Methods

setVV(frame: Frame)

Sets the current .vv frame for rendering.

setCamera(type: string, options: object)

Replaces the current camera with a new one.

getCamera() → object

Returns current camera parameters.

Virtual Filesystem

This library uses Emscripten’s virtual filesystem (FS) to handle file I/O inside the browser. The following information is important when working with functions which take file paths as input.

All file paths (e.g. /persistent/output.splv) refer to a sandboxed in-memory filesystem. To make data persistent across sessions, the virtual filesystem can be backed by IndexedDB using the IDBFS backend.

Before using any objects/functions which use file paths, you must set up and sync the filesystem like this:

FS.mkdir('/persistent');
FS.mount(IDBFS, {}, '/persistent');

FS.syncfs(true, (err) => {
    if(err) 
        console.error("Initial sync failed:", err);
    else
        // filesystem is ready
});

After you're done writing files (e.g. after encoder.finish()), sync again to persist:

FS.syncfs(false, (err) => {
    if(err)
        console.error("Final sync failed:", err);
    else 
        console.log("File saved to IndexedDB!");
});

You can then read the file from /persistent/... using FS.readFile() or download it as a Blob.

⚠️ Files are held in memory until FS.syncfs(false) is called. Always call syncfs after encoding to flush data to disk!

Classes

Frame

Represents a 3D volume - a structured grid of voxels. This is a single frame of an splv file.

Constructors

Create an empty frame:

const frame = new SPLV.Frame(width, height, depth);

• width (number): Frame width in voxels.
• height (number): Frame height in voxels.
• depth (number): Frame depth in voxels.

Methods

Create an identical copy of a frame:

const cloned = frame.clone(); // SPLV.Frame

Load a frame from a buffer:

const frame = SPLV.Frame.load(buf); // SPLV.Frame

• buf (Uint8Array): The buffer to load from.

Load a frame from a file:

const frame = SPLV.Frame.load(path); // SPLV.Frame

• path (string): The path to load from, must be present in SPLV.FS.

Create a frame from a color + depth image:

const result = SPLV.Frame.fromRGBD(
    imgWidth, imgHeight, 
    colorImg, depthImg, 
    intrinsics, extrinsics, 
    xMin, yMin, zMin, xMax, yMax, zMax, 
    width, height, depth
); // object

• imgWidth, imgHeight (Number): The dimensions of the images to generate from.
• colorImg (Uint8Array): The color image from which to generate the frame. Must contain imgWidth * imgHeight color values in the format [r, g, b].
• depthImg (Float32Array): The corresponding depth image from which to generate the frame. The depth. Must contain imgWidth * imgHeight depth values. The depth values are in camera space.
• intrinsics (Float32Array): The camera intrinsics matrix. Must be a 3x3 matrix.
• extrinsics (Float32Array): The camera extrinsics matrix, must be a 3x4 matrix (affine transform).
• xMin, yMin, zMin (Number): The minimum world-space position to be included in the frame.
• xMax, yMax, zMax (Number): The maximum world-space position to be included in the frame.
• width, height, depth: The dimensions of the returned frame.

Returns:

• The frame populated with voxels from the images (result.frame).
• The minimum world-space position of any voxel, regardless of whether or not it was clipped by minPos/maxPos. This is helpful if you're not yet sure what the world bounds should be. (result.xWorldBoundMin, result.yWorldBoundMin, result.zWorldBoundMin).
• The maximum world-space position of any voxel, regardless of whether or not it was clipped by minPos/maxPos. This is helpful if you're not yet sure what the world bounds should be. (result.xWorldBoundMax, result.yWorldBoundMax, result.zWorldBoundMax).

Save a frame to a buffer:

const buf = frame.save(); // Uint8Array

Save a frame to a file:

frame.save(path);

• path (string): The path to save to, will be present in SPLV.FS.

Get the dimensions of a frame:

const { width, height, depty } = frame.getDims(); // object

Get the number of nonempty "bricks" in the frame (BRICK_SIZE^3 regions of voxels):

const numBricks = frame.getNumBricks(); // number

Get the number of voxels in a frame:

const voxels = frame.getNumVoxels(); // number

Get the voxel at a given position:

const voxel       = frame.getVoxel(x, y, z); // null
const { r, g, b } = frame.getVoxel(x, y, z); // object

• x, y, z (number): Position within frame.

Set the voxel at a given position:

frame.setVoxel(x, y, z, { r: redChannel, g: greenChannel, b: blueChannel });
frame.setVoxel(x, y, z, null);

• x, y, z (number): Position within frame.
• voxel (object|null): RGB object ({r,g,b}), or null to remove.

Fill a region of a frame with a given voxel:

frame.fill(xMin, yMin, zMin, xMax, yMax, zMax, { r: redChannel, g: greenChannel, b: blueChannel })
frame.fill(xMin, yMin, zMin, xMax, yMax, zMax, null)

• xMin, yMin, zMin (int): The minimum voxel coordinates to be filled.
• xMax, yMax, zMax (int): The maximum voxel coordinates to be filled.
• voxel (object|null): RGB object ({r,g,b}), or null to remove.

Add a given frame into a frame:

frame.add(other, x, y, z);
frame.add(other, x, y, z, lrAxis, udAxis, fbAxis);
frame.add(other, x, y, z, lrAxis, udAxis, fbAxis, flipLR, flipUD, flibFB);

• src (SPLV.Frame): The frame to add.
• x, y, z (number): Offset for the added frame (default: 0).
• lrAxis, udAxis, fbAxis (string): Axis mapping for left-right, up-down, front-back. One of "x", "y", "z".
• flipLR, flipUD, flipFB (boolean): Whether to reflect the frame after mapping.

Subtract a given frame from a frame:

frame.subtract(other, x, y, z);
frame.subtract(other, x, y, z, lrAxis, udAxis, fbAxis);
frame.subtract(other, x, y, z, lrAxis, udAxis, fbAxis, flipLR, flipUD, flibFB);

• src (SPLV.Frame): The frame to subtract.
• x, y, z (number): Offset for the subtract frame (default: 0).
• lrAxis, udAxis, fbAxis (string): Axis mapping for left-right, up-down, front-back. One of "x", "y", "z".
• flipLR, flipUD, flipFB (boolean): Whether to reflect the frame after mapping.

Clip a frame to given bounds:

frame.clip(xMin, yMin, zMin, xMax, yMax, zMax);

• xMin, yMin, zMin (number): The minimum voxel coordinates that will remain in the frame after clipping.
• xMax, yMax, zMax (number): The maximum voxel coordinates that will remain in the frame after clipping.

Scale (resample) a frame to given dimensions:

const scaled = frame.resampled(width, height, depth, alphaCutoff); // SPLV.Frame

• width, height, depth (number): New dimensions.
• alphaCutoff (number): Alpha threshold for resampling.

Downscale a frame by an integer factor:

const coarsened = frame.coarsened(scale, alphaCutoff); // SPLV.Frame

• scale (int): The factor by which to downscale. The frame returned will have dimensions of the source frame multiplied by 1/scale (default: 2).
• alphaCutoff (float): Alpha threshold for coarsening (default: 0.0).

Extract a subregion of a frame:

const cropped = frame.subregion(xMin, yMin, zMin, xMax, yMax, zMax); // SPLV.Frame

• xMin, yMin, zMin (number): The minimum voxel coordinates to be contained in the subregion.
• xMax, yMax, zMax (number): The maximum voxel coordinates to be contained in the subregion.

Create a clone of a frame without voxels that are completely hidden by other voxels:

const newFrame = frame.withoutOccluded(); // SPLV.Frame

Create a clone of a frame without isolated voxels with no neighboring voxels:

const newFrame = frame.withoutOrphaned(); // SPLV.Frame

Encoder

Compresses and encodes frames into splv files. Performs all spatial encoding-related tasks.

Constructors

Create an encoder:

const encoder = new SPLV.Encoder(width, height, depth, framerate, outPath);
const encoder = new SPLV.Encoder(
  width, height, depth, framerate,
  audioParams, gopSize, motionVectors,
  vqRangeCutoff, outPath
);

• width (number): Width of each frame in pixels.
• height (number): Height of each frame in pixels.
• depth (number): Depth of each frame in slices.
• framerate (number): Target framerate of the output video.
• audioParams (object | null): Optional audio settings.
  • sampleRate (number)
  • numChannels (number)
  • bitDepth (number)
• gopSize (number): Group-of-pictures size.
• motionVectors (string): Motion vector mode. Accepted values: 'none', 'fast', 'fill'.
• vqRangeCutoff (number): VQ range cutoff threshold.
• outPath (string): Output file path.

Methods

Encode a frame:

encoder.encode(frame);

• frame (SPLV.Frame): The frame to encode. Encode audio data:

encoder.encodeAudio(buf);

• buf (Uint8Array): Raw PCM audio data (interleaved if multi-channel).

Finalize encoding, flush to file:

encoder.finish();

Decoder

Decompresses and decodes splv files into their constituent framess. Performs all spatial decoding-related tasks.

Constructors

Create a decoder from a buffer containing splv data:

const decoder = new SPLV.Decoder(buffer);

• buffer (Uint8Array): Binary contents of an .splv file.

Create a decoder from an splv file:

const decoder = new SPLV.Decoder(path);

• path (string): Path to an .splv on disk, must be in SPLV.FS.

Methods

Get the dimensions of a decoder's splv:

const { width, height, depth } = decoder.getDims(); // object

Get the framerate of the decoder's splv:

const framerate = decoder.getFramerate(); // number

Get the number of frames in the decoder's splv:

const count = decoder.getFrameCount(); // number

Decode a frame:

const frame = decoder.decode(idx); // SPLV.Frame

• idx (number): Frame index.