@mp4e/player

React player for MP4E interactive videos. The WASM engine is bundled inline for universal compatibility.

Installation

npm install @mp4e/player

Usage

import { MP4EPlayer } from '@mp4e/player';
import '@mp4e/player/styles.css';

function App() {
  return (
    <MP4EPlayer
      src="/video.mp4"
      onMetadataLoaded={(metadata) => {
        console.log('Loaded:', metadata.schemaVersion);
      }}
    />
  );
}

The player automatically extracts metadata embedded in the MP4 file - you typically only need the src prop.

Features

• Automatic metadata extraction - Reads MP4E metadata from the video file
• WASM engine bundled - Works offline, no external fetches
• Universal bundler support - Vite, webpack, Next.js
• TypeScript types included
• WebGL-accelerated rendering - GPU-powered object highlighting
• Worker mode - Offload engine to Web Worker for better performance

Props

Core Props

| Prop | Type | Default | Description | |------|------|---------|-------------| | src | string | required | Video URL | | metadata | MP4EMetadata | - | Override embedded metadata | | controls | boolean | true | Show playback controls | | autoplay | boolean | false | Auto-start playback | | muted | boolean | false | Start muted | | loop | boolean | false | Loop playback | | interactive | boolean | true | Enable object interactions | | showBboxes | boolean | false | Show object bounding boxes | | showCanvasLabels | boolean | false | Show labels on hover | | showObjectDisplays | boolean | true | Show hover/click displays | | useWorker | boolean | false | Run engine in Web Worker | | debug | boolean | false | Show debug UI |

Callback Props

| Prop | Type | Description | |------|------|-------------| | onEngineReady | () => void | Engine initialized | | onEngineError | () => void | Engine failed to load | | onMetadataLoaded | (metadata: MP4EMetadata) => void | Metadata loaded | | onTimeUpdate | (time: number, frame: number) => void | Time changed (throttled ~60fps) | | onFrameChange | (frame: number) => void | Frame changed (every frame) | | onVideoDuration | (duration: number) => void | Duration available | | onObjectClick | (objectId: string, data: ObjectData) => void | Object clicked | | onObjectHover | (objectId: string \| null, data: ObjectData \| null) => void | Object hovered | | onPlay | () => void | Video started | | onPause | () => void | Video paused | | onEnded | () => void | Video ended | | onSceneChange | (scene: Scene) => void | Scene changed |

Advanced Props

| Prop | Type | Description | |------|------|-------------| | initialTime | number | Start time in seconds | | objectGroups | ObjectGroup[] | Object grouping configuration | | defaultDisplaySettings | ObjectDisplaySettings | Default display behavior | | runtimeFlags | MP4ERuntimeFlags | Feature flags | | disableInteraction | boolean | Disable all hover/click interactions |

Ref API

Access player methods via ref:

import { useRef } from 'react';
import { MP4EPlayer, MP4EPlayerRef } from '@mp4e/player';

function App() {
  const playerRef = useRef<MP4EPlayerRef>(null);

  const handleSeek = () => {
    playerRef.current?.seek(10); // Seek to 10 seconds
  };

  return (
    <>
      <MP4EPlayer ref={playerRef} src="/video.mp4" />
      <button onClick={handleSeek}>Jump to 10s</button>
    </>
  );
}

Playback Methods

| Method | Returns | Description | |--------|---------|-------------| | play() | Promise<void> | Start playback | | pause() | void | Pause playback | | seek(time) | void | Seek to time (seconds) | | seekToFrame(frame) | void | Seek to frame number | | seekSilently(time) | void | Seek without triggering rules |

State Methods

| Method | Returns | Description | |--------|---------|-------------| | getCurrentTime() | number | Current time in seconds | | getCurrentFrame() | number | Current frame number | | getDuration() | number | Video duration in seconds | | isPaused() | boolean | Is video paused | | isMuted() | boolean | Is video muted | | getVolume() | number | Volume (0-1) | | setVolume(v) | void | Set volume (0-1) | | mute() | void | Mute audio | | unmute() | void | Unmute audio | | getPlaybackRate() | number | Playback speed | | setPlaybackRate(r) | void | Set playback speed |

Fullscreen Methods

| Method | Returns | Description | |--------|---------|-------------| | isFullscreen() | boolean | Is player fullscreen | | enterFullscreen() | Promise<void> | Enter fullscreen | | exitFullscreen() | Promise<void> | Exit fullscreen | | toggleFullscreen() | Promise<void> | Toggle fullscreen |

Element Access

| Method | Returns | Description | |--------|---------|-------------| | getVideoElement() | HTMLVideoElement \| null | Video element | | getCanvasElement() | HTMLCanvasElement \| null | Canvas element | | getOverlayContainerRef() | HTMLDivElement \| null | Overlay container |

Coordinate Conversion

| Method | Returns | Description | |--------|---------|-------------| | getVideoDisplayTransform() | VideoDisplayTransform \| null | Get scale/offset for coordinate conversion | | getOverlayPositions() | Map<string, RenderedOverlayPosition> | Get current overlay positions | | getEffectiveMetadata() | MP4EMetadata \| null | Get resolved metadata |

VideoDisplayTransform

interface VideoDisplayTransform {
  scale: number       // Scale factor from video-native to display
  offsetX: number     // Horizontal offset (letterboxing)
  offsetY: number     // Vertical offset (letterboxing)
  displayWidth: number
  displayHeight: number
  videoWidth: number
  videoHeight: number
}

Use this to convert between video-native coordinates and screen coordinates:

// Convert video-native position to screen position
const transform = playerRef.current?.getVideoDisplayTransform();
if (transform) {
  const screenX = videoNativeX * transform.scale + transform.offsetX;
  const screenY = videoNativeY * transform.scale + transform.offsetY;
}

Imperative Overlay Updates

For smooth 60fps position updates (e.g., during drag operations):

// Update overlay position imperatively (no React re-render)
playerRef.current?.updateOverlayStyle('overlay-id', {
  x: 100,  // video-native coordinates
  y: 50,
  width: 200,
  height: 100,
});

This directly updates the DOM element for smooth visual feedback, bypassing React's rendering cycle. Useful for:

• Drag operations
• Real-time position updates
• Animation

Types

MP4EMetadata

interface MP4EMetadata {
  schemaVersion: string;
  videoDetails: VideoDetails;
  objects: {
    registry: ObjectRegistry;
    timeline: FrameData[];
  };
  layers: Layer[];
  variables?: Variable[];
  rules?: Rule[];
  scenes?: Scene[];
}

ObjectData

interface ObjectData {
  id: string;
  label: string;
  userLabel?: string;
  confidence: number;
  trackingType: ObjectTrackingType;
  groupIds?: string[];
  hidden?: boolean;
  data?: Record<string, any>;
}

Development

# Run demo app
npm run dev

# Build library
npm run build:lib

# Build types
npm run build:types

# Publish to npm
npm publish

Architecture

Player (React) → Engine (WASM) → Pure State Machine

• Player is the "body" - handles UI, video, rendering
• Engine is the "brain" - handles events, state, rules
• Zero coupling - player listens to engine events

Performance

The player includes several performance optimizations:

1. Phase 1: Visibility-based re-rendering - only re-render when object visibility changes
2. Phase 2: DOM pooling - reuse overlay elements instead of creating/destroying
3. Phase 3: WebGL rendering - GPU-accelerated bounding box rendering
4. Worker mode: Offload engine computation to Web Worker

Enable worker mode for complex videos:

<MP4EPlayer src="/video.mp4" useWorker />

Changelog

v0.2.79

• Added disableInteraction prop to disable all hover/click interactions
• Added onFrameChange callback (fires on every frame change)
• Added onVideoDuration callback (fires when duration is available)
• Added getVideoDisplayTransform() ref method for coordinate conversion
• Added getOverlayPositions() ref method to get current overlay positions
• Added updateOverlayStyle() ref method for imperative 60fps position updates
• Added getEffectiveMetadata() ref method
• Added getOverlayContainerRef() ref method

v0.2.78

• WebGL-accelerated bounding box rendering (Phase 3 optimization)
• Improved performance for videos with many objects

v0.2.77

• DOM pooling for group-bound overlays (Phase 2 optimization)
• Visibility-based re-rendering (Phase 1 optimization)