@memoreco/recorder

React-based screen/camera/area recorder for Memoreco SDKs. Preview-only (no upload) in this version.

Recent Updates

• 2025-11-19: Release 0.2.7 speeds up Voice Mode activation by lazy-loading the streaming service after getUserMedia, keeps the mic warm between chats, and exposes the new Voice Mode recorder hooks consumed by the SDK wrapper.
• 2025-11-12: Release 0.2.6 guards streaming status polling against race conditions so cleanup no longer throws “Cannot set properties of null” when the first session ends.
• 2025-11-12: Release 0.2.5 permanently disables shared AudioContext reuse after Firefox sample-rate mismatches, guaranteeing recording restarts can rebuild a fresh graph without user intervention.
• 2025-11-11: Release 0.2.4 adds a Firefox-safe AudioContext fallback so streaming transcription continues after sample-rate mismatches reported in production.
• 2025-11-11: Release 0.2.2 aligns with the regenerated API SDK and exposes the webhook test helpers required by the Web SDK while keeping recorder internals unchanged.
• 2025-11-11: Release 0.2.1 improves Speechmatics streaming diagnostics and guards against provider mismatches surfaced during voice-task sessions.

Install

Monorepo workspace dependency:

pnpm add @memoreco/recorder -w

Quick Start (React / Next.js)

"use client";
import { MemorecoRecorder } from "@memoreco/recorder";

export default function Page() {
  return <MemorecoRecorder mode="auto" />;
}

Hook Usage

"use client";
import { useMemorecoRecorder } from "@memoreco/recorder";

export default function CustomRecorder() {
  const {
    state,
    isRecording,
    selectArea,
    startRecording,
    stopRecording,
    recordings,
    enablePreview,
    disablePreview,
    previewStream,
    previewAudioData,
    pause,
    resume,
    isPaused,
  } = useMemorecoRecorder();

  const recordArea = async () => {
    await selectArea("click");
    await startRecording("area", { maxDuration: 10, countdown: 3 });
  };

  return (
    <div>
      <button onClick={() => enablePreview("video")}>Enable Camera Preview</button>
      <button onClick={() => enablePreview("audio")} className="ml-2">Enable Mic Preview</button>
      <button onClick={disablePreview} className="ml-2">Disable Preview</button>
      <button onClick={recordArea} className="ml-2">Record Selected Area</button>
      {isRecording && (
        <>
          <button onClick={pause} className="ml-2">Pause</button>
          <button onClick={resume} className="ml-2">Resume</button>
          <button onClick={stopRecording} className="ml-2">Stop</button>
        </>
      )}
      {recordings.map((r) => (
        <div key={r.id}>{r.file.name}</div>
      ))}
    </div>
  );
}

Area Selection API

• selectArea(method?: 'click' | 'drag') — returns selection bounds and element
• Pass selection implicitly via startRecording('area') after selectArea()
• Or provide element or coordinates in RecordingOptions

await selectArea("drag"); // user drags to pick a rectangle
await startRecording("area", { maxDuration: 15 });

Features Implemented

• Modes
  • audio: microphone recording with analyser data
  • video: camera recording
  • screen: screen/window/tab recording
  • area: selected area recording (element image or cropped screen video)
  • snapshot: camera still photo
  • screenshot: screen still photo (optionally cropped)
• UI & Controls
  • Countdown before start
  • Max-duration auto-stop
  • Pause/resume (MediaRecorder-based modes)
  • Pre-recording preview:
    • Camera preview (video-only)
    • Mic meter (audio-only) via analyser data
  • Live in-recording waveform for audio
• Advanced Features
  • Gaze tracking: Eye-gaze heatmap generation using WebGazer.js
  • Face detection: MediaPipe-based face validation for photos
  • NSFW detection: TensorFlow.js-based content filtering
  • Streaming transcription: Real-time audio transcription during recording
• Error surfacing via onError and hook error

Gaze Tracking (Eye-Gaze Heatmaps)

The SDK includes built-in gaze tracking powered by WebGazer.js, allowing you to track where users look during video and screen recordings.

Features

• ✅ Client-side eye tracking using WebGazer.js
• ✅ Face detection validation - only tracks when exactly 1 face is detected
• ✅ Auto-pause on camera off - pauses tracking if camera is turned off
• ✅ Live visualization options - face overlay, prediction dots, and live heatmap
• ✅ Post-processing utilities - generate static heatmaps, timeline heatmaps, gaze paths, and attention grids
• ✅ Per-frame gaze data - stores x/y coordinates, confidence, timestamps, and viewport dimensions

Basic Usage

1. Enable in SDK Settings Panel

The easiest way is to use the SDK's built-in Settings Panel:

// Render settings panel with gaze tracking options
const settingsPanel = memoreco.ui.renderSettings('settings-container', {
  features: {
    gazeTrackingEnabled: {
      enabled: true,
      label: '👁️ Enable Gaze Tracking',
      description: 'Track where eyes look during recording',
      default: false
    },
    gazeShowFaceOverlay: {
      enabled: true,
      label: 'Show Face Tracking Overlay',
      description: 'Display face detection overlay',
      default: false
    },
    gazeShowPredictionDots: {
      enabled: true,
      label: 'Show Gaze Prediction Dots',
      description: 'Display real-time gaze points',
      default: false
    },
    gazeShowLiveHeatmap: {
      enabled: true,
      label: 'Show Live Heatmap',
      description: 'Display heatmap overlay during recording',
      default: false
    },
    gazeHeatmapRadius: {
      enabled: true,
      label: 'Heatmap Point Radius',
      description: 'Size of heatmap points in pixels',
      default: 30,
      min: 10,
      max: 100
    }
  },
  onSettingsChange: (settings) => {
    // Settings are automatically applied
    console.log('Gaze tracking settings:', settings);
  }
});

2. Start Recording with Gaze Tracking

import { startRecording } from '@memoreco/recorder';

// Start video recording with gaze tracking
const result = await startRecording('video', {
  gazeTracking: {
    enabled: true,
    mode: 'video', // or 'screen'
    calibrationEnabled: false, // false = automatic calibration (default), true = manual calibration flow
    showFaceOverlay: false, // Hidden by default
    showPredictionDots: false,
    showHeatmapDuringRecording: false,
    heatmapPointRadius: 30,
    useKalmanFilter: true
  }
});

Calibration Options:

• calibrationEnabled: false (default) - Uses automatic calibration during recording (invisible to user)
• calibrationEnabled: true - Shows manual calibration flow before recording (30-second guided process for better accuracy)

3. Access Gaze Data After Recording

const result = await stopRecording();

if (result.gazeTracking) {
  console.log('Gaze samples:', result.gazeTracking.samples);
  console.log('Sample count:', result.gazeTracking.sampleCount);
  console.log('Mode:', result.gazeTracking.mode);
  
  // Each sample contains:
  // - x, y: coordinates on screen
  // - probability: confidence (0-1)
  // - relativeTimeMs: timestamp from recording start
  // - mode: 'video' or 'screen'
  // - viewport: {width, height}
}

Post-Processing Heatmaps

The SDK provides utilities to generate various visualizations from gaze data:

Static Heatmap (Summary)

import { generateStaticHeatmap, saveCanvasAsPNG } from '@memoreco/recorder';

// Generate heatmap showing all gaze points
const canvas = generateStaticHeatmap(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  pointRadius: 30,
  opacity: 0.6,
  color: { r: 255, g: 0, b: 0 }, // Red
  blurRadius: 25
});

// Download as PNG
await saveCanvasAsPNG(canvas, 'gaze-heatmap.png');

Timeline Heatmaps (Video Overlay)

import { generateTimelineHeatmaps } from '@memoreco/recorder';

// Generate heatmap for each video frame
const frames = generateTimelineHeatmaps(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  duration: result.duration, // milliseconds
  fps: 30,
  pointRadius: 30,
  opacity: 0.6
});

// frames[0] = heatmap for frame 0
// frames[1] = heatmap for frame 1
// ... etc

Gaze Path Visualization

import { generateGazePath } from '@memoreco/recorder';

// Show eye movement path
const canvas = generateGazePath(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  opacity: 0.8,
  color: { r: 0, g: 150, b: 255 } // Blue
});

Attention Grid (Heatmap Grid)

import { generateAttentionGrid } from '@memoreco/recorder';

// Divide screen into grid and show attention per cell
const { canvas, data } = generateAttentionGrid(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  gridSize: 10, // 10x10 grid
  opacity: 0.7
});

// data[row][col] = number of gaze samples in that cell

Configuration Options

interface GazeTrackingOptions {
  /** Enable gaze tracking (default: false) */
  enabled?: boolean;
  
  /** Mode: 'video' for camera, 'screen' for screen recording */
  mode?: "video" | "screen";
  
  /** Enable manual calibration flow (default: false - uses automatic) */
  calibrationEnabled?: boolean;
  
  /** Show face tracking overlay (default: false - hidden) */
  showFaceOverlay?: boolean;
  
  /** Show prediction dots/gaze points (default: false) */
  showPredictionDots?: boolean;
  
  /** Show live heatmap overlay (default: false) */
  showHeatmapDuringRecording?: boolean;
  
  /** Heatmap point radius in pixels (default: 30) */
  heatmapPointRadius?: number;
  
  /** Use Kalman filter for smoothing (default: true) */
  useKalmanFilter?: boolean;
  
  /** Pause WebGazer after init (default: true) */
  pauseOnInitialize?: boolean;
  
  /** Parent element for overlays (default: document.body) */
  overlayParent?: HTMLElement;
}

Server-Side Post-Processing

Automatic Heatmap Video Generation:

When gaze tracking is enabled, the Memoreco API automatically processes gaze data after recording completion and generates a new video with the heatmap overlay. The original video remains untouched.

How It Works:

1. Recording Upload: SDK sends gaze samples to API with video completion
2. Background Processing: Trigger.dev job generates heatmap frames and composites them onto video using FFmpeg
3. New Asset Creation: Heatmap video is stored as a separate asset (linked to original via metadata.parentVideoAssetId)
4. Media Proxy Access: Heatmap video is served via the same media proxy as thumbnails and audio

Access Heatmap Video:

// Get recording assets
const response = await fetch(`/v1/recordings/${recordingId}/assets`, {
  headers: { 'X-API-Key': apiKey }
});

const assets = await response.json();

// Find gaze heatmap video
const heatmapVideo = assets.data.find(asset => 
  asset.type === 'video' && 
  asset.metadata?.renderType === 'gaze_heatmap'
);

if (heatmapVideo) {
  // Access via media proxy (same as thumbnails/audio)
  const heatmapUrl = `/v1/media/${heatmapVideo.id}`;
  console.log('Heatmap video:', heatmapUrl);
}

Cascade Deletion:

Heatmap videos are automatically deleted when the original recording is deleted (via ON DELETE CASCADE on recording_id foreign key).

Processing Status:

Check if heatmap processing is complete:

const recording = await fetch(`/v1/recordings/${recordingId}`, {
  headers: { 'X-API-Key': apiKey }
}).then(r => r.json());

if (recording.data.metadata?.gazeHeatmapAssetId) {
  console.log('Heatmap processed:', recording.data.metadata.gazeHeatmapAssetId);
} else if (recording.data.metadata?.gazeHeatmapError) {
  console.error('Heatmap processing failed:', recording.data.metadata.gazeHeatmapError);
}

Gaze Sample Structure

interface GazeSample {
  /** X coordinate on screen (pixels) */
  x: number;
  
  /** Y coordinate on screen (pixels) */
  y: number;
  
  /** Confidence/probability (0-1 scale) */
  probability: number;
  
  /** Time offset from recording start (ms) */
  relativeTimeMs: number;
  
  /** Recording mode when captured */
  mode: "video" | "screen";
  
  /** Viewport dimensions at capture time */
  viewport: {
    width: number;
    height: number;
  };
}

Important Notes

⚠️ Face Detection Requirement: Gaze tracking only records samples when exactly 1 face is detected in the camera feed. If no face or multiple faces are detected, tracking automatically pauses.

⚠️ Camera Status: If the camera is turned off during a video recording, gaze tracking also pauses automatically.

⚠️ Screen Mode: For screen recordings, gaze tracking works without camera validation (WebGazer can function without camera feed).

⚠️ Performance: Gaze tracking adds minimal overhead (~5-10ms per frame) and runs in parallel with recording.

⚠️ Privacy: All gaze tracking happens client-side. No eye images or facial data are sent to servers - only x/y coordinates are stored.

Complete Example

import { 
  startRecording, 
  stopRecording,
  generateStaticHeatmap,
  generateGazePath,
  saveCanvasAsPNG
} from '@memoreco/recorder';

// 1. Start recording with gaze tracking
await startRecording('video', {
  gazeTracking: {
    enabled: true,
    mode: 'video',
    showHeatmapDuringRecording: true // Show live heatmap
  }
});

// 2. Record...
// User records their video while gaze tracking runs

// 3. Stop and get results
const result = await stopRecording();

if (result.gazeTracking) {
  console.log(`Captured ${result.gazeTracking.sampleCount} gaze samples`);
  
  // 4. Generate static heatmap
  const heatmap = generateStaticHeatmap(result.gazeTracking.samples, {
    width: result.dimensions?.width || 1920,
    height: result.dimensions?.height || 1080
  });
  
  await saveCanvasAsPNG(heatmap, 'gaze-heatmap.png');
  
  // 5. Generate gaze path
  const gazePath = generateGazePath(result.gazeTracking.samples, {
    width: result.dimensions?.width || 1920,
    height: result.dimensions?.height || 1080
  });
  
  await saveCanvasAsPNG(gazePath, 'gaze-path.png');
}

FFmpeg Video Overlay

To overlay heatmaps onto the recorded video using FFmpeg:

# Generate timeline heatmap frames first (in your code)
# Then use FFmpeg to overlay them:

ffmpeg -i recording.mp4 \
  -framerate 30 -i heatmap-frames/frame_%04d.png \
  -filter_complex "[0:v][1:v]overlay=0:0" \
  -c:a copy output-with-heatmap.mp4

Or use the helper function:

import { generateFFmpegOverlayCommand } from '@memoreco/recorder';

const command = generateFFmpegOverlayCommand(
  'recording.mp4',
  'heatmap-frames',
  'output-with-heatmap.mp4'
);

console.log('Run this command:', command);

The recorder SDK supports optional eye-gaze tracking during video and screen recordings using WebGazer.js. This feature captures where users are looking frame-by-frame, enabling post-processing heatmap overlays.

Basic Usage

import { useMemorecoRecorder } from "@memoreco/recorder";

function MyRecorder() {
  const { startRecording, stopRecording } = useMemorecoRecorder();

  const recordWithGaze = async () => {
    await startRecording("video", {
      gazeTracking: {
        enabled: true,
        showFaceOverlay: false,        // Hide from user (default)
        showPredictionDots: false,      // Hide gaze dots (default)
        showHeatmapDuringRecording: false  // Don't show live heatmap
      }
    });
  };

  const stopAndGetGazeData = async () => {
    const result = await stopRecording();
    
    if (result.gazeTracking) {
      console.log('Gaze samples:', result.gazeTracking.samples);
      console.log('Sample count:', result.gazeTracking.sampleCount);
      console.log('Mode:', result.gazeTracking.mode); // 'video' or 'screen'
      
      // Optional: Heatmap data URL (PNG)
      if (result.gazeTracking.heatmapDataUrl) {
        console.log('Heatmap:', result.gazeTracking.heatmapDataUrl);
      }
    }
  };

  return (
    <>
      <button onClick={recordWithGaze}>Start Recording with Gaze</button>
      <button onClick={stopAndGetGazeData}>Stop & Get Data</button>
    </>
  );
}

Configuration Options

interface GazeTrackingOptions {
  enabled?: boolean;                    // Enable gaze tracking (default: false)
  mode?: "video" | "screen";            // Recording mode (auto-detected)
  showFaceOverlay?: boolean;            // Show face tracking box (default: false)
  showPredictionDots?: boolean;         // Show gaze prediction dots (default: false)
  showHeatmapDuringRecording?: boolean; // Show live heatmap overlay (default: false)
  heatmapPointRadius?: number;          // Heatmap point radius in pixels (default: 30)
  useKalmanFilter?: boolean;            // Smooth gaze predictions (default: true)
  pauseOnInitialize?: boolean;          // Pause after init, resume on start (default: false)
  overlayParent?: HTMLElement;          // Parent element for overlays (default: document.body)
}

Gaze Data Structure

interface GazeSample {
  x: number;              // Screen X coordinate (pixels)
  y: number;              // Screen Y coordinate (pixels)
  probability: number;    // Confidence score (0-1)
  relativeTimeMs: number; // Milliseconds from recording start
  mode: "video" | "screen";  // Capture mode
  viewport: {
    width: number;        // Viewport width at capture time
    height: number;       // Viewport height at capture time
  };
}

// Recording result includes:
result.gazeTracking = {
  samples: GazeSample[];       // All captured gaze points
  sampleCount: number;         // Total samples
  heatmapDataUrl?: string;     // Optional PNG data URL
  mode: "video" | "screen";    // Recording mode
};

Post-Processing Heatmaps

Use the gaze samples to render heatmaps post-recording:

Option 1: Using the built-in heatmap

// Enable live heatmap generation during recording
await startRecording("screen", {
  gazeTracking: {
    enabled: true,
    showHeatmapDuringRecording: true,  // Generate heatmap
    heatmapPointRadius: 30              // Customize point size
  }
});

const result = await stopRecording();
// Access heatmap as PNG data URL
const heatmapImg = result.gazeTracking?.heatmapDataUrl;

Option 2: Custom post-processing with Remotion

// Remotion example for video overlay
import { Composition } from "remotion";

export const GazeOverlay = ({ gazeS samples, fps }) => {
  const frame = useCurrentFrame();
  const timeMs = (frame / fps) * 1000;
  
  // Find gaze points for this frame
  const currentGaze = samples.filter(s => 
    Math.abs(s.relativeTimeMs - timeMs) < 50
  );
  
  return (
    <AbsoluteFill>
      {currentGaze.map((point, i) => (
        <div
          key={i}
          style={{
            position: "absolute",
            left: point.x,
            top: point.y,
            width: 60,
            height: 60,
            borderRadius: "50%",
            background: `rgba(255, 0, 0, ${point.probability * 0.3})`,
            transform: "translate(-50%, -50%)"
          }}
        />
      ))}
    </AbsoluteFill>
  );
};

Option 3: FFmpeg filter

# Export gaze samples as JSON
# Use FFmpeg drawbox filter to add gaze points per frame
ffmpeg -i video.webm -vf "drawbox=..." output.mp4

Screen Mode vs Video Mode

• Video mode: Tracks where user looks relative to their camera view
• Screen mode: Tracks where user looks on the screen (preferred for usability studies)
• Precedence: Screen recordings automatically use screen mode when both video and screen are possible

Performance Notes

• WebGazer.js loads ~2MB from CDN on first use
• Gaze tracking adds minimal CPU overhead (~5-10%)
• Initialization takes 1-2 seconds for model loading
• Samples are captured at ~30-60 Hz depending on device performance
• Each sample is ~100 bytes; a 5-minute recording generates ~18-36K samples (~1.8-3.6 MB)

Browser Compatibility

• Chrome/Edge: Full support
• Firefox: Full support
• Safari: Supported (may require camera permission even for screen recordings)
• Mobile: Limited support (WebGazer.js works best on desktop)

How to Test (Local)

1. Start the web app:
   • Visit /recorder-demo (Next.js demo page is included)
2. Try modes:
   • Audio: enable mic preview (optional), start recording, observe waveform, stop and play back
   • Video: enable camera preview (optional), start, pause, resume, stop
   • Screen: start and select a window/tab/screen; stop and play back
   • Area: click “Start Recording” in area mode; you’ll be prompted to select an element (overlay), then recording begins
   • Snapshot/Screenshot: select mode, click “Start Recording” to capture a still automatically
3. Permission flows:
   • If a preview fails (denied), re-enable preview to retry
   • The demo page will show a toast on onError

Where files are saved

• All captures are created in-memory and shown via URL.createObjectURL(...).
• In the demo, files are not uploaded or persisted; use the Download button to save locally.
• In your app, use the onRecordingComplete(result) callback from MemorecoRecorder or the hook’s onRecordingComplete to upload or store result.file.

Examples

• Next.js demo page: /apps/web/app/recorder-demo/page.tsx

Roadmap

• Upload integration via API SDK
• Better permission guidance panel and tailored tips
• Screen/tab preview options
• File processing/transcoding (mp4/mp3 conversion)

Modes Overview

• Audio: Records your microphone only. Live meter shows levels; output is an audio file (webm/opus). Countdown/max duration/pause/resume apply.

• Video: Records your webcam (with mic). Optional live camera preview before recording; output is a video file (webm). Countdown/max duration/pause/resume apply.

• Screen: Records your screen/window/tab. Checkbox lets you also record your microphone. Some browsers may also include tab/system audio when available. Output is a video file (webm). Countdown/max duration/pause/resume apply.

• Area: Records a specific on‑page area you pick.

  • If an element is selected, it captures that element as an image (png).
  • If a rectangle capture is used, it records that region as a video (webm) by cropping the screen stream.
  • Countdown/max duration apply when it’s a video capture; not for single-image captures.

• Snapshot: Takes a single still photo from the webcam (png). One tap (“Take Photo”) — no countdown/max duration/pause.

• Screenshot: Takes a single still image (png) of the screen/window/tab you choose in the browser picker. One tap (“Take Screenshot”) — no countdown/max duration/pause.

Notes:

• “Show live preview” is for video/screen modes to verify camera before recording.
• “Record microphone audio with screen” merges your mic with the screen capture; you’ll see a separate mic permission prompt.
• If mic is denied or muted, audio wave/meter may stay flat and recordings will be silent.