AgentFoundry Replay Renderer

Video rendering service for AgentFoundry mini-game replays using Puppeteer and FFmpeg.

Architecture

This renderer uses a Puppeteer-based approach where:

1. The actual game is loaded in a headless browser
2. A replay manifest is injected into the game
3. The game enters "replay mode" and plays back the events
4. Screenshots are captured at each frame
5. FFmpeg combines screenshots into video

This approach ensures the video looks exactly like the gameplay with zero code duplication.

Static Bundle Hosting

The replay-server can host game bundles internally, eliminating the need for a separate game server:

┌──────────────────────────────────────────────────────────────┐
│                     replay-server                            │
│  ┌─────────────┐   ┌──────────────┐   ┌─────────────────┐    │
│  │ Bundle Host │   │   Puppeteer  │   │   HTTP API      │    │
│  │ /bundles/*  │◄──│   Renderer   │◄──│   POST /render  │    │
│  └─────────────┘   └──────────────┘   └─────────────────┘    │
│         │                                      ▲             │
│         ▼                                      │             │
│  bundles/game-life-restart/          manifest.json           │
└──────────────────────────────────────────────────────────────┘

With bundleId in the manifest, the server:

1. Serves the game from /bundles/<bundleId>/
2. Puppeteer loads the game from the internal static server
3. No external game server required

Prerequisites

• Node.js 18+
• pnpm 9.1.0+ (this is a pnpm monorepo)
• FFmpeg installed and in PATH
• Option A: Game running at http://localhost:5173 (development mode)
• Option B: Game bundle in bundles/ directory (production mode)

Installing pnpm

npm install -g [email protected]
# or
corepack enable
corepack prepare [email protected] --activate

Installing FFmpeg

macOS:

brew install ffmpeg

Windows:

choco install ffmpeg
# or download from https://ffmpeg.org/download.html

Linux:

sudo apt install ffmpeg

Usage

Setup

# From project root - install all dependencies
pnpm install

# Or install only this package and its dependencies
cd packages/replay-server
pnpm install

Render Pipeline Script (Recommended)

The render-pipeline.sh script provides a complete end-to-end workflow that automates:

• Server health checking (with auto-start option)
• Bundle verification and preloading
• Job submission via HTTP API
• Status polling with progress display
• Automatic video download

cd packages/replay-server

# Basic usage (server must be running)
./scripts/render-pipeline.sh -m samples/life-replay-lr-mkcdfzc2-u8cqbs.json

# With custom output and settings
./scripts/render-pipeline.sh \
  -m samples/life-replay-lr-mkcdfzc2-u8cqbs.json \
  -o videos/my-replay.mp4 \
  --width 720 \
  --height 1280 \
  --fps 30

# Auto-start server if needed
./scripts/render-pipeline.sh -m samples/replay.json --start-server

# Build bundle locally if missing
./scripts/render-pipeline.sh -m samples/replay.json --build-bundle

# Verbose output for debugging
./scripts/render-pipeline.sh -m samples/replay.json --verbose

# Show help
./scripts/render-pipeline.sh --help

Prerequisites:

• curl - HTTP requests (pre-installed on most systems)
• jq - JSON parsing (brew install jq on macOS, apt install jq on Linux)

Features:

• ✅ Idempotent - safe to run multiple times
• ✅ Automatic bundle downloading from OSS if bundleUrl provided
• ✅ Real-time progress display with status updates
• ✅ Colored output for better readability
• ✅ Robust error handling with clear messages
• ✅ Uses /api/bundles API endpoints (separated from static file serving at /bundles)

Note: Bundle management APIs are under /api/bundles/* prefix, while static bundle files are served at /bundles/<bundleId>/. This separation prevents route conflicts between API endpoints and static file serving.

CLI Tool (Direct Rendering)

For direct rendering without the HTTP server:

# From project root (recommended)
pnpm --filter replay-server render -- --manifest ./packages/replay-server/samples/replay.json --output ./packages/replay-server/output/output.mp4

# From package directory
cd packages/replay-server
pnpm run render -- --manifest ./samples/replay.json --output ./output/output.mp4

# Alternative: Use tsx directly
cd packages/replay-server
pnpm exec tsx src/cli/render.ts --manifest ./samples/replay.json --output ./output/output.mp4

# Render with custom settings
pnpm --filter replay-server render -- \
  --manifest ./replay.json \
  --output ./output.mp4 \
  --game-url http://localhost:5173 \
  --width 1080 \
  --height 1920 \
  --fps 30 \
  --seconds-per-age 2

HTTP Server

# From project root (recommended)
pnpm dev:replay-server

# Or from package directory
cd packages/replay-server
pnpm run dev

# Production build and start
# From project root
pnpm build:replay-server
cd packages/replay-server
pnpm start

# Or from package directory
cd packages/replay-server
pnpm run build
pnpm start

API Endpoints

POST /render - Start a render job

With bundleId (recommended - no external game server needed):

curl -X POST http://localhost:3001/render \
  -H "Content-Type: application/json" \
  -d '{
    "manifest": {
      "schema": "lifeRestart.replay.v1",
      "bundleId": "game-life-restart",
      "gameId": "test-game",
      "timeline": [...],
      "highlights": []
    },
    "config": {
      "width": 1080,
      "height": 1920
    }
  }'

With external gameUrl (legacy mode):

curl -X POST http://localhost:3001/render \
  -H "Content-Type: application/json" \
  -d '{
    "manifest": { ... },
    "config": {
      "gameUrl": "http://localhost:5173",
      "width": 1080,
      "height": 1920
    }
  }'

Response:

{
  "jobId": "5eaee18f-48a7-4193-b594-86756ac76ce2",
  "status": "pending",
  "message": "Render job started"
}

GET /api/bundles - List available game bundles

curl http://localhost:3001/api/bundles

Response:

{
  "bundles": [
    {
      "bundleId": "game-life-restart",
      "path": "/app/packages/replay-server/bundles/game-life-restart",
      "url": "http://localhost:3001/bundles/game-life-restart/",
      "ready": true
    }
  ],
  "count": 1
}

GET /jobs - List all jobs

curl http://localhost:3001/jobs

Response:

{
  "jobs": [
    {
      "jobId": "5eaee18f-48a7-4193-b594-86756ac76ce2",
      "status": "completed",
      "progress": { ... },
      "error": null,
      "createdAt": "2024-01-01T00:00:00.000Z",
      "completedAt": "2024-01-01T00:05:00.000Z"
    }
  ],
  "count": 1
}

GET /status/:jobId - Check job status

curl http://localhost:3001/status/5eaee18f-48a7-4193-b594-86756ac76ce2

GET /download/:jobId - Download completed video

curl -o video.mp4 http://localhost:3001/download/5eaee18f-48a7-4193-b594-86756ac76ce2

Replay Modes

The renderer supports two replay modes, detected automatically based on manifest fields:

| Mode | Detection | Use Case | Example | |------|-----------|----------|---------| | Continuous | summary.durationSeconds present | Real-time games with continuous animation | JumpArena | | Age-based | timeline[] array present | Turn-based or discrete-state games | LifeRestart |

Mode Detection Priority

1. If manifest.summary.durationSeconds exists and is > 0 → Continuous mode
2. If manifest.timeline[] exists and has items → Age-based mode
3. Fallback: Continuous mode with 10s default duration

Continuous Mode

For games that animate in real-time (like JumpArena):

• Renderer captures frames at regular intervals (e.g., 16 FPS = every 62.5ms)
• Game runs its own simulation based on elapsed time
• No external events dispatched during capture
• Duration determined by manifest.summary.durationSeconds

Manifest structure:

{
  "schema": "jumpArena.replay.v1",
  "bundleId": "game-jump-arena",
  "gameId": "ja-abc123",
  "gameRun": { "seed": 1234, "inputs": [...], "events": [...] },
  "summary": {
    "durationSeconds": 8.5,
    "totalScore": 10
  }
}

Age-based Mode

For games with discrete states (like LifeRestart):

• Renderer dispatches replay-next-age events to advance the game
• Each "age" is displayed for secondsPerAge seconds (default: 1.2s)
• Frames captured while displaying each age
• Duration = timeline.length × secondsPerAge

Manifest structure:

{
  "schema": "lifeRestart.replay.v1",
  "bundleId": "game-life-restart",
  "gameId": "lr-abc123",
  "timeline": [
    { "age": 0, "eventId": 1, "description": "..." },
    { "age": 1, "eventId": 2, "description": "..." }
  ],
  "summary": { "finalAge": 80 }
}

Game Integration

For the renderer to work, the game must support replay mode. The requirements differ slightly based on replay mode.

Common Requirements (Both Modes)

1. Listen for replay manifest

// In your game's main component or hook
useEffect(() => {
  const urlParams = new URLSearchParams(window.location.search);
  if (urlParams.get('mode') === 'replay') {
    // Wait for manifest injection
    const handleManifest = (e: CustomEvent) => {
      enterReplayMode(e.detail);
    };
    
    window.addEventListener('replay-manifest-loaded', handleManifest);
    
    // Also check if already injected
    if ((window as any).__REPLAY_MANIFEST__) {
      enterReplayMode((window as any).__REPLAY_MANIFEST__);
    }
    
    return () => {
      window.removeEventListener('replay-manifest-loaded', handleManifest);
    };
  }
}, []);

2. Signal when ready for capture

// Add data attribute when replay is ready
<div data-replay-ready={isReplayReady ? "true" : "false"}>
  {/* game content */}
</div>

Continuous Mode Requirements

For real-time games like JumpArena:

1. Start simulation from time zero

function enterReplayMode(manifest: ReplayManifest) {
  // Load the game state from manifest
  loadGameRun(manifest.gameRun);
  
  // CRITICAL: Reset simulation time to 0
  simulationTime = 0;
  
  // Mark as replay mode
  setIsReplayMode(true);
  setIsReplayReady(true);
}

2. Use elapsed time for simulation (not wall clock)

// In your animation loop
function gameLoop() {
  const now = performance.now();
  const dt = (now - lastFrameTime) / 1000;
  lastFrameTime = now;
  
  // Advance simulation time
  if (isReplayMode) {
    simulationTime += dt;
  }
  
  // Render game state at simulationTime
  const state = simulateAtTime(gameRun, simulationTime);
  render(state);
  
  requestAnimationFrame(gameLoop);
}

3. Key implementation notes

• Do NOT skip to final state: Even if gameRun.result exists, animate from the beginning
• simulateAtTime must be deterministic: Given the same inputs and time, return the same state
• Include summary.durationSeconds: Calculate total replay duration when exporting manifest

Age-based Mode Requirements

For discrete-state games like LifeRestart:

1. Handle replay-next-age event

window.addEventListener('replay-next-age', () => {
  // Advance to next age/state in timeline
  currentAgeIndex++;
  displayAge(timeline[currentAgeIndex]);
});

2. Start from first age

function enterReplayMode(manifest: ReplayManifest) {
  timeline = manifest.timeline;
  currentAgeIndex = -1; // Will be incremented on first replay-next-age
  setIsReplayMode(true);
  setIsReplayReady(true);
}

Configuration

Environment variables:

| Variable | Default | Description | |----------|---------|-------------| | PORT | 3001 | Server port | | GAME_URL | http://localhost:5173 | Fallback URL when no bundleId specified | | BUNDLES_DIR | ./bundles | Directory containing game bundles | | OUTPUT_DIR | ./output | Directory for rendered videos | | BFF_BASE_URL | http://localhost:11001 | BFF API URL for STS credentials | | BFF_SERVICE_TOKEN | - | Required for bundle downloads. Set to BFF's SUPABASE_JWT_SECRET | | MAX_CACHE_SIZE | 10737418240 | Max bundle cache size in bytes (10GB) |

Service-to-Service Authentication

For dynamic bundle downloads from OSS, the replay-server needs to authenticate with the BFF service. This uses a service-to-service authentication mechanism:

1. Get SUPABASE_JWT_SECRET from BFF: Copy the value of SUPABASE_JWT_SECRET from the BFF service's .env file
2. Set BFF_SERVICE_TOKEN: Configure this value in replay-server's .env:

   BFF_SERVICE_TOKEN=your-supabase-jwt-secret-from-bff

3. How it works: The replay-server calls POST /studio/service/sts on the BFF with this token as a Bearer token. BFF validates the token matches its SUPABASE_JWT_SECRET and returns STS credentials for OSS access.

Note: This endpoint does not require user authentication - it's designed for internal service communication only.

Game URL Resolution Order

When rendering a video, the game URL is resolved in this order:

1. config.gameUrl (explicit URL in render request)
2. manifest.bundleId + manifest.bundleUrl (downloads from OSS if not cached)
3. manifest.bundleId (serves from cached /bundles/<bundleId>/)
4. GAME_URL environment variable (fallback)

Dynamic Bundle Downloads

The replay-server supports downloading game bundles on-demand from Alibaba Cloud OSS.

How It Works

1. Client includes bundleId and bundleUrl in the render request manifest
2. Server checks if bundle is cached locally
3. If not cached, server gets STS credentials from BFF API
4. Server downloads and extracts the TAR.GZ bundle from OSS
5. Bundle is cached for future requests (LRU eviction when cache is full)

Bundle Management Endpoints

POST /api/bundles/preload - Preload a bundle (async download)

curl -X POST http://localhost:3001/api/bundles/preload \
  -H "Content-Type: application/json" \
  -d '{
    "bundleId": "game-vocab-master",
    "bundleUrl": "https://oss-bucket.oss-cn-beijing.aliyuncs.com/bundles/game-vocab-master.tar.gz"
  }'

GET /api/bundles/stats - Get cache and download statistics

curl http://localhost:3001/api/bundles/stats

Response:

{
  "cache": {
    "used": 15728640,
    "max": 10737418240,
    "usedPercent": 0.15,
    "bundleCount": 2
  },
  "downloads": {
    "active": 1,
    "completed": 15,
    "failed": 0
  }
}

DELETE /api/bundles/:bundleId - Remove bundle from cache

curl -X DELETE http://localhost:3001/api/bundles/game-vocab-master

Render Request with bundleUrl

curl -X POST http://localhost:3001/render \
  -H "Content-Type: application/json" \
  -d '{
    "manifest": {
      "schema": "lifeRestart.replay.v1",
      "bundleId": "game-vocab-master",
      "bundleUrl": "https://oss-bucket.oss-cn-beijing.aliyuncs.com/bundles/game-vocab-master.tar.gz",
      "gameId": "vm-abc123",
      "timeline": [...],
      "highlights": []
    },
    "config": {
      "width": 1080,
      "height": 1920
    }
  }'

Output

Videos are rendered in:

• Format: MP4 (H.264)
• Resolution: 720x1080 (portrait, for mobile/Feed)
• FPS: 16 (configurable)
• Duration: ~1.2s per age (configurable)

Building Game Bundles

Before deploying, you need to build game bundles that will be hosted by the replay-server.

Option A: Using the build script

# Build a specific game
cd packages/replay-server
./scripts/build-bundle.sh game-life-restart

# This will:
# 1. Build the game from repo/game-life-restart
# 2. Copy dist/ to bundles/game-life-restart/

Option B: Manual build

# Build the game
cd repo/game-life-restart
pnpm install
pnpm build

# Copy to bundles
mkdir -p packages/replay-server/bundles/game-life-restart
cp -r dist/* packages/replay-server/bundles/game-life-restart/

Option C: Docker (automatic)

The Dockerfile includes a multi-stage build that automatically builds and embeds the game bundle. No manual steps required.

Deployment Options

Local Development

# From project root (recommended)
pnpm dev:replay-server

# Or from package directory
cd packages/replay-server
pnpm run dev

Docker with Embedded Bundle (Recommended)

The Dockerfile uses multi-stage builds to:

1. Build the game-life-restart bundle
2. Build the replay-server
3. Create a runtime image with both embedded

# Build from project root
docker build -f packages/replay-server/Dockerfile -t replay-server:latest .

# Or use docker-compose for local testing
cd packages/replay-server
docker-compose -f docker-compose.local.yml up --build

Serverless (Aliyun FC)

Build and deploy to Aliyun Function Compute:

1. Login to Aliyun Container Registry (ACR)

# Login using Aliyun CLI (recommended)
aliyun cr GetAuthorizationToken --region cn-beijing --InstanceId xxx
docker login --username=<your-username> --password=<token> registry.cn-beijing.aliyuncs.com

# Or use temporary password from Aliyun console
docker login registry.cn-beijing.aliyuncs.com

2. Build Docker Image

# Build from project root (must be from root, as it needs access to multiple packages in monorepo)
cd /path/to/agent-foundry
docker build -f packages/replay-server/Dockerfile -t replay-server:latest .

# Note:
# - Dockerfile uses multi-stage builds
# - Stage 1: Builds game-life-restart bundle
# - Stage 2: Builds replay-server
# - Stage 3: Creates runtime with embedded bundle
# - Includes Chinese fonts (Noto CJK, WQY Zenhei) for proper text rendering

3. Local Docker Testing

Before pushing to ACR, test the Docker image locally:

3.1 Using Docker Compose (Recommended)

cd packages/replay-server
docker-compose -f docker-compose.local.yml up --build

# In another terminal:
curl http://localhost:3001/health
curl http://localhost:3001/bundles

3.2 Using Docker Run

# Run container with embedded bundle
docker run -d \
  --name replay-server-test \
  -p 3001:3001 \
  --shm-size=2g \
  -e PORT=3001 \
  -v $(pwd)/packages/replay-server/output:/app/packages/replay-server/output \
  replay-server:latest

# View container logs
docker logs -f replay-server-test

3.3 Test Health Check

# Check if the service started normally
curl http://localhost:3001/health

# Expected response:
# {
#   "status": "ok",
#   "gameUrl": "http://localhost:5173",
#   "bundlesDir": "/app/packages/replay-server/bundles",
#   "bundleCount": 1
# }

# List available bundles
curl http://localhost:3001/api/bundles

3.4 Test Render API with bundleId

# Submit a render task using embedded bundle (no external game server needed!)
curl -X POST http://localhost:3001/render \
  -H "Content-Type: application/json" \
  -d '{
    "manifest": {
      "schema": "lifeRestart.replay.v1",
      "bundleId": "game-life-restart",
      "gameId": "test-game",
      "timeline": [],
      "highlights": []
    },
    "config": {
      "width": 1080,
      "height": 1920,
      "fps": 30
    }
  }'

# Or use local manifest file
curl -X POST http://localhost:3001/render \
  -H "Content-Type: application/json" \
  -d "{\"manifest\": $(cat packages/replay-server/samples/life-replay-lr-mkcdfzc2-u8cqbs.json)}"

# View task list
curl http://localhost:3001/jobs

# Check specific task status (replace <job-id> with actual task ID)
curl http://localhost:3001/status/<job-id>

# Download completed video (replace <job-id> with actual task ID)
curl -o test-video.mp4 http://localhost:3001/download/<job-id>

3.5 View Output Files

# View generated video files locally
ls -lh packages/replay-server/output/

# Or enter container to view
docker exec -it replay-server-test ls -lh /app/packages/replay-server/output/

3.6 Stop and Cleanup

# Stop container
docker stop replay-server-test

# Remove container
docker rm replay-server-test

# Or stop and remove in one command
docker rm -f replay-server-test

# Remove image (optional)
docker rmi replay-server:latest

3.7 Debugging Tips

# Enter container for debugging
docker exec -it replay-server-test /bin/bash

# Check processes inside container
docker exec replay-server-test ps aux

# Check environment variables
docker exec replay-server-test env

# View container resource usage
docker stats replay-server-test

4. Deploy to ACR

Use the deploy script for easy deployment:

# Set your ACR namespace
export ACR_NAMESPACE="your-namespace"

# Deploy (builds and pushes to ACR)
cd packages/replay-server
./scripts/deploy-aliyun.sh

# Or with a custom version tag
./scripts/deploy-aliyun.sh v1.0.0

Or manually:

# Replace <your-namespace> with your namespace
export ACR_REGISTRY="registry.cn-beijing.aliyuncs.com"
export ACR_NAMESPACE="<your-namespace>"
export IMAGE_NAME="replay-server"
export IMAGE_TAG="latest"

# Tag image
docker tag replay-server:latest ${ACR_REGISTRY}/${ACR_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}

# Push image
docker push ${ACR_REGISTRY}/${ACR_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}

5. Configure in Aliyun Function Compute

** Rebuild to be compatible with Aliyun FC **

# make sure you are at root dir
# build disabling attestation/provennace
docker buildx build \
  --platform linux/amd64 \
  --provenance=false \
  --sbom=false \
  -t ${ACR_REGISTRY}/${ACR_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG} \
  -f packages/replay-server/Dockerfile \
  --push \
  . 

# imagetools inspectation
docker buildx imagetools inspect ${ACR_REGISTRY}/${ACR_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}

Environment Variables:

• PORT: Server port (default: 9000, FC will set this automatically)
• BUNDLES_DIR: /app/packages/replay-server/bundles (already set in Dockerfile)
• OUTPUT_DIR: /tmp/output (use /tmp for FC writable directory)

Function Configuration Requirements:

• Memory: Minimum 2048 MB (2GB), recommended 4GB
  • Chromium requires significant memory to launch and render
  • Less than 2GB may cause browser crashes or timeouts
• Timeout: Minimum 300 seconds (5 minutes), recommended 600 seconds (10 minutes)
  • Cold start + browser launch can take 30-60 seconds
  • Video rendering time depends on replay length
• Instance Concurrency: 1 (required)
  • Prevents resource contention between concurrent renders
  • Each instance needs dedicated CPU/memory for Chromium
• CPU: Auto-scaled with memory (FC default is fine)

Important Notes for FC Deployment:

• The image includes all Chromium dependencies pre-installed
• Browser launch parameters are optimized for containerized environments
• Cold starts may take longer on first invocation (expect 30-60s for browser initialization)

6. Verify Deployment

# Test if function started normally
curl https://<your-function-url>/health

# Check available bundles
curl https://<your-function-url>/api/bundles

# **NEW: Debug Chromium installation**
curl https://<your-function-url>/debug/chrome
# This endpoint verifies Chromium is properly installed and all dependencies are available
# Expected response:
# {
#   "executablePath": "/usr/bin/chromium",
#   "tests": {
#     "version": { "ok": true, "output": "Chromium 120.0.6099.224" },
#     "dependencies": { "ok": true, "summary": "All libraries found" },
#     "chineseFonts": { "ok": true, "count": 15, "sample": ["Noto Sans CJK...", "..."] }
#   }
# }

# Submit a render job
curl -X POST https://<your-function-url>/render \
  -H "Content-Type: application/json" \
  -d '{
    "manifest": {
      "schema": "lifeRestart.replay.v1",
      "bundleId": "game-life-restart",
      ...
    }
  }'