@etaio/video-analysis

Shared video analysis services for ShotRush - frame extraction, AI analysis, and summary generation.

Features

• Frame Extraction: Extract frames from videos using FFmpeg with quality optimization
• AI Analysis: Analyze frames using OpenAI Vision API for scenes, objects, and actions
• Summary Generation: Generate video summaries and identify key moments
• Scene Detection: Automatic scene change detection
• Relevance Scoring: Score frames for auto-editing purposes

Installation

pnpm install
pnpm build

Usage

Extract Frames

import { extractFrames, getVideoMetadata } from '@etaio/video-analysis';

// Get video info
const metadata = await getVideoMetadata('/path/to/video.mp4');
console.log(`Duration: ${metadata.duration}s, FPS: ${metadata.fps}`);

// Extract frames
const frames = await extractFrames('/path/to/video.mp4', {
  frameInterval: 1,        // 1 frame per second
  maxFrames: 60,           // Max 60 frames
  frameQuality: 'medium'   // 1280x720 JPEG 80% quality
});

console.log(`Extracted ${frames.length} frames`);

Analyze Frames

import { analyzeFramesBatch } from '@etaio/video-analysis';

const results = await analyzeFramesBatch(
  frames.map(f => ({
    buffer: f.buffer,
    frameNumber: f.frameNumber,
    timestamp: f.timestamp,
    timestampMs: f.timestampMs
  })),
  {
    apiKey: process.env.OPENAI_API_KEY!,
    analyzeObjects: true,
    analyzeActions: true,
    analyzeScenes: true,
    contextDescription: 'Product demo video'
  },
  (current, total) => {
    console.log(`Analyzing frame ${current}/${total}`);
  }
);

console.log('Analysis complete:', results);

Generate Summary

import { generateVideoSummary } from '@etaio/video-analysis';

const { summary, keyMoments, tokenUsage } = await generateVideoSummary(
  frameAnalyses,
  {
    apiKey: process.env.OPENAI_API_KEY!,
    videoDuration: metadata.duration,
    contextDescription: 'Product demo video',
    maxKeyMoments: 10
  }
);

console.log('Summary:', summary.brief);
console.log('Key Moments:', keyMoments.length);
console.log('Tokens Used:', tokenUsage.totalTokens);

Frame Quality Options

• micro: 320x180 JPEG 70% (~5-10 KB per frame)
• low: 640x360 JPEG 75% (~15-25 KB per frame)
• medium: 1280x720 JPEG 80% (~40-60 KB per frame)
• high: 1920x1080 JPEG 85% (~80-120 KB per frame)

Cost Estimation

Using gpt-4o model with medium quality frames:

• Input: ~1,000 tokens per frame (image + prompt)
• Output: ~200 tokens per frame (analysis)
• Summary: ~1,500 input + 500 output tokens

Example (60-second video at 1 fps):

• 60 frames × 1,200 tokens = 72,000 tokens
• Summary: 2,000 tokens
• Total: ~74,000 tokens ≈ $0.38 USD

Architecture

Used in:

• Cloud Run Worker (services/video-analysis-worker): Heavy processing
• Firebase Functions (services/serverless/functions): API gateway
• Frontend Services (apps/web/src/services): Client SDK (future)

Development

# Watch mode
pnpm dev

# Build
pnpm build

# Clean
pnpm clean

Dependencies

• fluent-ffmpeg: Frame extraction
• sharp: Image optimization
• axios: OpenAI API calls
• firebase-admin: Firestore integration
• @etaio/core-types: Shared TypeScript types