TTV Middleware

Provider-agnostic Text-to-Video middleware with async polling, retry logic, and comprehensive error handling. Currently supports OpenAI Sora (Sora 2, Sora 2 Pro) and Google Veo (Veo 2, Veo 3, Veo 3.1). Features image-to-video, video extension, configurable progress callbacks, and optional auto-download.

• Features
• Quick Start
• Prerequisites
• Configuration
• Providers & Models
• Image-to-Video
• Video Extension
• API Reference
• Advanced Features
• Testing
• Contributing
• License
• Links

Features

• Multi-Provider Architecture: Unified API for all TTV providers
  • OpenAI Sora: Sora 2 & Sora 2 Pro with text-to-video and image-to-video
  • Google Veo: Veo 2, Veo 3, Veo 3.1 (+ fast variants) with video extension support
• Async Polling: Both APIs are asynchronous - the middleware handles polling internally with configurable intervals and exponential backoff
• Image-to-Video: Animate a still image into a video (both providers)
• Video Extension: Continue/extend an existing video (Google Veo)
• Progress Callbacks: Optional onProgress callback for real-time generation status updates
• Auto-Download: Optionally download generated videos to Buffer (downloadToBuffer)
• Retry Logic: Exponential backoff with jitter for transient errors (429, 408, 5xx, timeouts)
• TypeScript First: Full type safety with comprehensive interfaces
• Logging Control: Configurable log levels via environment or API
• Debug Logging: Markdown file logging for debugging prompts and responses
• Error Handling: Typed error classes including PollingTimeoutError and ContentModeratedError
• Dry Mode: Validate requests without API calls (no costs during development)

Quick Start

Installation

Install from npm:

npm install @loonylabs/ttv-middleware

# For OpenAI Sora provider:
npm install openai

# For Google Veo provider:
npm install google-auth-library

Or install directly from GitHub:

npm install github:loonylabs-dev/ttv-middleware

Basic Usage

import { TTVService, OpenAISoraProvider, TTVProvider } from '@loonylabs/ttv-middleware';

// Create service and register provider
const service = new TTVService();
service.registerProvider(new OpenAISoraProvider({
  apiKey: process.env.OPENAI_API_KEY,
}));

// Generate a video
const result = await service.generate({
  prompt: 'A cat sitting on a windowsill watching rain fall outside',
  model: 'sora-2',
  duration: 8,         // 8 seconds
  aspectRatio: '16:9',
});

console.log('Video URL:', result.videos[0].url);
console.log('Duration:', result.metadata.duration, 'ms (total)');
console.log('Generation time:', result.metadata.generationTime, 'ms');

import { TTVService, GoogleVeoProvider, TTVProvider } from '@loonylabs/ttv-middleware';

const service = new TTVService();
service.registerProvider(new GoogleVeoProvider({
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  location: 'us-central1',
}));

const result = await service.generate({
  prompt: 'A cinematic aerial shot of a mountain range at sunrise',
  model: 'veo-3.0-generate-001',
  duration: 8,
  aspectRatio: '16:9',
  resolution: '1080p',
  generateAudio: true,  // Veo 3+ generates audio natively
});

console.log('Video URL:', result.videos[0].url);

const result = await service.generate({
  prompt: 'A futuristic city with flying cars',
  model: 'sora-2',
  duration: 12,
  onProgress: (progress) => {
    console.log(`Status: ${progress.status}`, progress.message || '');
    // Status: queued
    // Status: in_progress
    // Status: completed
  },
});

import * as fs from 'fs';

const result = await service.generate({
  prompt: 'Ocean waves crashing on a rocky shore',
  model: 'sora-2',
  downloadToBuffer: true,  // Download video into memory
});

// Save to disk
fs.writeFileSync('output.mp4', result.videos[0].buffer!);

// Use OpenAI Sora
const soraResult = await service.generate({
  prompt: 'A mountain landscape timelapse',
  model: 'sora-2',
}, TTVProvider.OPENAI_SORA);

// Use Google Veo
const veoResult = await service.generate({
  prompt: 'A mountain landscape timelapse',
  model: 'veo-3.0-generate-001',
}, TTVProvider.GOOGLE_VEO);

Prerequisites

• Node.js 18+
• TypeScript 5.3+

For OpenAI Sora provider:

npm install openai

For Google Veo provider:

npm install google-auth-library

Configuration

Create a .env file in your project root:

# Default provider
TTV_DEFAULT_PROVIDER=openai-sora

# Logging level (debug, info, warn, error, silent)
TTV_LOG_LEVEL=info

# OpenAI Sora
OPENAI_API_KEY=sk-...

# Google Veo (Vertex AI)
GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=./service-account.json
GOOGLE_CLOUD_REGION=us-central1

Providers & Models

OpenAI Sora

| Model | ID | Duration | Resolutions | Audio | Image-to-Video | |-------|-----|----------|-------------|-------|----------------| | Sora 2 | sora-2 | 4, 8, 12s | 720p, 1080p | Yes | Yes | | Sora 2 Pro | sora-2-pro | 10, 15, 25s | 720p, 1080p | Yes | Yes |

Pricing: ~$0.10-$0.50/sec depending on model and resolution.

Google Veo

| Model | ID | Duration | Resolutions | Audio | Video Extension | |-------|-----|----------|-------------|-------|-----------------| | Veo 2 | veo-2.0-generate-001 | 5-8s | 720p | No | Yes | | Veo 3 | veo-3.0-generate-001 | 4-8s | 720p, 1080p | Yes | Yes | | Veo 3 Fast | veo-3.0-fast-generate-001 | 4-8s | 720p, 1080p | Yes | Yes | | Veo 3.1 | veo-3.1-generate-001 | 4-8s | 720p, 1080p, 4K | Yes | Yes | | Veo 3.1 Fast | veo-3.1-fast-generate-001 | 4-8s | 720p, 1080p, 4K | Yes | Yes |

Pricing: ~$0.15-$0.75/sec depending on model.

Image-to-Video

Animate a still image into a video. Both providers support this:

import * as fs from 'fs';

// Load a reference image
const imageBase64 = fs.readFileSync('character.png').toString('base64');

// OpenAI Sora: image becomes first frame
const result = await service.generate({
  prompt: 'She turns around and smiles, then slowly walks out of frame',
  model: 'sora-2',
  referenceImage: {
    base64: imageBase64,
    mimeType: 'image/png',
  },
  duration: 8,
});

// Google Veo: first frame + optional last frame for interpolation
const veoResult = await service.generate({
  prompt: 'Camera slowly zooms out revealing the landscape',
  model: 'veo-3.0-generate-001',
  referenceImage: {
    base64: firstFrameBase64,
    mimeType: 'image/png',
  },
  lastFrameImage: {         // Veo-specific: interpolate between two keyframes
    base64: lastFrameBase64,
    mimeType: 'image/png',
  },
}, TTVProvider.GOOGLE_VEO);

Video Extension

Extend an existing video (Google Veo only):

const extended = await service.extend({
  prompt: 'The camera continues to pan revealing a hidden waterfall',
  videoBuffer: existingVideoBuffer,
  videoMimeType: 'video/mp4',
  duration: 7,  // Extend by 7 seconds
  downloadToBuffer: true,
}, TTVProvider.GOOGLE_VEO);

fs.writeFileSync('extended.mp4', extended.videos[0].buffer!);

API Reference

TTVService

class TTVService {
  registerProvider(provider: BaseTTVProvider): void;
  generate(request: TTVRequest, provider?: TTVProvider): Promise<TTVResponse>;
  extend(request: TTVExtendRequest, provider?: TTVProvider): Promise<TTVResponse>;
  getProvider(name: TTVProvider): BaseTTVProvider | undefined;
  getAvailableProviders(): TTVProvider[];
  listAllModels(): Array<{ provider: TTVProvider; models: ModelInfo[] }>;
  findProvidersWithCapability(capability: keyof TTVCapabilities): Array<{ provider: TTVProvider; models: ModelInfo[] }>;
}

TTVRequest

interface TTVRequest {
  prompt: string;
  model?: string;              // 'sora-2', 'veo-3.0-generate-001', etc.
  duration?: number;           // Desired duration in seconds
  aspectRatio?: string;        // '16:9', '9:16'
  resolution?: '720p' | '1080p' | '4k';
  n?: number;                  // Number of videos (default: 1)

  // Image-to-video
  referenceImage?: TTVReferenceImage;
  lastFrameImage?: TTVReferenceImage;  // Google Veo only

  // Audio & content
  generateAudio?: boolean;
  negativePrompt?: string;

  // Output control
  downloadToBuffer?: boolean;  // Download video to Buffer (default: false)
  onProgress?: TTVProgressCallback;

  // Retry & debug
  retry?: boolean | RetryOptions;
  dry?: boolean;

  providerOptions?: Record<string, unknown>;
}

TTVResponse

interface TTVResponse {
  videos: TTVVideo[];
  metadata: {
    provider: string;
    model: string;
    region?: string;
    duration: number;          // Total request time (ms)
    generationTime?: number;   // Polling time only (ms)
  };
  usage: {
    videosGenerated: number;
    totalDurationSeconds: number;
    modelId: string;
  };
  billing?: {
    cost: number;
    currency: string;
    source: 'provider' | 'estimated';
  };
}

TTVVideo

interface TTVVideo {
  url?: string;           // Video URL (Sora) or undefined (Veo returns buffer directly)
  buffer?: Buffer;        // Video data (if downloadToBuffer)
  contentType: string;    // 'video/mp4'
  duration?: number;      // Duration in seconds
}

Advanced Features

Video generation is asynchronous. The middleware polls for completion automatically. You can configure the polling behavior:

import { GoogleVeoProvider } from '@loonylabs/ttv-middleware';

const provider = new GoogleVeoProvider({
  projectId: 'my-project',
  polling: {
    intervalMs: 10000,       // Start polling every 10s (default)
    maxIntervalMs: 30000,    // Cap at 30s between polls
    backoffMultiplier: 1.5,  // Increase interval by 1.5x each time
    timeoutMs: 600000,       // Give up after 10 minutes (default)
  },
});

| Option | Default | Description | |--------|---------|-------------| | intervalMs | 10000 | Initial polling interval (ms) | | maxIntervalMs | 30000 | Maximum polling interval (ms) | | backoffMultiplier | 1.5 | Multiplier per poll attempt | | timeoutMs | 600000 | Maximum wait time (10 minutes) |

Automatic retry with exponential backoff and jitter for transient errors (429, 408, 5xx, network timeouts):

// Default: 3 retries, exponential backoff (1s -> 2s -> 4s), jitter enabled
const result = await service.generate({
  prompt: 'A sunset over mountains',
  // retry: true (default)
});

// Custom retry configuration
const result = await service.generate({
  prompt: 'A sunset over mountains',
  retry: {
    maxRetries: 5,
    delayMs: 1000,
    backoffMultiplier: 2.0,
    maxDelayMs: 30000,
    jitter: true,
  },
});

// Disable retry
const result = await service.generate({
  prompt: 'A sunset over mountains',
  retry: false,
});

Retryable errors: 429, 408, 500, 502, 503, 504, timeouts, ECONNRESET, ECONNREFUSED, socket hang up Not retried: 400, 401, 403, and other client errors

Control logging via environment variable or API:

import { setLogLevel } from '@loonylabs/ttv-middleware';

// Set log level programmatically
setLogLevel('warn');  // Only show warnings and errors

// Or via environment variable
// TTV_LOG_LEVEL=error

Available levels: debug, info, warn, error, silent

Log all TTV requests and responses to markdown files for debugging:

import { TTVDebugger } from '@loonylabs/ttv-middleware';

// Enable via environment variable
// DEBUG_TTV_REQUESTS=true

// Or programmatically
TTVDebugger.setEnabled(true);
TTVDebugger.setLogsDir('./logs/ttv/requests');

TTVDebugger.configure({
  enabled: true,
  logsDir: './logs/ttv/requests',
  consoleLog: true,
});

Typed error classes for precise error handling:

import {
  TTVError,
  InvalidConfigError,
  QuotaExceededError,
  ProviderUnavailableError,
  GenerationFailedError,
  NetworkError,
  CapabilityNotSupportedError,
  PollingTimeoutError,
  ContentModeratedError,
} from '@loonylabs/ttv-middleware';

try {
  const result = await service.generate({ prompt: 'test', duration: 8 });
} catch (error) {
  if (error instanceof PollingTimeoutError) {
    console.log('Video generation timed out - try again or increase timeout');
  } else if (error instanceof ContentModeratedError) {
    console.log('Content was blocked by safety filters');
  } else if (error instanceof QuotaExceededError) {
    console.log('Rate limit hit, try again later');
  } else if (error instanceof CapabilityNotSupportedError) {
    console.log('Model does not support this feature');
  } else if (error instanceof TTVError) {
    console.log(`TTV Error [${error.code}]: ${error.message}`);
  }
}

Test your integration without making API calls or incurring costs:

const result = await service.generate({
  prompt: 'A test video',
  duration: 8,
  dry: true,  // No API call, returns placeholder response
});

console.log(result.videos.length);       // 1
console.log(result.metadata.duration);   // 0 (no actual generation)

Use providerOptions as an escape hatch for provider-specific features:

// Google Veo: seed for deterministic output
const result = await service.generate({
  prompt: 'A sunset timelapse',
  model: 'veo-3.0-generate-001',
  providerOptions: {
    seed: 42,
    personGeneration: 'allow_adult',
    enhancePrompt: true,
    storageUri: 'gs://my-bucket/output/',  // Direct output to GCS
  },
}, TTVProvider.GOOGLE_VEO);

// OpenAI Sora: remix (reinterpret an existing video)
// Use the Sora API directly for remix via providerOptions

Testing

# Run all tests
npm test

# Unit tests only
npm run test:unit

# Unit tests with watch mode
npm run test:unit:watch

# Unit tests with coverage report
npm run test:unit:coverage

# Integration tests (requires TTV_INTEGRATION_TESTS=true)
npm run test:integration

# CI/CD mode
npm run test:ci

Manual Smoke Tests

The scripts/ directory contains manual smoke tests for real API calls:

# Text-to-video with Google Veo (veo-3.0-fast)
npx ts-node scripts/manual-test-veo.ts

# Image-to-video with Google Veo (veo-3.0-fast)
npx ts-node scripts/manual-test-veo-i2v.ts

Both scripts use veo-3.0-fast-generate-001 (cheapest model), generate short 4s/720p videos, and save output to the output/ directory. Requires Google Veo credentials in .env.

Integration Tests

Integration tests make real API calls and cost money. They are skipped by default.

# Enable and run integration tests
TTV_INTEGRATION_TESTS=true npm run test:integration

Prerequisites:

• OPENAI_API_KEY for Sora tests
• GOOGLE_CLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS for Veo tests

Contributing

We welcome contributions! Please ensure:

1. Tests: Add tests for new features

2. Linting: Run npm run lint before committing

3. Conventions: Follow the existing project structure

4. Fork the repository

5. Create your feature branch (git checkout -b feature/amazing-feature)

6. Commit your changes (git commit -m 'Add some amazing feature')

7. Push to the branch (git push origin feature/amazing-feature)

8. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Links

• Documentation
• Issues
• NPM Package

Made with care by the LoonyLabs Team