video-generation-mcp

MCP server for video generation APIs (Runway, Pika, Luma, Kling, MiniMax).

Overview

This package provides a unified MCP (Model Context Protocol) interface for multiple video generation providers. It allows AI assistants to generate videos from text prompts, images, or existing videos through a consistent API.

Installation

npm install
npm run build

Usage

As an MCP Server (Recommended)

Add to your mcp-proxy configuration (~/.mcp-proxy/mcp-config.json):

{
  "video-generation": {
    "command": "node",
    "args": ["/path/to/video-generation-mcp/dist/index.js"],
    "transport": "stdio",
    "idleTimeout": 0,
    "env": {
      "REPLICATE_API_TOKEN": "${REPLICATE_API_TOKEN}",
      "RUNWAY_API_KEY": "${RUNWAY_API_KEY}",
      "PIKA_API_KEY": "${PIKA_API_KEY}"
    }
  }
}

Environment Variables

Configure API keys for the providers you want to use:

| Variable | Provider | Required | |----------|----------|----------| | REPLICATE_API_TOKEN | Luma Ray via Replicate | For Luma | | RUNWAY_API_KEY | Runway ML | For Runway Gen-3 | | PIAPI_API_KEY | PiAPI Aggregator | For Kling/Hailuo | | PIKA_API_KEY | Pika Labs | For Pika | | KLING_API_KEY | Kling AI (direct) | For Kling direct | | MINIMAX_API_KEY | MiniMax | For MiniMax |

Tools

This server provides 6 MCP tools for video generation workflows:

video_text_to_video

Generate a video from a text prompt. Returns a job_id to track progress.

Parameters:

• prompt (required): Text description of the video to generate
• provider: Video generation provider (luma, runway, pika, kling, minimax). Default: luma
• duration_seconds: Desired video duration (provider limits apply)
• aspect_ratio: Video aspect ratio (16:9, 9:16, 1:1, 4:3, 21:9). Default: 16:9
• style_options: Provider-specific style options (negative_prompt, seed, cfg_scale)

Returns: { success: true, job_id, status, provider, message }

video_image_to_video

Animate a static image into a video. The image becomes the first frame.

Parameters:

• image_url (required): URL of the input image (must be publicly accessible)
• prompt (required): Description of the motion to apply
• provider: Video generation provider. Default: luma
• duration_seconds: Desired video duration
• motion_params: Motion control parameters (motion_strength, camera_motion, end_image_url)

Returns: { success: true, job_id, status, provider, message }

video_check_status

Check the status of a video generation job.

Parameters:

• job_id (required): The job ID from video_text_to_video or video_image_to_video
• provider (required): The provider that created the job

Returns: { success: true, job_id, status, progress_percent, video_url, message }

video_download

Download a completed video to a local file.

Parameters:

• job_id (required): The job ID of a completed video
• provider (required): The provider that created the job
• output_path: Local file path (default: /tmp/mcp-media/video/{job_id}.mp4)

Returns: { success: true, file_path, file_size_bytes, file_size_mb, message }

video_list_providers

List available video generation providers with their capabilities.

Parameters:

• filter_by_capability: Optional filter (text_to_video, image_to_video, video_to_video)

Returns: { success: true, providers, count, message }

video_get_pricing

Get estimated cost for video generation with specific parameters.

Parameters:

• provider (required): The provider to get pricing for
• duration_seconds: Video duration in seconds. Default: 5
• resolution: Video resolution (720p, 1080p, 4k). Default: 1080p
• input_type: Input type for generation (text, image, video). Default: text

Returns: { success: true, provider, estimated_cost_usd, pricing_model, parameters, disclaimer }

Response Format

All tools return a consistent response format:

Success:

{
  "success": true,
  "job_id": "pred_abc123",
  "status": "pending",
  "message": "Video generation started. Use video_check_status to track progress."
}

Error:

{
  "success": false,
  "error": "Provider \"invalid\" is not yet implemented. Use video_list_providers to see available providers.",
  "details": {
    "provider": "invalid"
  }
}

Provider Status

| Provider | Status | Implementation | |----------|--------|----------------| | Luma | Implemented | Via Replicate API (luma/ray model) | | Runway | Implemented | Direct API (Gen-4 Turbo, Veo3.1) | | Kling | Implemented | Via PiAPI (Kling 2.5) | | Hailuo | Implemented | Via PiAPI (Hailuo v2.3) | | Pika | Planned | Direct API | | MiniMax | Planned | Direct API |

Provider Capabilities

| Provider | Text-to-Video | Image-to-Video | Video-to-Video | Max Duration | Notes | |----------|--------------|----------------|----------------|--------------|-------| | Luma | Yes | Yes | No | 5s | Via Replicate | | Runway | Yes | Yes | No | 10s | Gen-4 Turbo (image), Veo3.1 (text) | | Kling | Yes | Yes | No | 10s | Via PiAPI | | Hailuo | Yes | Yes | No | 10s | Via PiAPI | | Pika | Yes | Yes | No | 4s | Planned | | MiniMax | Yes | Yes | No | 6s | Planned |

PiAPI Examples (Kling / Hailuo)

Text-to-Video with Kling

// Using MCP tools
const response = await callTool('video_generate', {
  provider: 'piapi',
  input_type: 'text',
  prompt: 'A futuristic city with flying cars and neon lights at night',
  duration_seconds: 10,
  aspect_ratio: '16:9',
  // provider_options for model selection
  provider_options: { model: 'kling' },
});

// Check status
const status = await callTool('video_check_status', {
  provider: 'piapi',
  job_id: response.job_id,
});

Text-to-Video with Hailuo (Cinematic Quality)

const response = await callTool('video_generate', {
  provider: 'piapi',
  input_type: 'text',
  prompt: 'A dramatic sunset over mountain ranges, cinematic lighting',
  duration_seconds: 6,
  aspect_ratio: '16:9',
  provider_options: { model: 'hailuo' },
});

Image-to-Video with Kling

const response = await callTool('video_generate', {
  provider: 'piapi',
  input_type: 'image',
  image_url: 'https://example.com/portrait.jpg',
  prompt: 'Gentle head movement, natural blinking, subtle smile',
  provider_options: { model: 'kling' },
});

Using PiAPI Adapter Directly (Node.js)

import { PiAPIAdapter, createKlingAdapter, createHailuoAdapter } from 'video-generation-mcp';

// Option 1: Generic adapter with model selection per request
const adapter = new PiAPIAdapter({
  apiKey: process.env.PIAPI_API_KEY,
  debug: true,
});

const result = await adapter.generateVideo({
  input_type: 'text',
  prompt: 'A robot dancing in a factory',
  duration_seconds: 10,
  provider_options: { model: 'kling' },
});

// Option 2: Pre-configured Kling adapter
const klingAdapter = createKlingAdapter({ apiKey: process.env.PIAPI_API_KEY });
await klingAdapter.generateVideo({
  input_type: 'text',
  prompt: 'Ocean waves at sunset',
});

// Option 3: Pre-configured Hailuo adapter
const hailuoAdapter = createHailuoAdapter({ apiKey: process.env.PIAPI_API_KEY });
await hailuoAdapter.generateVideo({
  input_type: 'text',
  prompt: 'Cinematic mountain landscape',
});

Model Comparison: Kling vs Hailuo

| Feature | Kling | Hailuo | |---------|-------|--------| | Model versions | 1.5, 1.6, 2.1, 2.5, 2.6 | v2.3, v2.3-fast | | Max duration | 10 seconds | 10 seconds | | Default duration | 5 seconds | 6 seconds | | Native audio | Yes (v2.6) | No | | Negative prompt | Yes | No | | Character consistency | Excellent | Good | | Cinematic quality | Good | Excellent | | Physics simulation | Good | Excellent |

Luma Ray Examples

Text-to-Video

// Using the MCP tools directly
const response = await callTool('video_generate', {
  provider: 'luma',
  input_type: 'text',
  prompt: 'A cat walking through a garden with sunlight filtering through the leaves',
  aspect_ratio: '16:9',
});

// response.job_id can be used to check status
const status = await callTool('video_check_status', {
  provider: 'luma',
  job_id: response.job_id,
});

// When completed, download the video
if (status.status === 'completed') {
  const download = await callTool('video_download', {
    provider: 'luma',
    job_id: response.job_id,
    output_path: '/tmp/my-video.mp4',
  });
}

Image-to-Video

const response = await callTool('video_generate', {
  provider: 'luma',
  input_type: 'image',
  prompt: 'A gentle breeze moves the trees and clouds drift across the sky',
  image_url: 'https://example.com/landscape.jpg',
  aspect_ratio: '16:9',
});

Using the Adapter Directly (Node.js)

import { ReplicateAdapter } from 'video-generation-mcp';

const adapter = new ReplicateAdapter({
  apiToken: process.env.REPLICATE_API_TOKEN,
  debug: true, // Enable debug logging
});

// Generate a video
const result = await adapter.generateVideo({
  input_type: 'text',
  prompt: 'Ocean waves crashing on a beach at sunset',
  aspect_ratio: '16:9',
});

// Poll until complete
const completed = await adapter.pollUntilComplete(result.job_id, {
  timeout_ms: 300000, // 5 minutes
  initial_interval_ms: 5000,
});

// Download the video
const filePath = await adapter.downloadVideo(completed.job_id);
console.log(`Video saved to: ${filePath}`);

Runway Gen-4 Examples

Environment Setup

export RUNWAY_API_KEY="your-api-key-from-runway-developer-portal"

Text-to-Video with Veo3.1

// Using the MCP tools
const response = await callTool('video_generate', {
  provider: 'runway',
  input_type: 'text',
  prompt: 'A cinematic timelapse of clouds moving across a mountain range at sunset',
  duration_seconds: 8,
  aspect_ratio: '21:9', // Ultra-wide cinematic
});

// Check status
const status = await callTool('video_check_status', {
  provider: 'runway',
  job_id: response.job_id,
});

Image-to-Video with Gen-4 Turbo

const response = await callTool('video_generate', {
  provider: 'runway',
  input_type: 'image',
  prompt: 'Camera slowly zooms in while clouds drift in the background',
  image_url: 'https://example.com/landscape.jpg',
  duration_seconds: 5,
  aspect_ratio: '16:9',
  seed: 42, // For reproducibility
});

Using the RunwayAdapter Directly (Node.js)

import { RunwayAdapter } from 'video-generation-mcp';

const adapter = new RunwayAdapter(); // Uses RUNWAY_API_KEY from environment

// Text-to-video with veo3.1
const textResult = await adapter.generateVideo({
  input_type: 'text',
  prompt: 'A futuristic city with flying vehicles',
  duration_seconds: 6,
  aspect_ratio: '16:9',
  provider_options: {
    model: 'veo3.1',      // Default for text-to-video
    audio: true,           // Include audio generation (default: true)
  },
});

// Image-to-video with gen4_turbo
const imageResult = await adapter.generateVideo({
  input_type: 'image',
  image_url: 'https://example.com/photo.jpg',
  prompt: 'Add gentle camera movement',
  duration_seconds: 5,
  seed: 12345,
  provider_options: {
    model: 'gen4_turbo',   // Default for image-to-video
  },
});

// Poll until complete
const completed = await adapter.pollUntilComplete(imageResult.job_id);

// Download the video
const filePath = await adapter.downloadVideo(completed.job_id, '/tmp/output.mp4');

Runway Models

| Model | Type | Description | |-------|------|-------------| | veo3.1 | Text-to-video | Default text-to-video model with audio support | | veo3.1_fast | Text-to-video | Faster variant of veo3.1 | | veo3 | Text/Image | Supports both text and image-to-video | | gen4_turbo | Image-to-video | Default for image-to-video, fastest Gen-4 model | | gen3a_turbo | Image-to-video | Legacy model |

Runway Duration Limits

• Text-to-video: 4, 6, or 8 seconds
• Image-to-video: 2-10 seconds

Runway Aspect Ratios

| Aspect Ratio | Text-to-Video | Image-to-Video | |--------------|---------------|----------------| | 16:9 | 1920x1080 | 1280x720 | | 9:16 | 1080x1920 | 720x1280 | | 1:1 | 960x960 | 960x960 | | 4:3 | 1104x832 | 1104x832 | | 21:9 | 1584x672 | 1584x672 |

Error Handling

The adapter throws specific error types for different failure modes:

import {
  VideoGenerationError,
  AuthenticationError,
  RateLimitError,
} from 'video-generation-mcp';

try {
  await adapter.generateVideo({ ... });
} catch (error) {
  if (error instanceof AuthenticationError) {
    console.error('Invalid API token');
  } else if (error instanceof RateLimitError) {
    console.error(`Rate limited. Retry after ${error.retryAfterMs}ms`);
  } else if (error instanceof VideoGenerationError) {
    console.error(`Generation failed: ${error.message}`);
    console.error(`HTTP status: ${error.statusCode}`);
  }
}

Development

# Build
npm run build

# Watch mode
npm run build:watch

# Run tests
npm test

# Watch tests
npm run test:watch

# Clean build artifacts
npm run clean

Job Manager Service

The package includes a centralized async job manager for tracking video generation jobs across all providers.

Features

• Unified job tracking - Track jobs from all providers (Replicate, Runway, PiAPI/Kling/Hailuo)
• File-based persistence - Jobs persist across server restarts with atomic writes
• Automatic timeout detection - Jobs are marked as timed out after configurable duration (default 10 minutes)
• Automatic cleanup - Expired jobs removed automatically in background
• Filtering - Query jobs by status, provider, or time range

Usage

import { JobManager, type Job, type JobFilter } from 'video-generation-mcp';

// Create and initialize the job manager
const jobManager = new JobManager({
  dataDir: '/path/to/data',      // Default: /tmp/video-generation-mcp
  defaultTimeoutMs: 600000,       // Default: 10 minutes
  cleanupIntervalMs: 60000,       // Default: 1 minute (0 to disable)
});
await jobManager.initialize();

// Create a new job
const jobId = jobManager.createJob('replicate', {
  type: 'text-to-video',
  prompt: 'A flying cat',
  options: { aspect_ratio: '16:9' },
}, { timeoutMs: 300000 }); // Optional custom timeout

// Update job status
jobManager.updateJobStatus(jobId, 'submitted', {
  providerJobId: 'replicate-abc123',
});

jobManager.updateJobStatus(jobId, 'succeeded', {
  result: {
    videoUrl: 'https://cdn.example.com/video.mp4',
    duration: 5.0,
  },
});

// Get a single job (also checks for timeout)
const job = jobManager.getJob(jobId);

// List jobs with filtering
const pendingJobs = jobManager.listJobs({
  status: 'pending',
  provider: 'replicate',
  limit: 10,
});

const recentJobs = jobManager.listJobs({
  status: ['succeeded', 'failed'],
  createdAfter: Date.now() - 24 * 60 * 60 * 1000, // Last 24 hours
});

// Get statistics
const stats = jobManager.getStats();
console.log(`Total: ${stats.total}, Pending: ${stats.byStatus.pending}`);

// Cleanup expired jobs manually
const cleaned = await jobManager.cleanupExpiredJobs(24 * 60 * 60 * 1000); // 24h retention

// Graceful shutdown
await jobManager.shutdown();

Job Status Values

| Status | Description | |--------|-------------| | pending | Job created, not yet submitted to provider | | submitted | Sent to provider API, awaiting processing | | processing | Provider is actively generating video | | succeeded | Video generation complete, ready for download | | failed | Generation failed with error | | timeout | Job exceeded maximum wait time |

Filter Options

interface JobFilter {
  status?: JobManagerStatus | JobManagerStatus[];  // Filter by status(es)
  provider?: JobManagerProvider | JobManagerProvider[];  // Filter by provider(s)
  createdAfter?: number;   // Unix timestamp ms
  createdBefore?: number;  // Unix timestamp ms
  limit?: number;          // Pagination limit
  offset?: number;         // Pagination offset
}

Persistence

Jobs are persisted to {dataDir}/jobs.json using atomic writes (temp file + rename) to prevent corruption. The file is loaded on initialize() and saved after each modification.

Architecture

src/
├── index.ts          # MCP server entry point
├── types.ts          # Shared type definitions
├── adapters/
│   ├── base.ts       # Abstract adapter interface
│   ├── replicate.ts  # Replicate adapter (Luma Ray)
│   ├── piapi.ts      # PiAPI aggregator adapter (Kling, Hailuo)
│   └── runway.ts     # Runway adapter (Gen-4 Turbo, Veo3.1)
└── services/
    ├── index.ts          # Service exports
    ├── credentials.ts    # Credential management
    ├── rate-limiter.ts   # Rate limiting
    └── job-manager.ts    # Async job manager

tests/
├── adapters/
│   ├── replicate.test.ts  # Replicate adapter unit tests
│   ├── piapi.test.ts      # PiAPI adapter unit tests
│   └── runway.test.ts     # Runway adapter unit tests
└── services/
    ├── credentials.test.ts  # Credential manager tests
    └── job-manager.test.ts  # Job manager tests

Pricing

• Luma Ray via Replicate: ~$0.45 per video generation
• Video output URLs from Replicate expire after ~1 hour, so download promptly
• PiAPI (Kling/Hailuo): Pay-as-you-go credits via piapi.ai
• Runway: Credit-based pricing via runwayml.com (Gen-4 Turbo uses premium credits)

Troubleshooting

Authentication Errors

Error: Authentication failed for <provider>. Check API key.

Solution:

1. Verify the correct environment variable is set for your provider
2. Check that the API key is valid and not expired
3. Ensure there are no trailing spaces or newlines in your API key
4. For Replicate, ensure your account has billing enabled

Rate Limiting

Error: Rate limited by <provider>

Solution:

1. Wait a moment before retrying (most providers have 1-minute windows)
2. Check your plan limits on the provider's dashboard
3. Consider upgrading your plan for higher rate limits
4. Use video_list_providers to see rate limit information

Job Timeout

Error: Polling timeout after Xs. Job status: processing, progress: Y%

Solution:

1. Video generation can take several minutes - try checking status manually
2. Complex prompts or high resolutions take longer
3. Provider may be experiencing delays - check their status page
4. Try with shorter duration or lower resolution

Video URL Expired

Error: Failed to download video or empty video_url

Solution:

1. Replicate URLs expire after ~1 hour - download immediately after completion
2. Use video_check_status to get a fresh URL
3. If job is still valid, the URL should be regenerated

Provider Not Implemented

Error: Provider "<name>" is not yet implemented

Solution:

1. Use video_list_providers to see which providers are available
2. Check the Provider Status table in this README
3. Use an implemented provider: luma, runway, or piapi (for Kling/Hailuo)

Invalid Input Parameters

Error: Invalid input: <validation details>

Solution:

1. Check parameter types match the schema (strings, numbers, etc.)
2. Ensure URLs are valid and publicly accessible
3. Verify aspect_ratio is one of: 16:9, 9:16, 1:1, 4:3, 21:9
4. Ensure duration_seconds is within provider limits

Image URL Issues (Image-to-Video)

Error: Failed to fetch image or blank video

Solution:

1. Use HTTPS URLs (not HTTP)
2. Ensure the image URL is publicly accessible (no auth required)
3. Check that the image format is supported (JPG, PNG, WebP)
4. Use images at least 512x512 pixels for best results
5. Avoid images with watermarks or text

Debug Mode

Enable detailed logging for troubleshooting:

// For adapters
const adapter = new ReplicateAdapter({ debug: true });

// For rate limiter
const limiter = new RateLimiter({ debug: true });

Testing

Run the test suite:

npm test                 # Run all tests
npm run test:watch       # Watch mode
npm run test:coverage    # With coverage report

Coverage Targets:

• Overall: 80%+
• Adapters: 90%+
• Services: 85%+
• Tools: 80%+

License

Private - Internal use only