@txtvid/cogvideo

A powerful TypeScript library for text-to-video and image-to-video generation using the open-source CogVideoX model. Inspired by the API design of @huggingface/transformers.

Features

• 🎬 Real AI Video Generation - Uses actual CogVideoX models (2B and 5B variants)
• 📝 Text-to-Video - Generate videos from text descriptions
• 🖼️ Image-to-Video - Animate still images with motion
• ⚡ GPU Accelerated - CUDA support for fast inference
• 💾 Memory Optimization - CPU offloading for limited VRAM
• 📊 Progress Tracking - Real-time generation progress via events
• 🎞️ Video Utilities - Save, convert, and process videos
• 🔧 TypeScript Native - Full type safety and IDE support
• 🌐 Cross-Platform - Works on Windows, Linux, macOS

Installation

1. Install npm package

npm install @txtvid/cogvideo

2. Install Python dependencies

The library requires Python backend for running the CogVideoX model:

# Install Python dependencies
npm run install-python

# Or manually:
pip install -r node_modules/@txtvid/cogvideo/python/requirements.txt

Python Requirements:

• Python 3.8 or higher
• PyTorch 2.0+
• CUDA-capable GPU (recommended, 8GB+ VRAM)

3. Install FFmpeg (optional, for video processing)

# Ubuntu/Debian
sudo apt-get install ffmpeg

# macOS
brew install ffmpeg

# Windows
# Download from https://ffmpeg.org/download.html

Quick Start

CPU-Based Generation (No GPU Required)

import { TxtVid, CogVideoModel } from '@txtvid/cogvideo';

const video = await TxtVid.textToVideo({
  pipeline: {
    model: CogVideoModel.COGVIDEOX_2B,
    device: 'cpu',      // Use CPU
    dtype: 'float32',   // CPU doesn't support float16 well
  },
  params: {
    prompt: 'A cat walking on the beach at sunset',
    numInferenceSteps: 30,  // Reduced for faster CPU generation
    guidanceScale: 6.0,
  },
  save: {
    outputPath: './output.mp4',
  },
});

console.log(`Generated in ${video.metadata.generationTime}s on ${video.metadata.device}`);

⚠️ CPU Performance Note: CPU generation is significantly slower (5-15 minutes per video). For faster results:

• Reduce numInferenceSteps to 20-30
• Use lower resolution (480p)
• Reduce numFrames if needed

GPU-Based Generation (Recommended)

import { TxtVid, CogVideoModel } from '@txtvid/cogvideo';

const video = await TxtVid.textToVideo({
  pipeline: {
    model: CogVideoModel.COGVIDEOX_2B,
    device: 'cuda',
  },
  params: {
    prompt: 'A cat walking on the beach at sunset, cinematic lighting',
    numInferenceSteps: 50,
    guidanceScale: 6.0,
  },
  save: {
    outputPath: './output.mp4',
  },
});

console.log(`Generated ${video.numFrames} frames in ${video.metadata.generationTime}s`);

Using Pipeline API (like @huggingface/transformers)

import { TxtVid, CogVideoModel } from '@txtvid/cogvideo';

// Create pipeline
const pipe = await TxtVid.textToVideoPipeline({
  model: CogVideoModel.COGVIDEOX_2B,
  device: 'cuda',
  dtype: 'float16',
});

// Listen to progress
pipe.on('progress', (progress) => {
  console.log(`Progress: ${progress.percentage.toFixed(1)}%`);
});

// Generate video
const video = await pipe.generateVideoFromText(
  'A serene Japanese garden with cherry blossoms falling',
  {
    negativePrompt: 'blurry, low quality',
    numInferenceSteps: 50,
    guidanceScale: 6.0,
    numFrames: 49,
    height: 480,
    width: 720,
    seed: 42,
  }
);

// Save video
const { VideoUtils } = await import('@txtvid/cogvideo');
await VideoUtils.saveVideo(video, {
  outputPath: './garden.mp4',
});

Image-to-Video

import { TxtVid, CogVideoModel } from '@txtvid/cogvideo';

const video = await TxtVid.imageToVideo({
  pipeline: {
    model: CogVideoModel.COGVIDEOX_5B_I2V,
    device: 'cuda',
  },
  params: {
    image: './input.jpg',
    prompt: 'Gentle camera pan to the right with subtle movement',
    numInferenceSteps: 50,
    guidanceScale: 6.0,
    strength: 0.8,
  },
  save: {
    outputPath: './output.mp4',
  },
});

API Reference

TxtVid Class

High-level API for quick video generation:

• TxtVid.textToVideo(request) - Generate video from text (one-liner)
• TxtVid.imageToVideo(request) - Generate video from image (one-liner)
• TxtVid.textToVideoPipeline(config) - Create reusable text-to-video pipeline
• TxtVid.imageToVideoPipeline(config) - Create reusable image-to-video pipeline
• TxtVid.isCudaAvailable() - Check if CUDA is available

Pipeline API

Similar to @huggingface/transformers:

const pipe = await TxtVid.textToVideoPipeline({
  model: CogVideoModel.COGVIDEOX_2B,
  device: 'cuda',
  dtype: 'float16',
  cpuOffload: false,
  cacheDir: './models',
});

// Generate
const video = await pipe.generateVideoFromText('Your prompt', {
  numInferenceSteps: 50,
  guidanceScale: 6.0,
  onProgress: (progress) => console.log(progress),
});

// Cleanup
await pipe.dispose();

Available Models

enum CogVideoModel {
  COGVIDEOX_2B = 'THUDM/CogVideoX-2b',           // Faster, 2B parameters
  COGVIDEOX_5B = 'THUDM/CogVideoX-5b',           // Higher quality, 5B parameters
  COGVIDEOX_5B_I2V = 'THUDM/CogVideoX-5b-I2V',   // Image-to-video variant
}

Video Utilities

import { VideoUtils } from '@txtvid/cogvideo';

// Save video
await VideoUtils.saveVideo(video, {
  outputPath: './output.mp4',
  overwrite: true,
});

// Convert to GIF
await VideoUtils.toGif(video, './output.gif', fps=8);

// Extract frames
const frames = await VideoUtils.extractFrames(video, './frames/');

// Get metadata
const metadata = await VideoUtils.getVideoMetadata(video);

// Concatenate videos
const combined = await VideoUtils.concatenate([video1, video2], './combined.mp4');

Configuration Options

Pipeline Configuration

interface PipelineConfig {
  model: CogVideoModel | string;       // Model to use
  device?: 'cpu' | 'cuda' | 'mps';     // Device for inference
  dtype?: 'float16' | 'float32' | 'bfloat16';  // Precision
  cpuOffload?: boolean;                // Enable CPU offloading
  sequentialOffload?: boolean;         // Sequential offloading (low VRAM)
  cacheDir?: string;                   // Model cache directory
  progressBar?: boolean;               // Show progress bars
}

Generation Parameters

interface TextToVideoParams {
  prompt: string;                      // Text description
  negativePrompt?: string;             // What to avoid
  numInferenceSteps?: number;          // Default: 50
  guidanceScale?: number;              // Default: 6.0
  numFrames?: number;                  // Default: 49
  height?: number;                     // Default: 480
  width?: number;                      // Default: 720
  seed?: number;                       // For reproducibility
  numVideos?: number;                  // Number of videos
}

Examples

Check out the examples directory:

• cpu-example.ts - CPU-based generation (no GPU required)
• node-example.ts - Text-to-video examples (GPU)
• image-to-video-example.ts - Image animation examples

Run examples:

# CPU example (no GPU needed)
npx ts-node examples/cpu-example.ts

# GPU examples
npm run example:node

# Image-to-video
npx ts-node examples/image-to-video-example.ts

Performance Tips

GPU Memory Optimization

For limited VRAM (< 8GB):

const pipe = await TxtVid.textToVideoPipeline({
  model: CogVideoModel.COGVIDEOX_2B,
  device: 'cuda',
  sequentialOffload: true,  // Minimal VRAM usage
  dtype: 'float16',         // Half precision
});

Faster Generation

• Use COGVIDEOX_2B instead of COGVIDEOX_5B
• Reduce numInferenceSteps (30-40 for draft quality)
• Lower resolution (480p instead of 720p)
• Use torch.compile optimization (enabled by default)

Requirements

Hardware

For CPU Generation (No GPU Required):

• CPU: Any modern multi-core processor
• RAM: 16GB+ recommended (8GB minimum)
• Storage: 10GB+ for model weights
• ⚠️ Performance: 5-15 minutes per video

For GPU Generation (Recommended):

• GPU: NVIDIA GPU with 8GB+ VRAM (RTX 3060 or better)
• CPU: Any modern processor
• RAM: 16GB+
• Storage: 10GB+ for model weights
• ✅ Performance: 30 seconds - 2 minutes per video

Software

• Node.js 18+
• Python 3.8+
• PyTorch 2.0+
• CUDA 11.8+ (for GPU acceleration)

Architecture

This library uses a TypeScript-to-Python bridge architecture:

1. TypeScript Layer - User-friendly API, type safety, event handling
2. Python Bridge - Spawns Python subprocess for model execution
3. Python Backend - Runs CogVideoX via Hugging Face Diffusers
4. Video Utils - FFmpeg integration for video processing

This design ensures:

• ✅ Real AI model execution (no simulation)
• ✅ Full access to CogVideoX capabilities
• ✅ TypeScript developer experience
• ✅ Cross-platform compatibility

Development

# Clone repository
git clone https://github.com/txtvid/txtvid-cogvideo.git
cd txtvid-cogvideo

# Install dependencies
npm install

# Build
npm run build

# Watch mode
npm run build:watch

# Run tests
npm test

License

Apache-2.0

Acknowledgments

• CogVideoX - Open-source video generation model by THUDM
• Hugging Face Diffusers - Model pipeline implementation
• @huggingface/transformers - API design inspiration

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

Made with ❤️ by the TxtVid Team