TwelveLabs MCP Server

Overview

The TwelveLabs MCP Server is a Model Context Protocol (MCP) server that provides seamless integration with the TwelveLabs platform. This server enables AI assistants and applications to interact with TwelveLabs powerful video analysis capabilities through a standardized MCP interface.

What it does

This server provides tools for:

• Video Index Management: Create, list, and delete video indexes for organizing video content
• Video Indexing: Upload and index videos (single file, folder, or URL) to make them searchable and analyzable
• Video Search: Search through indexed videos using natural language queries
• Video Analysis: Generate titles, topics, hashtags, summaries, chapters, and highlights from videos
• Video Embeddings: Create and retrieve video embeddings for advanced AI applications

Key Features

• Natural Language Search: Search videos using descriptive queries that understand visual elements, actions, sounds, and on-screen text
• AI-Powered Analysis: Generate comprehensive video insights including summaries, chapters, and highlights
• Flexible Indexing: Support for both generative and embedding models
• Local Folder Indexing: Index all video files in a local folder in a single operation
• User Metadata: Attach custom key-value metadata to videos at indexing time, and retrieve it in search results
• Thumbnail Support: Indexes created with the thumbnail addon (enabled by default) return per-segment thumbnail URLs in search results
• Task Management: Monitor and track video processing tasks
• Default Index Handling: Automatic creation and management of default indexes

Available Prompts

Twelvelabs Onboarding

A prompt that helps new users onboard on the TwelveLabs Platform.

Available Tools

Index Management

create-index

Create a new TwelveLabs video index for organizing and storing video data.

Parameters:

• name (string, required): Name of the index
• models (array, optional): Type of models to use. Options: ["generative", "embedding"]. Defaults to both models if not specified.
• addons (array, optional): Addons to enable. Options: ["thumbnail"]. Defaults to ["thumbnail"] if not specified. Thumbnail addon generates per-segment thumbnail URLs in search results.

list-indexes

List all available TwelveLabs video indexes.

Parameters:

• page (number, optional): Page number. Defaults to the first page (10 indexes per page).

delete-index

Delete a TwelveLabs video index.

Parameters:

• indexId (string, required): ID of the index to delete

Video Indexing

start-video-indexing-task

Start indexing a video to make it searchable and analyzable. Supports remote URLs, local file uploads, and batch indexing from a local folder.

Parameters:

• videoUrl (string, optional): URL of the video to index. Supports direct URLs and Google Drive links (file or public folder).
• videoFilePath (string, optional): Absolute path to a local video file to upload (e.g., /path/to/video.mp4). Only available with stdio transport (twelvelabs-mcp-stdio).
• folderFilePath (string, optional): Absolute path to a local folder containing video files. All supported video files (.mp4, .mov, .avi, .mkv, .webm) in the folder will be indexed. Only available with stdio transport.
• indexId (string, optional): ID of the index. If not provided, uses the default index.
• userMetadata (string, optional): JSON string of key-value pairs to attach as metadata to the video(s). Keys must be strings, values can be string, integer, float or boolean. Example: '{"source_path": "/path/to/file", "project": "my_project"}'

Note: Provide either videoUrl, videoFilePath, or folderFilePath. If videoUrl is provided, it takes precedence.

get-video-indexing-tasks

Get the status of video indexing tasks.

Parameters:

• taskId (string, optional): ID of the specific task. If not provided, returns status of the latest 10 tasks.

Possible Statuses:

• Validating: Video upload complete, API validating requirements
• Pending: Platform spawning worker server
• Queued: Worker assigned, preparing to index
• Indexing: Transforming video into embeddings
• Ready: Indexing complete
• Failed: Indexing failed

list-videos

List all videos in a video index.

Parameters:

• indexId (string, optional): ID of the index. If not provided, uses the default index.
• page (number, optional): Page number. Defaults to the first page.

Video Search

search

Search through indexed videos using natural language descriptions. Returns matching segments with relevance ranking, user metadata, and thumbnail URLs.

Parameters:

• query (string, optional): Natural language search query. Required for text-only and composed searches.
• indexId (string, optional): ID of the index to search. If not provided, searches the default index.
• queryMediaUrl (string, optional): URL of an image to search with. For composed search, provide both this and a text query.
• queryMediaFile (string, optional): Absolute path to a local image file to search with. Alternative to queryMediaUrl.
• queryMediaType (enum, optional): Type of media query. Currently only "image" is supported.

Returns: For each matching video: filename, videoId, userMetadata (if attached during indexing), and a list of segments with rank, start/end times, and thumbnail URLs (if the index has the thumbnail addon enabled).

Video Analysis

sync-analyse-video

Synchronously analyse a pre-indexed video with Pegasus 1.2 and return the generated text inline. Requires a videoId pointing at a video already indexed in a Pegasus-enabled index (≤ 1 hour duration).

If you don't already have a videoId — or your video is longer than 1 hour, or you want structured time-based metadata, or multimodal reference-image prompting — use async-analyse-video instead.

Parameters:

• videoId (string, required): ID of an indexed video to analyse.
• prompt (string, required): Custom prompt to guide the analysis (max 2000 tokens).

async-analyse-video

Asynchronously analyse a video with Pegasus 1.5 (default). Accepts a URL, asset ID, or base64 directly — no prior indexing required. Handles videos up to 2 hours (longer videos are rejected by the Twelve Labs API with a 400). Fire-and-forget: returns a taskId immediately, poll with get-analyse-task.

Use this when you don't have a videoId, your video is over 1 hour, you want time-based metadata segmentation, or you want multimodal reference-image prompting.

Pegasus 1.5 supports both prompt-only general analysis and time_based_metadata segmentation. Pegasus 1.2 is available via modelName="pegasus1.2" for prompt-only analysis.

Parameters:

• videoUrl (string, optional): Direct http(s) URL to a raw media file. Share links (YouTube / Drive / Dropbox) are not accepted.
• assetId (string, optional): ID of a previously uploaded asset. Must be in status ready. See Uploading a video as an asset for how to obtain one.
• base64Video (string, optional): Base64-encoded video data, max 30 MB.
• modelName ("pegasus1.2" | "pegasus1.5", optional): Defaults to pegasus1.5.
• prompt (string, optional): Natural-language prompt for general-mode analysis (max 2000 tokens). Works on both 1.2 and 1.5. Not allowed with analysisMode="time_based_metadata".
• temperature (number, optional): 0–1. Default 0.2.
• maxTokens (integer, optional): Pegasus 1.2: 1–4096. Pegasus 1.5: 2048–32768 (default 32768).
• analysisMode ("time_based_metadata", optional): Requires modelName="pegasus1.5" (the default) and segmentDefinitions.
• segmentDefinitions (array, optional): 1–10 entries. Each has id, description, optional fields[] (max 20) and mediaSources[] (max 4 reference images for multimodal prompting).
• jsonSchema (object, optional): JSON Schema (Draft 2020-12 subset) for structured JSON output in general mode. Cannot be combined with segmentDefinitions.
• minSegmentDuration (number, optional): Minimum seconds per segment. Time-based metadata mode only.
• maxSegmentDuration (number, optional): Maximum seconds per segment. Time-based metadata mode only.

Rules:

• Provide exactly one of videoUrl, assetId, base64Video.
• prompt cannot be combined with analysisMode="time_based_metadata".
• segmentDefinitions requires analysisMode="time_based_metadata".
• jsonSchema and segmentDefinitions are mutually exclusive.
• analysisMode="time_based_metadata" requires modelName="pegasus1.5" (the default) — it is rejected on Pegasus 1.2.

Returns: taskId, status, modelName. Poll with get-analyse-task.

Uploading a video as an asset

Use create-asset with either a url or a local file. The same tool covers image uploads (used as entity reference images) and video uploads (used as assetId input for async-analyse-video). Wait until status == "ready" before passing the returned id to a downstream tool.

Analyse Tasks

get-analyse-task

Retrieve the status and result of an async analyse task. Poll every 5–10 seconds until status is ready or failed.

Parameters:

• taskId (string, required): The task ID returned by async-analyse-video.

Result: When status === "ready", the response includes result.data. In time-based metadata mode, result.data is a JSON-encoded string keyed by each segmentDefinition.id — parse it client-side.

list-analyse-tasks

List async analyse tasks in your account, newest first. Supports pagination and filtering.

Parameters:

• page (integer, optional), pageLimit (integer, optional, max 50).
• status (optional): queued | pending | processing | ready | failed.
• videoUrl (string, optional): filter by exact URL.
• assetId (string, optional): filter by asset ID.
• analysisMode ("time_based_metadata", optional).

delete-analyse-task

Delete an async analyse task. Tasks that are still processing cannot be deleted (returns a friendly message).

Parameters:

• taskId (string, required).

Video Embeddings

start-video-embeddings-task

Start creating video embeddings for a video. Provide either a remote URL or a local file path.

Parameters:

• videoUrl (string, optional): URL of the video.
• videoFilePath (string, optional): Absolute path to a local video file to upload.

get-video-embeddings-tasks

Get the status of video embeddings tasks.

Parameters:

• taskId (string, optional): ID of the specific task. If not provided, returns status of the latest 10 tasks.

Possible Statuses:

• processing: Task is being processed
• ready: Task completed successfully
• failed: Task failed

retrieve-video-embeddings

Retrieve video embeddings from a completed embedding task or an indexed video.

Parameters:

• taskId (string, optional): ID of the video embeddings task. If provided, indexId & videoId are ignored.
• indexId (string, optional): ID of the index containing the video. If not provided we will try to use the default index if it exists.
• videoId (string, optional): ID of the video to retrieve.

Assets

create-asset

Upload an image or video file as a TwelveLabs asset. The same tool covers both:

• Image assets → reference images for entities (see create-entity).
• Video assets → input to async-analyse-video via the assetId parameter (alternative to videoUrl/base64Video).

The asset's media type is inferred from the file content. Wait until status == "ready" before using the returned id.

Parameters:

• url (string, optional): Publicly accessible URL of the image or video file. Provide either this or file.
• file (string, optional): Absolute path to a local image or video file.

Returns: Asset ID, status, fileType (MIME type — image/* or video/*).

delete-asset

Delete an asset by ID. Rejected by default if any indexed video references the asset; pass force=true to delete anyway (entity associations are unlinked).

Parameters:

• assetId (string, required).
• force (boolean, optional): defaults to false.

Entity Collections (Marengo 3.0)

Entity collections let you find specific people or objects in indexed videos. The flow is: create a collection → upload reference images via create-asset → register an entity that references those image asset ids → search using <@entity_id> in your query.

create-entity-collection

Create a new entity collection. Group related entities (e.g. one collection per sports team or film cast).

Parameters:

• name (string, required): Name of the collection.

list-entity-collections

List all entity collections.

delete-entity-collection

Delete an entity collection and all its entities.

Parameters:

• collectionId (string, required).

create-entity

Create an entity within a collection. Each entity references one or more image asset ids.

Parameters:

• collectionId (string, required): ID of the entity collection.
• name (string, required): Name of the entity (e.g. a person's name).
• assetIds (array of strings, required): Asset ids of reference images created via create-asset. Multiple images improve accuracy across angles/lighting.

Returns: Entity ID and a search-format hint. Use <@entity_id> action description in search to find the entity in indexed videos.

list-entities

List all entities in a collection.

Parameters:

• collectionId (string, required).

delete-entity

Delete an entity from a collection.

Parameters:

• collectionId (string, required).
• entityId (string, required).

Prerequisites

• Node.js 18+
• A TwelveLabs API key. For HTTP transport, pass it in the x-api-key header. For stdio transport, set the TWELVELABS_API_KEY environment variable.

Environment Variables

The following environment variables can be configured:

• TWELVELABS_API_KEY (required): Your TwelveLabs API key
• MCP_HTTP_PORT (optional): Port for the MCP server (default: 3000)

Installation

1. Clone the repository:

git clone <repository-url>
cd twelvelabs-mcp

2. Install dependencies:

npm install

Usage

Development

Start the development server with hot-reload:

npm run dev

The server will start on http://localhost:3000 and automatically restart when you make changes to the source code.

Production Build

Build the project for production:

npm run build

The compiled JavaScript will be output to the dist/ directory.

Running the Inspector

Use the MCP inspector tool to test your server:

npm run inspector

Using with Claude Code (stdio transport)

For CLI tools like Claude Code that use stdio transport, use twelvelabs-mcp-stdio:

npx twelvelabs-mcp-stdio

Configure in your MCP settings (e.g., .mcp.json):

{
  "mcpServers": {
    "twelvelabs-mcp": {
      "command": "npx",
      "args": ["twelvelabs-mcp-stdio"],
      "env": {
        "TWELVELABS_API_KEY": "your-api-key"
      }
    }
  }
}

API Endpoints

• POST /mcp - Main MCP communication endpoint
• GET /mcp - Returns "Method not allowed" (405)
• DELETE /mcp - Returns "Method not allowed" (405)