VE Video Recognize MCP

Async MCP server for video understanding via ve-backend proxy.

Environment

• API_URL: backend proxy url, default https://api.visionengine-tech.com/api/v1/video
• API_KEY: user API key from VisionEngine backend (required for submit/query and remote upload)
• MODEL: platform model id, default @preset/vec-1-0-video-recognize
• WORKDIR: local workspace root
• FILE_MODE: local file handling mode, local or remote, default remote
• REMOTION_WORK_DIR: shared mount root used in local mode, default /vec
• BASE_URL: backend public base url used for /save and /shared links, default https://api.visionengine-tech.com
• Remote upload path is built-in in code: public/videos

Tools

• submit
• query

submit

Submit an async video understanding task and receive a taskId for later polling.

Supported task types:

• understand
• cut_effect_points
• emotion_analysis
• script_generate
• style_analyze

Input uses a single parameter:

• video: can be either a public video URL or a local file path

When video is a local file path:

• FILE_MODE=local: after validating the file is under REMOTION_WORK_DIR, MCP sends a path relative to REMOTION_WORK_DIR, and backend resolves it as local file input internally
• FILE_MODE=remote (default): upload local file to backend /save, then convert returned path to /shared/...?...download=true URL

First version always submits with stream=false and returns a stable task-oriented payload. Use query to retrieve final results.

Supported optional analysis range parameter:

• analysisRange.type: time or frame
• analysisRange.startSec / analysisRange.endSec: select a time range in seconds
• analysisRange.startFrame / analysisRange.endFrame: select a frame range

Optional source timeline mapping parameter:

• sourceTimeRange.startSec / sourceTimeRange.endSec: declare where the submitted clip sits on the original full-length video timeline

Use sourceTimeRange when the submitted file is already a trimmed segment from a larger source video and you want backend to align all returned timestamps back to the original video timeline.

Rules:

• type=time only allows startSec / endSec
• type=frame only allows startFrame / endFrame
• at least one boundary is required
• single-sided ranges are supported, for example { type: "time", startSec: 30 }

Example submit parameters:

{
  "video": "https://example.com/demo.mp4",
  "sourceTimeRange": {
    "startSec": 30,
    "endSec": 45
  },
  "analysisRange": {
    "type": "time",
    "startSec": 5,
    "endSec": 20
  },
  "taskType": "understand",
  "responseFormat": "json_object"
}

In the example above, the uploaded clip itself corresponds to 30s ~ 45s of the original source video, while analysisRange further limits analysis to 5s ~ 20s inside the submitted video. Backend will align final returned timestamps to the original video timeline.

query

Query a submitted task by taskId.

• If the task is still running, the tool returns the current status and asks the caller to try again later.
• If the task succeeds or partially succeeds, the tool automatically fetches /task/{taskId}/result and returns the final structured result.
• If the task failed or was canceled, the tool returns the status and backend message/error.

Typical flow:

1. Call submit
2. Wait a short time
3. Call query with the returned taskId
4. Repeat query until the task finishes

Example MCP config

{
  "mcpServers": {
    "ve-video-recognize": {
      "command": "npx",
      "args": ["-y", "@visionengine/video-recognize@latest"],
      "transport": "stdio",
      "env": {
        "API_KEY": "<YOUR_API_KEY>",
        "WORKDIR": "./",
        "FILE_MODE": "remote",
        "REMOTION_WORK_DIR": "/vec"
      }
    }
  }
}