Zero-config

npx -y @sylphx/cue

Cue

Timeline proof for agents

Canonical package: @sylphx/cue · bin cue · public tools: read_video, video_evidence only.

Cue (canonical package @sylphx/cue) — local video timeline evidence, not frame-by-frame VLM.

Your agent watched the video. Did it read the timeline?

Evidence-first video reading for AI agents. One call turns any local video into a timeline document — ffprobe streams, chapters, embedded subtitles, scene boundaries, and warnings you can cite without frame-by-frame vision LLM calls.

Local-first · One smart read_video call · Timeline evidence + provenance · 20 tests

SOTA family roadmap: docs/roadmap/sota-family-roadmap.md.

⭐ Star this repo if agents should read video timelines with proof, not sampled frame captions. · Quick start · See it work · Why not frame-by-frame vision?

This repository is product SSOT. Sibling agent tools live in separate repos. Each instrument is an independent repository (marketplace + stars).

Read video structure (not N-second frame spam)

Cue is local-first timeline architecture: streams, dialogue, scene cuts, structural keyframes, and agent_index for text-only agents.

Spec: docs/specs/agent-video-read-contract.md

Local-first frontier: ffmpeg/ffprobe + structural keyframes, optional local whisper ASR. No cloud required.

Docs: Cue→Iris semantic timeline — structural keyframes become timestamped objects via Iris L2.

Product docs

| Doc | Purpose | | --- | --- | | docs/POSITIONING.md | Strategic positioning | | docs/COMPETITIVE.md | Peer anchors and wedge | | docs/EVIDENCE_CONTRACT.md | Evidence = result contract | | docs/TOOL_SURFACE.md | Few clear tools policy | | docs/PRODUCT_INDEPENDENCE.md | This repo is SSOT | | docs/IPPB.md | Independent public product bar | | docs/PUBLISH.md | npm/git publish status |

The problem

Videos are not a pile of frames. They are codecs, chapters, embedded subtitles, scene cuts, variable frame rates, and timelines measured in milliseconds.

Most agent stacks sample frames and ask a vision LLM what it sees. Subtitles get skipped. Scene boundaries vanish. Duration and stream metadata never reach context. Citations become "around minute two, I think." Then the agent hallucinates — confidently.

Video Reader MCP is built for the moment your agent needs a citeable timeline, not a slideshow summary.

Why not frame-by-frame vision?

| Typical vision path | Video Reader MCP | | --- | --- | | Sample N frames into a vision model | ffprobe format + stream metadata in one call | | Paraphrased "what happens" | Embedded subtitle cues with start_ms, end_ms, and provenance | | Scene changes guessed from captions | Optional ffmpeg scene filter with timestamp evidence | | Missing audio / VFR silently ignored | Warnings for missing ffmpeg/ffprobe, VFR, missing audio, skipped ASR | | Cloud API by default | Local-first — ffprobe + ffmpeg on your machine | | Ship and pray | 20 tests on parsers, fixture corpus, doctor, release gate, and integration |

See it work

Install (30 seconds)

npm install -g @sylphx/cue
cue doctor
claude mcp add cue -- npx @sylphx/cue

Install once. Call once.

{
  "sources": [{ "path": "/absolute/path/to/demo.mp4" }],
  "include_subtitles": true,
  "include_scenes": true
}

read_video builds a timeline document per source — no per-frame vision LLM calls:

{
  "source": "/absolute/path/to/demo.mp4",
  "success": true,
  "data": {
    "provenance": {
      "source": "/absolute/path/to/demo.mp4",
      "tool": "read_video",
      "version": "0.1.0",
      "extracted_at": "2026-07-09T12:00:00.000Z"
    },
    "format": {
      "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
      "duration_ms": 125500
    },
    "streams": [
      { "index": 0, "codec_type": "video", "width": 1920, "height": 1080 },
      { "index": 1, "codec_type": "audio", "channels": 2, "sample_rate": 48000 }
    ],
    "chapters": [
      { "id": 0, "start_ms": 0, "end_ms": 60250, "title": "Intro" }
    ],
    "subtitles": [
      {
        "index": 0,
        "start_ms": 1200,
        "end_ms": 3400,
        "text": "Welcome to the demo.",
        "provenance": { "method": "ffmpeg_extract", "format": "srt" }
      }
    ],
    "scenes": [
      {
        "index": 0,
        "time_ms": 45200,
        "provenance": { "method": "ffmpeg_scene_filter", "threshold": 0.4 }
      }
    ],
    "warnings": []
  }
}

Abbreviated shape — optional local ASR transcript hooks skip gracefully when no adapter is wired.

Prerequisites

• Node.js >=22.13
• ffprobe (required) and ffmpeg (recommended for subtitles + scenes) on PATH

MCP Tool Surface

| Tool | Use it when the agent needs to... | | --- | --- | | read_video | Read one or more local videos and return ffprobe metadata, chapters, subtitles, scenes, and timeline warnings. |

Supported formats: MP4, M4V, MKV, MOV, WebM, and other formats ffprobe can inspect.

Quick Start

Claude Code

Claude Desktop

Add this to claude_desktop_config.json:

{
  "mcpServers": {
    "cue": {
      "command": "npx",
      "args": ["@sylphx/cue"]
    }
  }
}

Any MCP Client

npx @sylphx/cue

HTTP transport (optional)

MCP_TRANSPORT=http MCP_HTTP_PORT=8080 npx @sylphx/cue

Security model

• Local-first — read_video inspects local files; remote URLs are not fetched by default.
• ffprobe/ffmpeg boundary — probe and frame tools shell out to configured binaries on PATH; missing tools return explicit errors.
• Fixture corpus — CI validates parser and safety fixtures; corrupted inputs fail closed with structured diagnostics.
• Evidence envelope — timestamps, frame indices, and extraction routes are preserved so agents can verify claims.

Release proof

Claims are backed by CI benchmark:release-gate, fixture corpus checks, and the shipped-path matrix (Rust-default primary tools).

bun run benchmark:release-gate

Artifact: benchmark-artifacts/video_reader_release_gate.json — must report status: passed before release.

Development

git clone https://github.com/SylphxAI/video-reader-mcp.git
cd video-reader-mcp
bun install
bun run build
bun test
bun run doctor
bun run benchmark:release-gate

Useful checks:

bun run check
bun run typecheck
bun run benchmark:release-gate

Example read_video requests live in examples/. CI runs parser, fixture corpus, doctor, and release-gate checks; integration tests exercise ffmpeg when available on the runner.

Support

• Issues
• npm package
• Portfolio orchestration: smart-reader-mcp

Help this reach more builders

If frame-by-frame vision guesses have wasted your context, your citations, or your trust in agent output, you are exactly who this project is for.

⭐ Star the repo — it is the fastest way to help more agent builders find evidence-first video reading. Share it in your MCP client setup, team wiki, or agent stack README.

Discovery (in progress)

| Channel | Status | | --- | --- | | Glama MCP directory | Listed — claim server for full discoverability | | Official MCP Registry | Listed — io.github.SylphxAI/video-reader-mcp @ v0.1.0 | | TensorBlock MCP Index PR #1113 | Open — multimedia/document processing listing | | MCP servers community issue #4500 | Open — community server highlight | | mcp.so listing issue #3068 | Open — directory submission request | | mcpservers.org submit | Not listed yet — free web-form submission |

Know another MCP directory? Open an issue with the link.

License

MIT © SylphxAI