dsh-youtube-plugin
v0.1.0
Published
Deep, token-efficient YouTube understanding for DSH: captions or local GPU speech-to-text, scene keyframes with local OCR/VLM, transcript search/map, one disk cache, one loopback diagnostic route.
Downloads
199
Maintainers
Readme
dsh-youtube-plugin
A DSH bundle that lets an agent understand a YouTube video deeply and cheaply: one cheap probe decides what is worth paying for, captions or a local GPU speech-to-text run produce a timestamped transcript, scene keyframes with local OCR/VLM cover what is on screen, and every heavy artifact stays on disk with a hard token budget on every tool.
The plugin never calls a paid LLM API. Speech-to-text, OCR and the optional VLM run on this machine (CUDA when available, CPU otherwise). API tokens are spent only on the compact slice the model actually reads.
- Host half: seven
youtube_*tools plus one loopback-only diagnostic route. - Engine:
engine/(URL, yt-dlp, captions, STT, cache, segments, search, map, budgets). - Media:
media/(scene keyframes, storyboards, dHash dedupe, OCR, optional VLM). - Python sidecars:
python/stt/(faster-whisper) andpython/vision/(easyocr, Qwen2.5-VL). - Skill:
skill/youtube-research/SKILL.mdteaches the coarse-to-fine workflow.
Tools and budgets
Every tool is registered globally on ctx.tools, so a single agent and every
AgentTeams member see them (unless an owner tool policy hides them).
| Tool | What it returns | Default budget |
|---|---|---|
| youtube_probe | title, channel, duration, chapters, caption languages, cache state, recommended next tool | <= 300 tokens |
| youtube_build | ensures transcript + map; compact per-step status (source, segments, cache hit) | <= 400 tokens |
| youtube_map | chronological sections with timestamps, key sentences, keywords | <= 1200 tokens |
| youtube_search | BM25 hits: timestamp + snippet + score, at most 10 | <= 600 tokens |
| youtube_read | one transcript window with timestamps; refuses an oversized window | <= 2500 tokens |
| youtube_visual | OCR text (+ VLM lines) for frames, or one frame/contact sheet | <= 400 tokens + <= 1 image |
| youtube_status | backends, GPU, models, cache, DNS, last error, Russian next step | <= 500 tokens |
Rule of thumb for a 60-minute talk: probe -> map -> 2-3 searches -> 1 read window is usually under 6k tokens and never puts the whole transcript in context.
Install (npm)
Requirements: Node 24 (provided by DSH), ffmpeg + ffprobe on PATH (winget install Gyan.FFmpeg), Python 3.12 on PATH for the optional tiers.
Add the bundle to a DSH profile:
dsh plugin --profile desktop add dsh-youtube-plugin
DSH installs the package from npm together with its @deepseek-ai/dsh-tools dependency, registers it in the profile bundles list, and composes the youtube row from cordis.patch.yml. Restart the desktop app once so the GUI process mounts it.
Bootstrap the machine-local backends (once, or after an update). The package is installed into the profile, so run the script from the installed package directory:
cd $env:USERPROFILE/.dsh/profiles/desktop/node_modules/dsh-youtube-plugin powershell -NoProfile -ExecutionPolicy Bypass -File scripts/install.ps1
Add -WithVision to install easyocr + Pillow and optionally the Qwen2.5-VL-3B model. The script creates .venv (yt-dlp), .venv-stt (faster-whisper + CUDA DLLs) and optionally .venv-vision inside the installed package, sets NO_PROXY=* for every Python install (the system SOCKS proxy breaks pip and Hugging Face), pins HF_HOME/HF_HUB_CACHE into .cache/huggingface, pins the EasyOCR model directory into .cache/easyocr, then runs scripts/doctor.mjs and prints what works.
A package update may replace the package directory and with it the venvs; re-run the script after dsh plugin updates. To keep large pieces elsewhere, override cacheDir, python or ytdlpPath in the youtube row config or with DSH_YOUTUBE_* environment variables.
Verify:
node scripts/check-bundle.mjs node scripts/doctor.mjs
Update:
dsh plugin --profile desktop add dsh-youtube-plugin@latest then re-run scripts/install.ps1 if the update changed the package layout.
Use it
Minimal agent loop (also what skill/youtube-research says):
youtube_probe url=...- always first. It says whether a transcript exists and which tool is worth calling next.- If no transcript:
youtube_build url=...once. It tries manual captions, then auto captions, then local STT. A second call is a cache hit with no download. youtube_map url=...for the whole-video outline (cheap, deterministic).youtube_search url=... query="..." limit=5for a topic; thenyoutube_read url=... from=12:00 to=13:30around the best hit.youtube_visual url=... at=1:00 mode=textfor slides/code/charts (requiresvision:trueand the vision venv), ormode=image attach:truefor one frame, ormode=contact at="0-600" n=9for a sheet. Frames work withoutvision:true; OCR/VLM do not.
Force local speech-to-text with youtube_build stt:"always" (for example when the
auto captions are garbage); forbid it with stt:"never" (captions only). Pick a
caption language with lang:"ru".
Diagnose anything with youtube_status. It reports backend availability, GPU, STT
model cache, OCR/VLM presence, cache size, the DNS state of www.youtube.com and the
last recorded error with a Russian next step. It never retries blindly: DNS,
403 and bot-check are reported, not looped.
Clean old artifacts: youtube_status clean:true maxAgeDays:30 (only directories the
cache index lists, only when their recorded age is known).
Configuration
Every key is optional and resolved at call time from the plugin config row (the
youtube row in cordis.patch.yml) and then from the environment; a bad value
falls back to its default, degrades only the tier that uses it, and is listed by
youtube_status and the route.
| Key | Env | Default | Meaning |
|---|---|---|---|
| cacheDir | DSH_YOUTUBE_DIR | $DSH_HOME/youtube | Root of every artifact (transcripts, maps, frames, index.json). |
| ytdlpPath | DSH_YOUTUBE_YTDLP | auto (.venv/Scripts/yt-dlp.exe, .venv-stt, PATH) | yt-dlp executable. |
| ffmpegPath | DSH_YOUTUBE_FFMPEG | auto (PATH) | ffmpeg; ffprobe is taken from the same directory. |
| python | DSH_YOUTUBE_PYTHON | auto (.venv-stt) | STT venv python. |
| preferredLangs | DSH_YOUTUBE_LANGS | en,ru | Caption language priority. |
| sttModel | DSH_YOUTUBE_STT_MODEL | large-v3-turbo | faster-whisper model. |
| sttDevice | DSH_YOUTUBE_STT_DEVICE | auto (auto/cuda/cpu) | STT device. |
| vision | DSH_YOUTUBE_VISION | false | Enable the OCR/VLM tier: youtube_visual mode=text needs it; mode=image/mode=contact do not. |
| vlmModel | DSH_YOUTUBE_VLM_MODEL | Qwen2.5-VL-3B-Instruct | Optional local VLM. |
| ocrBackend | DSH_YOUTUBE_OCR_BACKEND | easyocr (easyocr/rapidocr/auto) | OCR backend (easyocr reads Cyrillic). |
| maxFrames | DSH_YOUTUBE_MAX_FRAMES | 40 | Keyframe cap (1..200). |
| maxOutputTokens | DSH_YOUTUBE_MAX_TOKENS | unset | Global ceiling applied on top of the per-tool budgets. |
| cookiesPath | DSH_YOUTUBE_COOKIES | unset | Optional Netscape cookies file (never printed by the plugin). |
Example row override:
- id: youtube
name: dsh-youtube
config:
vision: true
maxFrames: 24
maxOutputTokens: 900
sttDevice: cudaImages (exact shape)
youtube_visual mode=image always returns a text pointer with the absolute JPEG path,
dimensions and size, so a text-only model can still hand the file to another tool.
With attach:true the tool also returns one image content block, using the same
convention as the built-in read_image tool:
{ type: 'image', attachment: { attachmentId: 'sha256:...', mediaType: 'image/jpeg', width: 640, height: 360, bytes: 88000, name: 'at-600_000.jpg' } }The image is stored through ctx.attachments.saveImage({ data, mediaType, name }),
so it becomes a durable attachment that survives a restart, and the model must declare
image input (otherwise DSH substitutes its own text placeholder). When no attachment
service is mounted, or the file is not PNG/JPEG/WebP, or saving fails, the tool
degrades to the file path plus an attachNote - it never fails the call.
Cache layout
<cacheDir>/<videoId>/meta.json, transcript/segments.jsonl,
transcript/transcript.txt, map.json, state.json, frames/ (keyframes,
ocr.jsonl, desc.jsonl, contact*.jpg), source/ (short-lived stream), and a root
index.json. Writes are atomic (temp + rename); a per-video lock prevents duplicate
downloads and stale locks (>30 min) are reported, never silently removed.
AgentTeams
The tools are registered globally, so a team member with the corresponding tool
policy sees them without extra wiring. The team integration - a video tool policy,
a video-research profile (collector/analyst/critic, captain planning) and the
researcher allow-list extension - ships as integration/agent-teams.patch.yml with
integration/apply.mjs and integration/verify-merge.mjs. It is applied to this
installation's desktop profile; docs/INSTALL.md records the exact
YAML, the additive verification and the rollback. The skill teaches the same
coarse-to-fine workflow to a single agent or to a whole team: collect a compact case
per video into research/<topic>/, never dump a transcript, and always cite
timestamps.
Troubleshooting
youtube_status is the entry point: it names the failing tier and the Russian next
step. The route GET /api/dsh-youtube?status=1 (loopback only) answers the same
diagnostics as JSON for scripts.
| Symptom | Diagnosis and fix |
|---|---|
| NETWORK_DNS / probe hangs before any request | www.youtube.com does not resolve on this machine. The VPN client must be up (this network resolves youtube.com only through it), or set DNS 1.1.1.1 in Windows or on the router; check with nslookup www.youtube.com 1.1.1.1. The plugin does not retry until DNS works. |
| BOT_CHECK / HTTP_403 | YouTube wants a signed-in session or blocks this IP: export cookies from a browser and set cookiesPath (DSH_YOUTUBE_COOKIES), update yt-dlp (youtube-plugin/.venv/Scripts/yt-dlp.exe -U). |
| Requested format is not available | The video does not expose the requested selector (some players offer only HLS). The plugin already uses ba/b selectors and the pinned protocol=https 360p stream; retry later or choose another video. |
| MISSING_YTDLP / MISSING_FFMPEG | Run scripts/install.ps1; ffmpeg must be on PATH or set in DSH_YOUTUBE_FFMPEG. |
| No captions (CAPTIONS_MISSING) | youtube_build falls back to local STT automatically. If STT is missing, run scripts/install.ps1 (it creates .venv-stt). |
| OCR/VLM missing | scripts/install.ps1 -WithVision; OCR needs .cache/easyocr models, the VLM needs .cache/models/Qwen2.5-VL-3B-Instruct (optional, ~7.5 GB). |
| pip / Hugging Face installs fail behind the proxy | Every Python install must run with NO_PROXY=*; install.ps1 sets it for you. |
| dsh --profile desktop --dump-config is refused | Documented in SPIKE 8.7: the Electron-managed desktopprofile refuses--dump-config. Validate composition on a throwaway copy of the profile (dsh --profile --dump-config`) instead. |
Known limitations (measured, not guesses)
- ffmpeg cannot fetch remote URLs here. TLS verification to
googlevideofails on this ffmpeg build, so--download-sections/HLS-through-ffmpeg paths are out. The plugin downloads one small stream with yt-dlp's native downloader and runs ffmpeg only on local files. - ffmpeg cannot decode the storyboard WebP sheets on this build (SPIKE
8.5); storyboards are decoded with Pillow in.venv-vision` instead. dsh --profile desktop --dump-configis refused (SPIKE `8.7) - validate on a throwaway profile copy.- Age-gated videos are untested (SPIKE §8.10); treat them as unsupported until measured.
- One video per call; playlists, channels, live streams, translation and diarization are out of scope for v0.1.
- The machine-global Hugging Face cache and
~/.EasyOCRare not usable here, so every cache is pinned insideyoutube-plugin/.cache: one canonical HF cache (.cache/huggingface, where the faster-whisper weights live and where anyHF_HOME-driven download goes),.cache/easyocrfor OCR models and.cache/modelsfor the optional VLM.install.ps1, the STT/vision bootstraps, the engine config anddoctorall name.cache/huggingface;doctorverifies it and STT reportsmodel cached=truefrom there. The legacy spike directory.cache/hfwas removed. - PyAV >= 19 is incompatible with the pinned faster-whisper 1.2.1; do not add it to
.venv-stt(doctorwarns when it is present).
Layout
youtube-plugin/
index.js package.json cordis.patch.yml locale/{en,ru}.json icon.svg
lib/ config.js fence.js tool-defs.js text.js diagnostics.mjs maintenance.mjs
engine/ deterministic core (url, ytdlp, captions, stt, cache, segments, search, map, budgets)
media/ frames: scene detect, storyboard, dHash, OCR/VLM glue, contact sheets
python/ stt/ (faster-whisper runner), vision/ (easyocr/VLM/storyboard), bootstraps
scripts/ install.ps1, doctor.mjs, check-bundle.mjs
integration/ agent-teams.patch.yml, apply.mjs, verify-merge.mjs,
install-skill.mjs, install-host-shim.mjs, smoke-tools.mjs, backups/, evidence/
skill/ youtube-research/SKILL.md
test/ node --test suites (tools.test.mjs + engine/media suites)
docs/ DESIGN.md, SPIKE.md, INSTALL.md, REVIEW.mdRollback
Remove the bundle from the profile with dsh plugin --profile desktop remove dsh-youtube
and restart; the cache and the plugin-owned venvs are plain directories that can be
deleted independently. The profile-side changes this installation made (bundle row,
youtube config row, AgentTeams policy/profile, the host shim and the installed skill)
and their exact reversal - including the backup directory to restore from - are listed
in docs/INSTALL.md.
