@ewjdev/ingest

Offline pipeline that turns curated YouTube debate videos into structured records in the debate library: persistent people, claims (with start/end timestamps), takes, transcript segments, and source provenance.

It is a pure producer: it downloads + transcribes + extracts locally, then POSTs a normalized payload to the web app's protected POST /api/ingest/debate route, which owns all D1 / Workers AI / Vectorize writes.

Quick start — transcribe a YouTube video

No clone, no build. You need yt-dlp on your PATH and a (free) Cloudflare API token; the tool walks you through the token on first run.

# 1. (optional) check your machine is ready
npx @ewjdev/ingest doctor

# 2. transcribe — on first run you're prompted for the Cloudflare token (with a link),
#    and it's saved to ~/.config/debate-ingest/secrets.env so you're never asked again.
npx @ewjdev/ingest transcribe "https://youtu.be/VIDEO_ID"
#   → prints the transcript AND writes ./<videoId>.transcript.txt

# pnpm / bun equivalents
pnpm dlx @ewjdev/ingest transcribe "https://youtu.be/VIDEO_ID"
bunx @ewjdev/ingest transcribe "https://youtu.be/VIDEO_ID"

debate-ingest setup enters/updates all keys up front (each shown with where to get it); debate-ingest setup --all also covers the credentials for the full ingest pipeline. Secrets live in ~/.config/debate-ingest/secrets.env (chmod 600); override the location with DEBATE_INGEST_CONFIG_DIR, or just set the matching env vars / a .env (those win).

Which secrets? Transcribing only needs CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID (Deepgram speech-to-text runs on Cloudflare Workers AI). A YouTube API key is not needed to transcribe — yt-dlp downloads the video directly. The YouTube key, Anthropic key, and ingest URL/secret are only for the discovery + claim-extraction + POST pipeline.

The npx/pnpm dlx/bunx commands above only resolve after the package is published (see Publishing below). Until then, run it from the repo: pnpm --filter @ewjdev/ingest transcribe "<url>".

Publishing (distribution)

The published package is a single bundled dist/cli.js (esbuild, plain JS) — so npx/pnpm dlx work on any Node ≥20 with no tsx and no build step on the consumer side. Only @anthropic-ai/sdk is installed at runtime; yt-dlp (and ffmpeg for clips) must already be on the runner's PATH.

One-time: publishing requires that the @ewjdev scope exists on npm and your npm account can publish to it. pnpm dlx @ewjdev/ingest resolves to whatever name is in package.json.

Publish via CI (recommended): add a repo secret NPM_TOKEN (npm Automation token with publish rights), then push a tag:

# bump packages/ingest/package.json "version" first, then:
git tag ingest-v0.1.0 && git push origin ingest-v0.1.0

.github/workflows/publish-ingest.yml builds and runs npm publish --access public. Run it manually first via the Actions tab (workflow_dispatch, dry-run on) to inspect the tarball.

Publish manually:

cd packages/ingest
npm publish --access public        # prepack builds dist/cli.js for you; needs `npm login`

Use it in GitHub Actions

The CLI is non-interactive on CI (it detects no TTY / CI=true), so pass secrets as env vars — it reads them and never prompts:

- uses: actions/setup-node@v4
  with: { node-version: 20 }
- run: pipx install yt-dlp            # or: sudo apt-get install -y yt-dlp
- run: pnpm dlx @ewjdev/ingest transcribe "$VIDEO_URL" --out transcript.txt
  env:
    VIDEO_URL: https://youtu.be/VIDEO_ID
    CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
    CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}

If a required secret is missing on CI the tool exits with a clear message naming the variable and where to get it (no hanging prompt).

Pipeline stages

| # | Stage | Tool | Artifact (.cache/youtube/{videoId}/) | |---|-------|------|----------------------------------------| | 1 | discovery | curated JSONL | — | | 2 | download | yt-dlp | audio.m4a, meta.json | | 3 | transcribe + diarize | Deepgram Nova-3 on Workers AI (REST) | transcript.json | | 4 | extract claims/takes | Claude (tool-use) | extraction.json | | 5 | resolve people | deterministic | people.json | | 6 | assemble payload | — | payload.json | | 7 | cut + upload clips | yt-dlp + ffmpeg → PUT /api/ingest/media | video.mp4, clips/*.{mp4,m4a}, clips.json | | 8 | post | POST /api/ingest/debate | post-result.json |

Each stage is cached; re-running resumes from the first missing artifact. --from-stage N forces a re-run from stage N. IDs/slugs are deterministic, and the route is idempotent on the source video_id, so re-posting is safe.

Clips (stage 7). Downloads a capped-resolution video.mp4 (≤480p, merged a/v) once, then cuts a 10–20s soundbite per claim — evenly sampled across the debate, capped by INGEST_MAX_CLIPS (default 24). Each claim yields a video clip (mp4, includes audio) and an audio clip (m4a), uploaded to R2 via the protected PUT /api/ingest/media route under historical/<platform>/<videoId>/claim-NNN.{mp4,m4a}. The R2 keys are written onto the payload claims so stage 8 persists them; the web app streams them at /api/media/<key> and plays them inline in the historical-debate view. Skip with --no-clips; cap with --max-clips N. Clip keys are deterministic, so re-runs overwrite.

Prerequisites

Only yt-dlp + ffmpeg/ffprobe on PATH (audio + video download, clip cutting). Transcription/diarization is a hosted call to Deepgram Nova-3 on Cloudflare Workers AI — no local ML stack.

brew install yt-dlp ffmpeg   # or your platform equivalent

Config (env / .env)

ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-sonnet-4-6          # optional
INGEST_URL=http://localhost:3000/api/ingest/debate   # plain http for local dev
INGEST_SECRET=<matches `wrangler secret put INGEST_SECRET` / .dev.vars>
CLOUDFLARE_API_TOKEN=<token with Workers AI run permission>
CLOUDFLARE_ACCOUNT_ID=927320d088a7c2ec3c070b43da0e85de  # optional (defaulted)
STT_MODEL=@cf/deepgram/nova-3               # optional
INGEST_MAX_CLAIMS=500                        # optional cap
INGEST_MAX_CLIPS=24                          # optional cap on clip soundbites
INGEST_MEDIA_URL=.../api/ingest/media        # optional; derived from INGEST_URL
INGEST_CACHE_DIR=.cache                      # optional

Create the Cloudflare token at dash.cloudflare.com → My Profile → API Tokens → Create Token → Workers AI (read). It only needs the Workers AI run permission.

Transcribe only (no DB write)

Just want the transcript of a YouTube video? The transcribe command runs only the download + speech-to-text steps and prints the transcript to stdout and writes it to a file. It needs only yt-dlp + CLOUDFLARE_API_TOKEN — no INGEST_URL/INGEST_SECRET or ANTHROPIC_API_KEY, and it posts nothing to the app.

# prints the transcript and writes ./<videoId>.transcript.txt
pnpm --filter @ewjdev/ingest transcribe "https://youtu.be/VIDEO_ID"

# just the first 6 minutes (recommended — Workers AI STT times out on long audio)
pnpm --filter @ewjdev/ingest transcribe "https://youtu.be/VIDEO_ID" --section 0:00-6:00

# write somewhere specific, as SRT subtitles or raw diarized JSON
pnpm --filter @ewjdev/ingest transcribe "<url>" --format srt  --out talk.srt
pnpm --filter @ewjdev/ingest transcribe "<url>" --format json --out talk.json

# stdout only (no file), or file only (quiet)
pnpm --filter @ewjdev/ingest transcribe "<url>" --no-file
pnpm --filter @ewjdev/ingest transcribe "<url>" --quiet --out talk.txt

Flags: --section <a-b>, --format txt|json|srt (default txt), --out <file> (default <videoId>.transcript.<ext>; - or --no-file skips the file), --quiet (no stdout), --force (re-download + re-transcribe, ignoring the cache). Progress lines go to stderr so stdout stays clean for piping.

Usage

# one video, stop before POST to inspect payload.json
pnpm --filter @ewjdev/ingest ingest run "https://youtu.be/VIDEO_ID" --dry-run

# full run
pnpm --filter @ewjdev/ingest ingest run "https://youtu.be/VIDEO_ID"

# batch from a curated list (one JSON object per line)
pnpm --filter @ewjdev/ingest ingest batch inputs/curated.jsonl

# consume the next approved source-library queue item
pnpm ingest:dev queue-next --section 0:00-10:00 --max-clips 8

Targets — local vs dev vs prod

Pick which app instance receives the POST (and clip uploads) with --target or the root shortcuts. Default (no target) uses INGEST_URL from .env.

pnpm ingest:local run "https://youtu.be/VIDEO_ID"   # → http://localhost:3000  (needs `pnpm dev` running)
pnpm ingest:dev   run "https://youtu.be/VIDEO_ID"   # → deployed dev worker
pnpm ingest:prod  run "https://youtu.be/VIDEO_ID"   # → PRODUCTION (writes real data)

# equivalent explicit flag (from packages/ingest):
pnpm ingest run "<url>" --target local

• The clip-upload endpoint (/api/ingest/media) is derived from the same host automatically.
• Secrets must match the target. INGEST_SECRET is sent as the bearer token; it must equal the target's secret — packages/web/.dev.vars for local, the wrangler secret for dev/prod. Use one shared value (simplest) or set per-target INGEST_SECRET_LOCAL / INGEST_SECRET_DEV / INGEST_SECRET_PROD in .env.
• Override hosts/port with INGEST_LOCAL_PORT (default 3000) or INGEST_{LOCAL,DEV,PROD}_URL.
• The queue endpoint (/api/ingest/queue) is derived from INGEST_URL; override with INGEST_QUEUE_URL only if routes are split.
• dev/prod require the routes deployed (pnpm deploy:dev / pnpm deploy) and the migrations applied to that D1.

inputs/curated.jsonl lines:

{"url":"https://youtu.be/VIDEO_ID","hints":{"topic":"...","affirmativeName":"...","negativeName":"..."}}

Curated ingestion workbench

The preferred discovery flow is now human-in-the-loop:

1. Open /admin/ingest with a WorkOS user listed in ADMIN_EMAILS or ADMIN_USER_IDS.
2. Search YouTube videos/channels using the server-side YouTube Data API integration.
3. Save/tag candidates with quality tags such as formal-debate, high-confidence, good-audio, or reject-slop.
4. Approve good videos and queue them with a bounded section.
5. Run pnpm ingest:dev queue-next to lease one queued item, execute the existing pipeline, and update queue/source status.

Required web env:

YOUTUBE_API_KEY=...
[email protected]               # comma-separated
# or
ADMIN_USER_IDS=user_...                    # comma-separated WorkOS user ids

Queue statuses are stored in D1 (ingest_queue) and linked back to youtube_videos. Successful queue ingests also preserve the selection context in debate_source_context, including selected tags, notes, quality score, and queue id. YouTube search results are persisted before filtering so the workbench can compare views, channel age, duration, captions, and curation history without re-spending quota on every filter change.

Automated ingestion (auto)

The auto command runs discovery + ingest unattended, on a GitHub Actions cron (.github/workflows/ingest-cron.yml, every 3h). Each run:

1. Picks a category — the --category flag / DISCOVER_CATEGORY env, else the rotation slot derived from the current 3h window (rotation in inputs/allowlist.json).
2. GET /api/ingest/known for the dedup set + daily-cap guard (exits if ingestedToday >= dailyCap).
3. Claude proposes --candidates N (default 5) two-sided debates for the category.
4. Each proposal is resolved via YouTube search.list, then filtered to the channel allowlist, duration window, embeddability, and the dedup set.
5. The first match runs the normal pipeline on a bounded --section (default 0:00-10:00) and POSTs to /api/ingest/debate with the category hint. One debate per slot; outcomes are written to the ingest_candidates ledger via POST /api/ingest/candidates.

# dry-run discovery against dev (resolves a video, writes payload.json, POSTs nothing):
pnpm ingest:dev auto --category politics --candidates 3 --section 0:00-5:00 --dry-run

# one real slot against dev:
pnpm ingest:dev auto

Allowlist (inputs/allowlist.json)

{
  "rotation": ["politics", "politics", "society", "business"],
  "channels": [
    { "channelId": "UC...", "name": "Open to Debate", "categories": ["politics", "business"] }
  ]
}

rotation weights category selection (repeat a category to weight it). A video is only ingested if its channelId is listed and the channel covers the run's category. No open crawling.

Required secrets (GitHub repo / environment)

| Secret | Used for | | --- | --- | | YOUTUBE_API_KEY | YouTube Data API v3 (search.list + videos.list) | | ANTHROPIC_API_KEY | Claude proposals + extraction | | INGEST_SECRET | Bearer auth for /api/ingest/* (must match the target deployment) | | CLOUDFLARE_API_TOKEN | Workers AI STT (transcription) + R2 clip uploads |

YouTube quota: the Data API allows 10,000 units/day; search.list costs 100 units. At 8 slots × 5 candidates × 1 search that's ~4,000 units/day. Lower --candidates / INGEST_DISCOVER_CANDIDATES if you approach the cap. Sectioned ingest (5–10 min) also bounds Deepgram/Claude cost and avoids the Workers AI STT long-audio timeout.

Scope & legal

• Two-sided debates only. Jobs with !=2 confidently-mapped speakers are flagged needs-review and skipped (see review/needs-review.json).
• Curated input list only — no mass crawling. Audio + full transcript stay local (gitignored); only derived data (timestamps, short attributed quotes, summaries) is persisted. Provenance (url, channel, license) is stored for takedown handling. Review copyright/ToS before any public use.