Support this project

If this was helpful to you, please buy me a beer on PayPal: Click Here to Buy Me a Beer

m3u8-grabber

HLS archiver that downloads m3u8 playlists, video segments, encryption keys, and init segments to Google Cloud Storage. Preserves the original directory structure as-is.

Supported Formats

| Format | Extensions | Status | |--------|-----------|--------| | MPEG-TS | .ts | Supported | | Fragmented MP4 (fMP4) | .m4s, .m4v, .mp4, .cmfv, .cmfa | Supported | | CMAF | .m4s, .mp4 | Supported | | Init segments | #EXT-X-MAP URI | Supported | | Encryption keys | #EXT-X-KEY URI | Supported | | Master playlists | Multi-variant with #EXT-X-STREAM-INF | Supported | | Audio playlists | #EXT-X-MEDIA URI | Supported |

Install

npm install m3u8-grabber

Requires Node.js >= 18.

Usage

import { Storage } from '@google-cloud/storage';
import { downloadM3u8 } from 'm3u8-grabber';

const storage = new Storage({
  credentials: require('./google-credentials.json'),
});

await downloadM3u8(
  storage,
  'https://example.com/hls/master.m3u8',
  'my-gcs-bucket',
  'output',          // local temp dir for m3u8 files
  'user-1234',       // GCS path prefix
);

With progress callbacks

await downloadM3u8(
  storage,
  'https://example.com/hls/master.m3u8',
  'my-gcs-bucket',
  'output',
  'user-1234',
  null,              // baseUrl (auto-detected)
  {
    onSegmentComplete: (file, completed, total) => {
      console.log(`[${completed}/${total}] ${file}`);
    },
    onSegmentError: (file, error) => {
      console.error(`Failed: ${file}`, error.message);
    },
  },
);

How It Works

1. Downloads the m3u8 manifest file
2. Parses it using a state-machine parser (not regex) to extract:
   • Child m3u8 playlists (variant streams, audio tracks)
   • Media segments (.ts, .m4s, .m4v, .mp4, etc.)
   • Init segments (#EXT-X-MAP URIs)
   • Encryption keys (#EXT-X-KEY URIs)
3. Recursively processes child playlists
4. Streams all segments directly to GCS (no local buffering of large files)
5. Size-aware skip: existing GCS files are only skipped if their size matches the expected Content-Length — zero-byte or partially-uploaded files are re-downloaded

Exported Functions

| Function | Description | |----------|-------------| | downloadM3u8(storage, url, bucket, localDir, gcsDir, baseUrl?, options?) | Download a full HLS stream to GCS | | streamDownloadToGcp(bucket, url, output, storage?, credsPath?, skip?) | Stream a single file to GCS | | uploadLocalFileToGCS(storage, bucket, src, dest, skip?, retry?) | Upload a local file to GCS with retry | | downloadFile(url, outputPath) | Download a file to local disk | | extractTsUrls(m3u8Content) | Extract segment URLs (.ts + fMP4) from manifest | | extractM3u8Urls(m3u8Content) | Extract child playlist URLs from manifest | | extractKeyUrls(m3u8Content) | Extract encryption key URLs from #EXT-X-KEY tags | | isValidManifestPath(url) | Check if a string is a valid file path (not a tag line) | | extractBaseUrl(url) | Get the base URL (everything before the filename) | | extractFilename(url) | Get the filename from a URL (strips query params) | | extractParentFolder(url, offset?) | Get a parent directory name from a URL | | createStorageClient(credsPath) | Create a GCS Storage client from a credentials file |

Retry & Resilience

• HTTP downloads: 5 retries with linear backoff (axios-retry)
• GCS uploads: 5 retries with size verification after each attempt
• GCS client: IdempotencyStrategy.RetryAlways, max 10 retries, 500s total timeout
• Resume support: size-aware skip checks Content-Length header against GCS blob metadata — partial uploads are detected and re-downloaded

Running Tests

Local: Place google-creds.json at project root (GCP service account JSON key).

CI/CD: Set GOOGLE_SERVICE_ACCOUNT_CREDS (JSON or base64-encoded JSON) and NODE_ENV=test as pipeline secrets.

npm test              # watch mode
npm run test:no-reloading  # single run

Unit tests (parsing) run without GCS credentials. Integration tests (download + upload) require valid credentials and create/delete a temporary GCS bucket.

Support this project

If this was helpful to you, please buy me a beer on PayPal: Click Here to Buy Me a Beer