@trillboards/edge-platform-rtsp

Platform-neutral RTSP / IP-camera frame provider for the Trillboards Edge AI SDK.

It implements the CameraProvider interface from @trillboards/edge-core using ffmpeg and Node's child_process. Because it depends on nothing OS-specific, it works on Linux, macOS, Windows, and Docker — and it is outbound-only, so it traverses NAT to reach cameras / NVRs behind a venue router.

This is the package to use whenever a camera is reachable over RTSP, regardless of the host OS. (V4L2 / DirectShow remain in @trillboards/edge-platform-linux and @trillboards/edge-platform-windows for locally-attached USB/CSI cameras.)

It also provides the two ffmpeg-based audio providers — RTSPAudioProvider (continuous audio off a camera/NVR channel) and MacAudioProvider (the macOS local microphone) — both implementing AudioProvider from @trillboards/edge-core.

Install

npm install @trillboards/edge-platform-rtsp
# ffmpeg must be on PATH (or pass ffmpegPath)

Usage

import { RTSPCameraProvider } from '@trillboards/edge-platform-rtsp';

const camera = new RTSPCameraProvider({
  rtspUrl: 'rtsp://user:[email protected]:554/Streaming/Channels/101',
  width: 640,
  height: 480,
  fps: 8,
  transport: 'tcp', // default; 'udp' for lower latency on a reliable LAN
});

await camera.open();
const rgb24Frame = await camera.getFrame(); // raw RGB24 Buffer
await camera.close();

NVR / multi-channel

An NVR typically exposes one RTSP URL per channel (…/Channels/101, …/Channels/201, …). Create one RTSPCameraProvider per channel — one feed = one screen = one sensing profile.

RTSPStreamCameraProvider — a look that costs nothing

RTSPCameraProvider spawns one ffmpeg per grab (-frames:v 1), so every frame pays a process start, a full RTSP DESCRIBE/SETUP/PLAY and a wait for the next keyframe. Measured against mediamtx re-streaming a real CCTV clip at a degraded 640x360 / 399 kbps / GOP 2 s profile, that is a p50 of 2,000 ms per frame — and the same ffmpeg args against a local file complete in 20-30 ms, so ~99% of it is session setup rather than imaging.

RTSPStreamCameraProvider holds one long-lived ffmpeg per feed, decoding continuously into a bounded in-memory ring. The handshake is paid once, at open(), and getFrame() becomes a read of the newest decoded frame — p50 0 ms, with frames arriving every 1000 / streamFps ms.

import { RTSPStreamCameraProvider } from '@trillboards/edge-platform-rtsp';

const camera = new RTSPStreamCameraProvider({
  rtspUrl: 'rtsp://user:[email protected]:554/Streaming/Channels/101',
  width: 640,
  height: 480,
  streamFps: 4,        // decoded+encoded frames/s; the freshest a look can be
  transport: 'tcp',
});

await camera.open();
const jpeg = await camera.getFrame();   // newest UNSERVED frame, from memory
console.log(camera.getStats());          // mode, frameAgeMs, restarts, staleReads…
await camera.close();

Four things about it that are not obvious:

• JPEG only. A persistent rgb24 stream is 921,600 bytes/frame — 3.7 MB/s per camera at 4 fps. The same stream as MJPEG is ~0.12 MB/s. Use RTSPCameraProvider when you need raw RGB (the ONNX face path).
• getFrame() never repeats a frame. If no new one has arrived it waits up to frameWaitMs and then throws RtspStreamStaleError with the age. Handing back a repeat is how a frozen camera reports as healthy forever.
• It degrades, it does not fail. If the recorder's persistent-session lane is full (maxStreamSessionsPerSource, default 16 per host:port) or ffmpeg cannot hold the session, it falls back to RTSPCameraProvider and reports getStats().mode === 'per_grab' with a degradedReason.
• Memory is bounded on admission. A ring of ringMaxFrames / ringMaxBytes plus a separate maxFrameBytes ceiling on the partially-received frame, with every eviction counted in getStats().

Audio

RTSPAudioProvider — continuous audio off a camera / NVR channel

Frame capture is one-shot per frame. Audio is not: it is a long-lived ffmpeg session feeding a bounded ring buffer, because sampling a few seconds per cadence misses most of what audio exists to catch.

That difference is why audio has its own admission lane. Long-lived audio sessions are counted per physical recorder (host:port — credentials and channel path excluded, so 32 NVR channels share one budget) in a pool separate from video pulls, capped by maxAudioSessionsPerSource (default 1). Audio therefore can never consume a frame-capture slot, and a 32-channel NVR opens one continuous audio session rather than 32.

import { RTSPAudioProvider } from '@trillboards/edge-platform-rtsp';

const audio = new RTSPAudioProvider({
  rtspUrl: 'rtsp://user:[email protected]:554/Streaming/Channels/101',
  sampleRate: 16000,   // mono Float32 out
});

// Continuous stream (what the SDK's onset detector attaches to):
const off = audio.onPcm((pcm, atMs) => { /* … */ });

await audio.open();                       // may reject — see below
const recent = await audio.getAudioChunk(1000); // AudioProvider pull contract
off();
await audio.close();                      // frees the recorder's audio slot

open() rejects with a typed, non-fatal error in the two normal cases, and callers should treat both as "this feed has no audio" rather than as a fault:

| Error | code | Meaning | |---|---|---| | RtspAudioAdmissionError | RTSP_AUDIO_SESSION_LIMIT | This recorder already runs its allowed audio session(s). Never queues. | | RtspAudioNoTrackError | RTSP_AUDIO_NO_TRACK | The stream carries no audio track (most CCTV channels). The slot is released immediately so a camera on the same recorder that does have a microphone can use it. |

Any other failure (dropped TCP, recorder reboot) is transient and retried — a running session with capped exponential backoff and no permanent retirement, and an opening one for openValidationAttempts tries (default 4). Supervising only the running case would leave a recorder that is merely slow at boot costing that feed its audio for the life of the process. getStats() derives liveness on every call — there is deliberately no cached capability boolean.

| Option | Default | Notes | |---|---|---| | rtspUrl | (required) | Credentials never reach argv (written to a 0600 ffconcat file). | | sampleRate | 16000 | Output rate, mono. | | bufferSeconds | 30 | Ring depth; also litert-lm's max clip length. | | maxAudioSessionsPerSource | 1 | Long-lived audio sessions per recorder host:port. | | transport | 'tcp' | As the camera provider. | | openValidationAttempts | 4 | Transient misses tolerated while opening. A no-audio-track stream is never retried. | | restartDelayMs / maxRestartDelayMs | 1000 / 60000 | Backoff bounds for transient restarts. |

MacAudioProvider — macOS local microphone

Captures via ffmpeg's avfoundation input. It exists because loadAudio() on darwin previously fell through to the Linux PulseAudioProvider, whose require succeeds on macOS and only throws later inside open() — into a swallowed catch, so the Mac heard nothing and reported no fault.

macOS gates the microphone behind TCC, and that is reported as itself rather than as broken hardware:

| Error | code | |---|---| | MacAudioPermissionError | MAC_AUDIO_TCC_DENIED — grant Microphone access in System Settings | | MacAudioNoDeviceError | MAC_AUDIO_NO_DEVICE — no capture device at all |

import { MacAudioProvider } from '@trillboards/edge-platform-rtsp';

const mic = new MacAudioProvider();       // auto-selects the first audio device
await mic.open();
const pcm = await mic.getAudioChunk(1000);
await mic.close();

Config

| Option | Default | Notes | |---|---|---| | rtspUrl | (required) | Full RTSP URL incl. credentials. | | width / height | 640 / 480 | ffmpeg scales each frame. | | fps | 8 | Advisory; cadence is driven by the capture loop. | | transport | 'tcp' | 'tcp' (reliable) or 'udp' (low-latency). | | timeoutUs | 12000000 | ffmpeg RTSP socket timeout, microseconds. | | ffmpegPath | 'ffmpeg' | Point at a bundled ffmpeg-static binary if not on PATH. | | maxReconnectAttempts | 10 | Consecutive frame failures before the stream is marked closed. | | openValidationAttempts | 4 | Transient decode/no-frame/timeout misses tolerated while opening before startup fails. | | transientRetryDelayMs | 1000 | Delay between startup retries for transient opening misses. |

License

MIT