replicas-computer

The versioned computer-use runtime for Replicas workspace images.

• src/daemon.ts owns the serialized agent command boundary.
• src/commands/ owns desktop, browser, and recording operations.
• src/media-server.ts owns the shared H.264 and Opus encoders, the viewer control plane, and the quality controller.
• src/encoder.ts supervises each encoder process and keeps the active encoder running during replacement.
• src/video-encoder.ts owns video backend selection, arguments, fallback, and capture metrics.
• src/webrtc.ts owns one WebRTC peer per viewer (node-datachannel / libdatachannel) and parses receiver feedback.
• src/pacer.ts owns the per-viewer send budget: fresh media, NACK repairs and protective copies share one token bucket.
• src/rtp.ts owns RTP continuity, keyframe detection, and frame reassembly for the socket fallback.
• src/stream-viewer.ts owns the browser side: WebRTC playback, input routing, the WebSocket fallback, diagnostics and reconnects.
• src/input-bridge.ts converts takeover input into desktop events, accumulating small trackpad movements while preserving full mouse-wheel steps.
• src/viewer.ts owns the optional noVNC fallback.
• assets/desktop/start-services.sh owns the X11 and stream process lifecycle.
• xsettingsd broadcasts theme changes to running GTK apps and Chrome. Chrome follows the dashboard theme without reopening tabs; launch it without a GTK_THEME override.
• assets/desktop/ owns the desktop theme: wallpapers/ (a light and dark JPEG per scene; each new desktop picks a different scene), tint2rc dock, openbox-theme/, gtk-settings.ini, the replicas-terminal wrapper, themes/ palettes that start-services.sh renders into those templates (--theme light|dark switches a running desktop), and the bundled fonts (static TTF instances of General Sans and Geist Mono; variable and WOFF2 builds render with wrong metrics in Pango and VTE).

The viewer and agent command paths are independent. start-services.sh runs Xvfb, the compositor, PulseAudio and the stream server (and so the encoders it spawns) at nice −10 when sudo is available, so CPU load from the apps on the desktop slows those apps instead of stalling the picture of them. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page. A viewer out of sight (the dashboard keeps one connected while the Desktop tab is hidden) reports standby, detected with an IntersectionObserver because a hidden iframe keeps its last layout size; while every viewer is in standby the encoder runs at 5 fps on the 1.5 Mbps rung, and the first viewer to become visible returns it to full rate in place.

Chrome disables GPU compositing when no /dev/dri/renderD* device is available. This avoids Mesa's CPU-rendered compositor overhead before capture while retaining WebGL support. Desktops with a hardware render device keep Chrome's normal compositing path.

How the stream works

The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. It reads control lines on stdin between frames: keyframe encodes the next frame as an IDR at once, quality <crf> <kbps> changes the rate control through x264_encoder_reconfig, and fps <n> changes the capture rate, so none of them restarts the process. x264 keeps the frame rate it opened with and sizes its rate cap by it, so the encoder always opens at the stream's full rate and --capture-fps sets the rate capture starts at. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (REPLICAS_DESKTOP_STREAM_COLOR=legacy restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.

WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose ?transport=websocket; REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to /stream is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises inputRace, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video (at least three after the connection completes, so a slow network is not abandoned just as it connects) switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser. A viewer that fell back after a WebRTC failure keeps trying WebRTC in the background, 15 seconds after the fallback and then at doubling intervals up to five minutes: the server opens a new peer while the socket keeps carrying media, the new peer plays into an unseen element without offering 4:4:4, and once it decodes video the viewer moves its picture, audio and input to it and tells the server to stop the socket media (retry-webrtc, then webrtc-ready). The dashboard records each return as desktop_stream_webrtc_recovered with the time spent on the socket.

The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: balanced (CRF 14, default chroma quantisation) and text (CRF 10, chroma QP offset −6); ?profile= picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a bitrate ladder down to 750 kbps that keeps resolution and frame rate and adds two CRF per rung under the default 20 Mbps cap; ?adapt=0 pins the top rung.

Every video byte for a WebRTC viewer passes through PacedSender. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. Loss, RTT growth and jitter count only while the encoder has sent over 500 kbps for three seconds: on a quiet screen they belong to the network, and no rung fixes them. Climbing back to the rung below the one a descent left takes a few clean seconds; retrying that rung waits 15, 30 and then 60 seconds as episodes repeat, and an episode is forgotten after three minutes. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss, a starved decoder or a cut send queue) steps, and only once it has stopped climbing below what the stream sends. The browser's available-bitrate stat is the same estimate and is reported but never used. scripts/probe-viewer.ts drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; scripts/probe-rtt-jitter.ts feeds the controller a synthetic WebRTC viewer with jittery RTT probes.

The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame and skips rapid catch-up captures after CPU stalls. Native capture sends no periodic keyframes or refresh wave: an IDR follows within a frame when a viewer joins, a receiver sends PLI, a send queue is cut, the viewer's watchdog sees no video, or a socket viewer whose frames were withheld has drained its backlog or reset its decoder. Requests within a round trip of the last keyframe share it. Browsers join only at an IDR, so the SPS that intra refresh puts before a recovery point is not treated as a keyframe. Rung and standby changes are applied in place; a change of H.264 profile, chroma or preset starts a replacement, which discards two initial frames, emits a forced keyframe and then takes over. Until then the outgoing encoder takes the new rate, and keyframe requests wait for the replacement's. The FFmpeg fallback cannot be told to make a keyframe, so it keeps periodic intra refresh for WebRTC viewers, switches to 250 ms IDR intervals for socket viewers, and makes requested keyframes by replacement. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.

The viewer exposes ?view=fit|native|pixel (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and ?diagnostics=1, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, the server's encoder settings, keyframe interval, CPU, pacer budget, queue, stale cuts, repairs and copies, the socket fallback's unsent bytes, withheld delta frames and decoder resets, and the latest server and viewer events. globalThis.__replicasDesktop carries the same state for tooling.

WebRTC receiver timings use one-second counter differences: decodeMs is decode time per decoded frame; receiverProcessingMs runs from the first received packet to the decoded frame and already includes buffering and decoding. Neither measures physical display latency. Unsupported counters, counter resets and windows without decoded frames leave these fields absent. The timings appear in diagnostics, __replicasDesktop and per-viewer telemetry. See the WebRTC stats definitions.

The server records every decision that changes the picture (viewer connects and transport fallbacks with their reason, WebRTC state changes, quality rung moves with the signals that voted for them, encoder swaps, preset changes and keyframe requests) and writes each to the stream log (/tmp/replicas-computer/stream.log). GET /telemetry returns the /health snapshot plus the last 200 events and five minutes of one-second samples (encoder rung, CRF, cap, keyframe interval, preset, fps and kbps; CPU; per-viewer transport, ICE path, pacer counters, socket backlog and the viewer's own reported stats). Fetch it inside the sandbox with curl -s localhost:6080/telemetry or through the authenticated preview URL; the dashboard's stream settings menu has Open stream telemetry. GET /snapshot returns one 768px JPEG of the current screen; the chat's computer-use tile loads it as a preview of the desktop. REPLICAS_DESKTOP_STREAM_TELEMETRY=1 also writes every sample to the stream log for sessions longer than the in-memory history.

| Variable | Default | Meaning | |---|---|---| | REPLICAS_DESKTOP_STREAM_TRANSPORT | webrtc | Transport for viewers whose URL does not choose one | | REPLICAS_DESKTOP_STREAM_PROFILE | balanced | Encoder profile for viewers whose URL does not choose one | | REPLICAS_DESKTOP_STREAM_ADAPTIVE | 1 | 0 pins the top quality rung by default | | REPLICAS_DESKTOP_STREAM_ICE_SERVERS | stun:stun.l.google.com:19302 | Comma-separated STUN/TURN servers; turn:user:secret@host:port?transport=tcp adds a relay. Percent-encode the username and password; the browser receives their decoded values. Re-read from the workspace runtime env on every viewer, so the monolith's DESKTOP_STREAM_ICE_SERVERS reaches running workspaces. Short-lived credentials for the TURN relay are fetched from the monolith and appended; the server itself only uses UDP relays | | REPLICAS_DESKTOP_STREAM_ICE_PORTS | ephemeral | begin-end UDP port range for the server's ICE candidates | | REPLICAS_DESKTOP_STREAM_WEBRTC | 1 | 0 serves the WebSocket transport only | | REPLICAS_DESKTOP_STREAM_COLOR | bt709 | legacy disables the explicit BT.709 conversion and signalling | | REPLICAS_DESKTOP_STREAM_VIDEO_BITRATE | 20M | Encoder bitrate cap across all quality rungs | | REPLICAS_DESKTOP_STREAM_FPS | 60 | Maximum capture and encode rate | | REPLICAS_DESKTOP_STREAM_CAPTURE | native when installed | ffmpeg bypasses native capture | | REPLICAS_DESKTOP_STREAM_TELEMETRY | unset | 1 writes one-second telemetry samples to the stream log while a viewer is connected |

replicas-computer setup --check
sudo replicas-computer setup
replicas-computer daemon

setup --check reports missing OS dependencies, including the native WebRTC module, without changing the machine. setup installs package-owned assets and the replicas-computer.service systemd unit. Operational commands use the daemon over a user-only Unix socket.

The sandbox image builds node-datachannel from pinned libdatachannel source with scripts/build-webrtc.sh, which applies patches/libdatachannel-dtls-bio.patch (upstream DTLS BIO synchronisation fix, 2059d2a) and patches/libdatachannel-nack-duplicates.patch. It also applies patches/libjuice-ice-role-presence.patch so valid zero-valued ICE role attributes do not break Chrome TURN/TCP connectivity. It needs CMake, Ninja, a C++ compiler, Git and OpenSSL development libraries. Remove each patch when the pinned addon includes its fix.

The sandbox image builds the Go module in native/ with scripts/build-capture.sh against its installed FFmpeg and X11 libraries. On x86-64, frames use a pinned, statically linked libyuv build; its chroma-rounding patch keeps SIMD conversion consistent with the reference converter. Builds without libyuv use libswscale for full-frame color conversion at the same resolution. Local builds need Go (the script selects Go 1.27.1), C/C++ compilers, CMake, Git, pkg-config, and development packages for X11, Xext, Xdamage, Xfixes, libavcodec, libavformat, libavutil and libswscale. The module has no third-party Go packages. The capture loop and CLI are ordinary Go; native/internal/capture confines cgo calls, native buffers, the small C callback adapter, and explicit cleanup to one package. Its stream stays on one OS thread for Xlib. X11, libyuv, FFmpeg and x264 remain native dependencies. The build enables the libyuv tag on x86-64; a plain go build uses libswscale only. Run bun run build:capture after the TypeScript build; the binary's libyuv license is included in dist. /health exposes captureBackend and native captureMetrics: elapsed time, busy time, damage lookup and screen copy, color conversion, encode/RTP time, sent frames and changed frames for the latest interval. Timings are milliseconds of wall time; divide stage totals by changed frames (copy/conversion) or sent frames (encode) for per-frame cost. Receiver FPS is compared with the current one-second sending rate so quiet desktops do not trigger congestion downgrades.

Run bun run test:capture on x86-64 with Xvfb and xauth installed. The isolated X11 test compares all YUV planes against a fresh full-frame conversion over 1,024 captures, covering I420/I444, both color matrices, odd damage coordinates, right/bottom clipping, and repeated region-size changes. The Desktop capture CI workflow runs it with strict cgo pointer checks.