@observertc/observer-js
v1.0.0-beta.23
Published
Server-side Node.js library for processing ObserveRTC Samples
Readme
ObserverTC — @observertc/observer-js
In one line: feed it WebRTC
getStats()snapshots, and get back a live, queryable model of every call plus a single typed event stream to react to.
observer-js is a server-side Node.js library for monitoring WebRTC sessions. A WebRTC
application (typically an SFU or a signaling/stats backend) feeds it ClientSample objects —
periodic snapshots of each participant's RTCPeerConnection.getStats() output plus
application events — and observer-js maintains a live, in-memory model of every call,
participant, peer connection, and media stream, derives per-interval and cumulative metrics,
and emits a single, unified stream of typed events the application can react to.
What you can do with it:
- Monitor calls live — a queryable in-memory tree of every call, client, peer connection, track, codec, ICE candidate and data channel, each holding current and cumulative metrics.
- React on one event bus — subscribe once on the
Observer; every payload carries its full ancestry (call → client → peer connection → stat), so you never walk the tree to subscribe. - Get derived metrics for free — counter-reset-safe per-tick deltas, bitrates, jitter, RTT, fraction-lost, remote-RTP (RTCP) correlation, and TURN/TCP usage from the selected candidate pair.
- Correlate across an SFU — link a publisher's outbound track to every subscriber's inbound
track (
RemoteTrackResolver), and observe mediasoup routers/transports/producers/consumers on the server side. - Detect server-only problems — cross-client
Detectors raisecall-issues for conditions no single client can see (e.g. everyone in a call degrading at once). - Persist every sample — per-client sinks (JSONL file, in-memory, or your own) for archival, streaming, and offline replay.
- Drop it in safely — warn-don't-throw, a pluggable logger, dual ESM + CommonJS, and no media-stack dependency in the core.
Status:
1.0.0-beta. The API described here is current and intended to be implemented against directly. This document is written to be self-sufficient: an engineer (or an AI agent) should be able to integrate the library, or develop it further, from this file alone. A companion doc,docs/logging.md, covers logging integration in depth.
Packaging: server-side, Node.js ≥ 22, shipped as a dual ESM + CommonJS build — so it works whether your project uses
import(ESM) orrequire()(CommonJS). Everything — including the built-in file sink — is exported from the single@observertc/observer-jsentry.
For AI agents:
llms.txtis a curated map of these docs (it belongs at the root of the docs site);AGENTS.mdcovers build/test commands and the conventions for working in this repository.
Table of contents
- Installation
- Quick start
- Data flow
- Entity hierarchy
- Ingestion:
accept(), context & lifecycle - When things update
- The event bus ← the core of the API
- API reference
- Schema types (
ClientSample) - Detectors (server-side extension point)
- Call summaries
- Remote track resolution (mediasoup / SFU)
- Mediasoup router observation
- Sinks (per-client sample persistence)
- Injecting data into a client
- Logging
- Design notes
- Error-handling philosophy
- Development & extension guide
Installation
npm install @observertc/observer-js
# or
yarn add @observertc/observer-jsServer-side, Node.js ≥ 22, dual ESM + CommonJS. The package ships both module formats, so it works the same whether your project is ESM or CommonJS — your import line is unchanged either way:
import { Observer, ClientSample, createJsonlFileSinkFactory } from '@observertc/observer-js';In an ESM project this resolves to the .mjs build; in a CommonJS project (where TypeScript
compiles your import down to require()) it resolves to the .js build. Everything is exported
from the single @observertc/observer-js entry. Written in TypeScript; ships type declarations for
both formats (dist/index.d.ts for require, dist/index.d.mts for import). Runtime
dependencies: @bufbuild/protobuf, events, uuid. The library does not bundle a logger or
any transport — see Logging.
ClientSample and friends are re-exported from this package, and are also published as the
shared schema in @observertc/schemas; samples
produced on the client (e.g. by @observertc/client-monitor-js) conform to the same shape.
Quick start
import { Observer, ClientSample } from '@observertc/observer-js';
// 1. Create an observer.
const observer = new Observer({
// a call updates when any of its clients does, and the observer when any of its calls does —
// both default to true, so this line is only here to show the knob exists:
autoUpdateOnCallUpdate: true,
// optional auto-teardown:
closeCallIfEmptyForMs: 20_000,
closeClientIfIdleForMs: 60_000,
});
// 2. Subscribe on the single bus. Every payload is an object with the ancestry.
observer.on('call-added', ({ observedCall }) => {
console.log('new call', observedCall.callId);
});
observer.on('client-issue', ({ observedClient, issue }) => {
console.warn(`[${observedClient.clientId}] ${issue.type}`, issue.payload);
});
observer.on('peer-connection-updated', ({ observedClient, observedPeerConnection }) => {
console.log(observedClient.clientId, 'RTT(ms):', observedPeerConnection.currentRttInMs);
});
observer.on('sample-rejected', ({ reason, sample }) => {
console.warn('dropped a sample:', reason);
});
// 3. Feed samples. `context` (optional) is transient per-accept data, carried to the
// `*-updated` events this accept triggers (never written to appData).
function onClientStats(sample: ClientSample) {
observer.accept(sample, { studioVersion: '1.2.3' });
}
// 4. Tear down.
process.on('SIGINT', () => observer.close());Data flow
client getStats() ──► ClientSample ──► observer.accept(sample, ctx?)
│
┌────────────────────────────────┘
▼
get-or-create ObservedCall ──► get-or-create ObservedClient ──► client.accept(sample, ctx)
│
per peerConnections[] in the sample
▼
get-or-create ObservedPeerConnection
.accept(pcSample, ctx) updates all sub-stats,
derives deltas/bitrates/RTT, correlates remote RTP
│
metrics roll up: PeerConnection → Client → Call → Observer
│
events emitted on the Observer bus ──► your handlers- A sample must have
callIdandclientId(the library sets them, or the app does). If either is missing, the sample is dropped andsample-rejectedis emitted. - Sub-entities that stop appearing in samples are garbage-collected via a "visited"
mark-and-sweep on each
ObservedPeerConnection.accept(), emitting the corresponding*-removedevents.
Entity hierarchy
| Class | Created by | Keyed on its parent as | Holds |
|-------|-----------|------------------------|-------|
| Observer | new Observer(config?) | — (root) | observedCalls: Map<string, ObservedCall>, global counters, the event bus |
| ObservedCall | observer.createObservedCall(settings) / lazily by accept | observedCalls | observedClients: Map<string, ObservedClient>, call-wide metrics, detectors, scoreCalculator |
| ObservedClient | call.createObservedClient(settings) / lazily | observedClients | observedPeerConnections: Map<string, ObservedPeerConnection>, per-client metrics |
| ObservedPeerConnection | lazily, from sample.peerConnections[] | observedPeerConnections | the 15 sub-stat maps below, transport/RTT/bitrate metrics |
| Sub-stats | lazily, from the PeerConnectionSample | maps on the PC | individual WebRTC stat objects |
ObservedPeerConnection sub-stat maps (all public readonly):
observedCertificates, observedCodecs, observedDataChannels,
observedIceCandidates, observedIceCandidatesPair, observedIceTransports,
observedInboundRtps, observedInboundTracks, observedMediaPlayouts,
observedMediaSources, observedOutboundRtps, observedOutboundTracks,
observedPeerConnectionTransports, observedRemoteInboundRtps, observedRemoteOutboundRtpsEach sub-stat class (ObservedInboundRtp, ObservedOutboundRtp, ObservedInboundTrack,
ObservedOutboundTrack, ObservedDataChannel, ObservedIceCandidate,
ObservedIceCandidatePair, ObservedIceTransport, ObservedCertificate, ObservedCodec,
ObservedMediaSource, ObservedMediaPlayout, ObservedPeerConnectionTransport,
ObservedRemoteInboundRtp, ObservedRemoteOutboundRtp) mirrors the corresponding stat
fields from the schema plus derived fields (deltas, bitrates).
Ingestion: accept(), context & lifecycle
observer.accept(sample, context?)
The single entry point. It:
- drops + emits
sample-rejectedif the observer is closed; - runs the sample through the global accept-middleware chain (see below);
- (chain terminal) drops + emits
sample-rejectedifcallId/clientIdis missing; - gets or lazily creates the
ObservedCallandObservedClient(theirappDatacomes from the configured factories, never fromcontext); - delegates to
client.accept(sample, context), which fans out to eachObservedPeerConnection.accept(pcSample, context).
Accept middlewares (global pre-dispatch hook)
observer.addAcceptMiddleware(...) registers middlewares run on every sample inside
accept(), in order, before the sample is dispatched to any call or client. Each middleware
gets a { sample, context } payload; it can inspect or mutate the sample (set/normalize
callId/clientId, enrich, redact) or the context, then call next(payload) to continue.
Not calling next drops the sample — nothing is created and no event fires. A throwing
middleware is caught and warns (the sample is dropped), never crashing accept().
import { Observer, AcceptMiddleware } from '@observertc/observer-js';
const observer = new Observer();
// derive callId/clientId from the app's own attachment, before dispatch
const route: AcceptMiddleware = ({ sample }, next) => {
sample.callId ??= sample.attachments?.roomId as string;
sample.clientId ??= sample.attachments?.peerId as string;
next({ sample });
};
// drop samples from a blocklisted client (never dispatched)
const filter: AcceptMiddleware = (payload, next) => {
if (blocked.has(payload.sample.clientId)) return; // no next() => dropped
next(payload);
};
observer.addAcceptMiddleware(route, filter);
// observer.removeAcceptMiddleware(route);This is a lightweight global injection point. When no middleware is registered, accept()
dispatches directly with no overhead.
context (the AcceptContext)
type AcceptContext = Record<string, unknown>;A single, optional, free-form object threaded down the whole accept chain
(Observer → Client → PeerConnection). It is transient request-scoped data — temporary or
contextual information the application wants available while an update is processed.
context is never written to appData and is not stored on any entity. The two are
deliberately distinct:
appData— application-assigned extra info that identifies/decorates an entity, fixed at creation (viasettings.appDataor thecreateCallAppData/createClientAppDatafactories), or assigned by the app on the*-addedevents. The library never changes it. The factories receive the context of theaccept()that triggered the creation, so a fact carried on the context can be baked intoappDataat birth — but it is copied by the factory, deliberately, not written across by the library.context— passed peraccept(), may differ on every call, and is carried straight through to the*-updatedevents that theaccept()triggers, then discarded.
client-updated and peer-connection-updated carry the exact context of that sample;
call-updated carries the context of the client accept() that drove the call update (absent
for interval- or teardown-driven call updates). When no context is given, the field is absent.
Get-or-create helpers
If you want to create/configure entities yourself before/without samples:
const call = observer.getOrCreateObservedCall({ callId, appData }); // ObservedCall | undefined
const client = call?.getOrCreateObservedClient({ clientId, appData }); // ObservedClient | undefinedThese return undefined (and warn) when the parent is closed; createObservedCall/
createObservedClient return the existing instance (and warn) if the id already exists.
Automatic teardown
closeClientIfIdleForMs— a client with no sample for this long auto-closes.closeCallIfEmptyForMs— a call with zero clients for this long auto-closes.- Closing cascades down (call → clients → peer connections → sub-stats), unsubscribing
listeners and emitting the
*-closed/*-removedevents.
When things update
"Update" means recompute aggregated metrics, run the detectors, and emit the *-updated event at
that level. Updates are event-driven — there is no built-in timer.
The rule is structural rather than configurable:
A call is updated when any of its clients is updated. The observer is updated when any of its calls is updated. Composed, that means the observer is updated exactly when any client anywhere is updated.
Two booleans, both defaulting to true, let you opt out of a link in that chain:
| Setting | Where | Effect when false |
|---------|-------|---------------------|
| autoUpdateOnClientUpdate | ObservedCallSettings | the call updates only when you call call.update() |
| autoUpdateOnCallUpdate | ObserverConfig | the observer updates only when you call observer.update() |
An app that wants a fixed cadence sets both to false and drives observer.update() from its own
setInterval. Note that observer-scoped detectors and validators run nowhere else — if the
observer never updates, they never run.
const observer = new Observer({ autoUpdateOnCallUpdate: false });
setInterval(() => observer.update(), 5_000);Earlier versions had an
updatePolicy/defaultCallUpdatePolicyenum ('update-on-any-…','update-when-all-…','none') and a pluggableUpdaterobject. Both are gone. "When all clients have updated" sounds appealing and deadlocks on the first client that stops sending — one silent participant froze the whole call's aggregation until it timed out.
The event bus
This is the primary API. Subscribe on the Observer instance — it is the single emitter
for the entire hierarchy. The ObservedCall / ObservedClient / ObservedPeerConnection
objects are themselves EventEmitters too, but those local events are reserved for internal
lifecycle/teardown wiring (see Local lifecycle events); application
code should use the Observer bus.
Payload shape: ancestry + subject
Every Observer event delivers exactly one argument: a payload object. The payload always contains the ancestry from the observer down to the entity that raised it, plus any event- specific subject:
type ObserverEventBase = { observer: Observer, context?: AcceptContext };
type ObservedCallScope = ObserverEventBase & { observedCall: ObservedCall };
type ObservedClientScope = ObservedCallScope & { observedClient: ObservedClient };
type ObservedPeerConnectionScope = ObservedClientScope & { observedPeerConnection: ObservedPeerConnection };So a peer-connection-level event hands you the observer, call, client, and peer connection:
observer.on('inbound-rtp-added', ({ observer, observedCall, observedClient, observedPeerConnection, observedInboundRtp }) => {
// all five are present and correctly typed
});observer.on/off/once/emit are fully typed against the event map — the handler argument is
inferred per event name.
Event catalogue
All payloads include the ancestry for their level (above). The Extra column lists the additional field(s) on top of that scope.
Observer level — scope { observer }
| Event | Extra payload | Fires when |
|-------|---------------|-----------|
| observer-updated | — | observer.update() ran (see When things update) |
| observer-closed | — | observer.close() |
| sample-rejected | { reason: 'observer-closed' \| 'missing-callId' \| 'missing-clientId', sample: ClientSample } | a sample was dropped by accept() |
| observer-issue | { issue: ObserverIssue } | observer.addIssue(...) — a cross-call / SFU-wide finding (see observer-level detectors) |
| validation-ready | { validator: string, report: ValidationReport } | a validator settled — fires once per check, not per tick |
Mediasoup level — scope { observer, observedMediasoupRouter }
| Event | Extra | Fires when |
|-------|-------|-----------|
| mediasoup-router-added | — | observer.createObservedMediasoupRouter(...) registered a router |
| mediasoup-router-matched-with-peer-connection | { observedCall, observedClient, observedPeerConnection } | a newly added peer connection's id matched one of the router's WebRTC transport ids. Opt-in via matchPeerConnectionByWebRtcTransportId: true. |
| mediasoup-router-removed | — | the underlying mediasoup router closed (its router.observer close fired) |
See Mediasoup router observation for the full design and examples.
Call level — scope { observer, observedCall }
| Event | Extra | Fires when |
|-------|-------|-----------|
| call-added | — | a call is created |
| call-updated | { context?: AcceptContext } | call.update() ran |
| call-closed | — | the call closed |
| call-empty | — | last client left the call |
| call-not-empty | — | first client joined a previously-empty call |
| call-issue | { issue: CallIssue } | call.addIssue(...) (server-side detector finding) |
| call-summary | { summary: CallSummary } | the call is closing and a summary was configured. Emitted inside close(), while the call is still reachable |
Client level — scope { observer, observedCall, observedClient }
| Event | Extra | Fires when |
|-------|-------|-----------|
| client-added | — | a client is created |
| client-sink-created | { sink: ClientSampleSink } | a per-client sink was created (only when createClientSink returns one); fires right after client-added |
| client-updated | { sample: ClientSample, elapsedTimeInMs: number, context?: AcceptContext } | the client processed a sample |
| client-closed | — | the client closed |
| client-joined | — | first CLIENT_JOINED event seen |
| client-left | — | CLIENT_LEFT seen (or inferred on close) |
| client-rejoined | { timestamp: number } | a later CLIENT_JOINED after an earlier join |
| client-issue | { issue: ClientIssue } | a client-reported issue arrived, or client.addIssue(...). A keyed issue also opens an entry in observedClient.activeIssues |
| client-issue-resolved | { resolvedIssue: ResolvedActiveClientIssue } | a stateful issue ended — the client sent its <type>-resolved companion, or the observer force-closed it. Carries the finished interval (durationInMs, resolvedBy) — see client issues |
| client-metadata | { metaData: ClientMetaData } | a client meta item arrived |
| client-extension-stats | { extensionStats: ExtensionStat } | an app-defined extension stat arrived |
| client-event | { event: ClientEvent } | any client event was processed |
Peer-connection level — scope { observer, observedCall, observedClient, observedPeerConnection }
| Event | Extra | Notes |
|-------|-------|-------|
| peer-connection-added / peer-connection-closed | — | lifecycle of the PC |
| peer-connection-updated | { context?: AcceptContext } | the PC processed a sample |
| ice-connection-state-changed / ice-gathering-state-changed / connection-state-changed | { state: string } | driven by client events |
| inbound-track-added / -updated / -removed / -muted / -unmuted | { observedInboundTrack } | |
| outbound-track-added / -updated / -removed / -muted / -unmuted | { observedOutboundTrack } | |
| inbound-rtp-added / -updated / -removed | { observedInboundRtp } | -updated fires every tick |
| outbound-rtp-added / -updated / -removed | { observedOutboundRtp } | -updated fires every tick |
| remote-inbound-rtp-added / -updated / -removed | { observedRemoteInboundRtp } | |
| remote-outbound-rtp-added / -updated / -removed | { observedRemoteOutboundRtp } | |
| data-channel-added / -updated / -removed | { observedDataChannel } | |
| ice-candidate-added / -updated / -removed | { observedIceCandidate } | |
| ice-candidate-pair-added / -updated / -removed | { observedIceCandidatePair } | |
| ice-transport-added / -updated / -removed | { observedIceTransport } | |
| codec-added / -updated / -removed | { observedCodec } | |
| media-source-added / -updated / -removed | { observedMediaSource } | |
| media-playout-added / -updated / -removed | { observedMediaPlayout } | |
| peer-connection-transport-added / -updated / -removed | { observedPeerConnectionTransport } | |
| certificate-added / -updated / -removed | { observedCertificate } | |
Volume note. The
*-updatedsub-stat events fire on every peer-connectionaccept()(i.e. per sample, per stream). For high-throughput servers, subscribe only to what you need, or read fields off the entities onclient-updated/call-updatedinstead.
Local lifecycle events
These remain on the individual entities (not the bus), for teardown/coordination. You can listen to them, but prefer the bus equivalents above for application logic.
| Entity | Local events |
|--------|--------------|
| ObservedCall | update, newclient, empty, not-empty, close |
| ObservedClient | update (sample, elapsedTimeInMs), close, joined, left |
| ObservedPeerConnection | removed-inbound-track, removed-outbound-track, close |
API reference
Observer
new Observer<AppData>(config?: ObserverConfig<AppData>)
type ObserverConfig<AppData = Record<string, unknown>> = {
// a call updates when any client does; the observer when any call does. Default true.
autoUpdateOnCallUpdate?: boolean;
appData?: AppData;
closeClientIfIdleForMs?: number;
closeCallIfEmptyForMs?: number;
// accumulate a per-call summary (see Call summaries). Absent or null = off, and nothing
// subscribes to anything. `{}` is valid: a summary with no built-in sections.
callSummary?: Partial<CallSummaryConfig> | null;
// appData factories — run when an entity is created without explicit appData
// (incl. lazily by accept()). appData is application-owned; accept `context` never touches it.
createCallAppData?: (p: { callId: string; observer: Observer; acceptCtx?: AcceptContext }) => Record<string, unknown>;
createClientAppData?: (p: { clientId: string; observedCall: ObservedCall; acceptCtx?: AcceptContext }) => Record<string, unknown>;
// sink factory — produces a per-client sink that receives every accepted sample (see Sinks).
createClientSink?: (p: { clientId: string; observedCall: ObservedCall }) => ClientSampleSink | undefined;
// remote-track-resolver factory — produces a call's RemoteTrackResolver (see Remote track resolution).
createRemoteTrackResolver?: (observedCall: ObservedCall) => RemoteTrackResolver | undefined;
};appData factories. Instead of pre-creating a call/client (or assigning on call-added /
client-added) just to enrich its appData, register a factory once. It runs whenever the entity
is created without an explicit settings.appData — including the lazy creation inside accept().
The client factory receives the already-created parent observedCall, so it can derive fields
from it.
Both also receive acceptCtx: the AcceptContext of the
accept() that caused the creation, or undefined when you created the entity yourself. This is
what lets an accept middleware resolve something
once — a tenant, a trace id — and have it land in appData at birth, instead of every factory
re-deriving it from the sample.
const observer = new Observer({
createCallAppData: ({ callId, acceptCtx }) => ({ callId, startedAt: Date.now(), tenant: acceptCtx?.tenant }),
createClientAppData: ({ clientId, observedCall }) => ({ clientId, tenant: observedCall.appData.tenant }),
});
observer.accept(sample, { tenant: 'acme' });appData stays application-owned: the context is offered to the factory, never written across by
the library, and it is still not stored on any entity.
Key members:
accept(sample: ClientSample, context?: AcceptContext): voidaddAcceptMiddleware(...mw: AcceptMiddleware[]): this/removeAcceptMiddleware(...mw): this— global pre-dispatch sample hooks (see Accept middlewares)getObservedCall<T>(callId): ObservedCall<T> | undefinedcreateObservedCall<T>(settings, acceptCtx?): ObservedCall<T> | undefinedgetOrCreateObservedCall<T>(settings, acceptCtx?): ObservedCall<T> | undefinedaddIssue(issue: Omit<ObserverIssue, 'scope'>): void— raise an observer-level finding → emitsobserver-issue.scopeis stamped for youupdate(): void— force an aggregation/observer-updatedtickaddObserverDetector(name, config?): this— build a cross-call detector ontoobserver.detectorsaddCallDetector(name, config?): this— register a call-scoped detector for every call created from now onremoveCallDetector(name, { includeOpenCalls? }): number— stop building it, and (by default) drop it from calls already open. Returns how many live instances were removedremoveObserverDetector(name): number— remove an observer-scoped detector. For one specific instance useobserver.detectors.remove(detector)addValidator(name, config?): this— start a one-shot structural checkcancelValidator(name | validator, reason?): number— stop a running check; it finishesinconclusivewith the reason and emitsvalidation-readyclose(): voidreadonly detectors: Detectors— observer-scoped registry. Starts empty; nothing is implicitreadonly callDetectorConfigs: Map<name, config>— whataddCallDetectorrecordedreadonly callSummaryCollector?: CallSummaryCollector— owns the resolvedconfig.callSummary, the summary subscriptions, and the summaries.undefinedwhen summaries are off, which is the only place that answer livesreadonly validators: Set<RunningValidator>— normally empty; each removes itself on finishingreadonly activeIssuesRegistry: ActiveIssuesRegistry— the fleet's open client issuesreadonly observedCalls: Map<string, ObservedCall>readonly observedTURN: ObservedTURNget appData(),get numberOfCalls()- counters:
numberOfClients,numberOfClientsUsingTurn,numberOfInboundRtpStreams,numberOfOutboundRtpStreams,numberOfDataChannels,numberOfPeerConnections,totalAddedCall,totalRemovedCall,closed on/off/once/emittyped against the event map
ObservedCall
type ObservedCallSettings<AppData = Record<string, unknown>> = {
// update this call whenever one of its clients accepts a sample. Default true.
autoUpdateOnClientUpdate?: boolean;
callId: string;
appData?: AppData;
closeCallIfEmptyForMs?: number;
};Key members:
readonly callId: string,appData: AppDatareadonly observedClients: Map<string, ObservedClient>,get numberOfClients()getObservedClient<T>(clientId),createObservedClient<T>(settings, acceptCtx?),getOrCreateObservedClient<T>(settings, acceptCtx?)(all… | undefined)addIssue(issue: Omit<CallIssue, 'scope'>): void— raise a call-level finding → emitscall-issue.scopeis stamped for youaddDetector(name, config?): this— build a call-scoped detector onto this call onlyremoveDetector(name): number— remove it from this call,close()ing it. For one specific instance usecall.detectors.remove(detector)readonly detectors: Detectors— server-side detector registry (empty by default; see Detectors)readonly activeIssuesRegistry: ActiveIssuesRegistry— this call's open client issues, propagating into the observer'sreadonly unconsumedOutboundTracks: Set<ObservedOutboundTrack>— maintained by the resolverscoreCalculator: ScoreCalculator,get score(),readonly calculatedScoreremoteTrackResolver?: RemoteTrackResolver— set fromObserverConfig.createRemoteTrackResolverat call creation (see Remote track resolution)- aggregates:
numberOfIssues,numberOfPeerConnections,numberOfInboundRtpStreams,numberOfOutboundRtpStreams,numberOfDataChannels,maxNumberOfClients,clientsUsedTurn: Set<string>,startedAt?,endedAt?,closedAt?,closed summary?: CallSummary— the live record of this call, when summaries are onupdate(),close()
ObservedClient
type ObservedClientSettings<AppData = Record<string, unknown>> = {
clientId: string;
appData?: AppData;
closeClientIfIdleForMs?: number;
};Key members:
readonly clientId: string,appData: AppData,readonly call: ObservedCallreadonly observedPeerConnections: Map<string, ObservedPeerConnection>readonly sink?: ClientSampleSink— the per-client sink (see Sinks), ifcreateClientSinkis configured; listen on it forclose/error- Injection API (queue app data to be merged into the next sample processing):
injectEvent(ClientEvent),injectIssue(ClientIssue),injectMetaData(ClientMetaData),injectExtensionStat(ExtensionStat),injectAttachment(attachments: Record<string, unknown>) - Direct add API (process immediately):
addIssue(ClientIssue),addMetadata(ClientMetaData),addExtensionStats(ExtensionStat) - Metrics (current/derived):
currentAvgRttInMs?,currentMinRttInMs?,currentMaxRttInMs?,receivingAudioBitrate,receivingVideoBitrate,sendingAudioBitrate,sendingVideoBitrate,usingTURN,usingTCP,availableIncomingBitrate,availableOutgoingBitrate - Counts:
numberOfInboundRtpStreams,numberOfOutboundRtpStreams,numberOfInbundTracks,numberOfOutboundTracks,numberOfDataChannels,numberOfPeerConnections - Per-tick deltas:
deltaReceivedAudioBytes,deltaSentAudioBytes, … (see source for the full set) - Lifecycle:
joinedAt?,leftAt?,closedAt?,closed,get score() - Metadata:
browser?,engine?,platform?,operationSystem?,mediaDevices,mediaConstraints accept(sample, context?),close()
ObservedPeerConnection
Key members:
readonly peerConnectionId: string,readonly client: ObservedClient,appData?- The 15
observed*sub-statMaps (listed above), plus array getters:codecs,inboundRtps,outboundRtps,remoteInboundRtps,remoteOutboundRtps,mediaSources,mediaPlayouts,dataChannels,peerConnectionTransports,iceTransports,iceCandidates,iceCandidatePairs,certificates,selectedIceCandidatePairs,selectedIceCandiadtePairForTurn - State:
connectionState?,iceConnectionState?,iceGatheringState?,usingTURN,usingTCP - Metrics:
currentRttInMs?,iceRttInMs?,rtcpRttInMs?,sfuHopRttInMs?,currentJitter?,availableIncomingBitrate,availableOutgoingBitrate, sending/receiving bitrates, packet rates, andtotal*/delta*byte/packet counters accept(pcSample, context?),close(),get score()
Two different round trips — don't mix them. iceRttInMs comes from ICE/STUN consent checks and
measures the trip to whatever terminates ICE: in an SFU topology that is the SFU, so it is the
client↔SFU leg. rtcpRttInMs comes from RTCP receiver reports and is an end-to-end media-path
round trip. They are not interchangeable, and averaging them together produces a number that moves
as streams come and go for reasons unrelated to the network. currentRttInMs therefore prefers
RTCP and falls back to ICE — always one kind within a tick, never a blend. sfuHopRttInMs
(rtcp − ice) estimates everything past the SFU, which separates "this client's last mile is slow"
from "the path beyond the SFU is slow".
Counter-reset boundaries. Chrome resets an SSRC's cumulative counters when the codec switches
(crbug/webrtc/5361, open since 2015),
which otherwise shows up as a sawtooth spike or a negative bitrate. ObservedInboundRtp /
ObservedOutboundRtp therefore set counterResetBoundary on any tick where codecId,
encoder/decoderImplementation or scalabilityMode changed, and suppress every delta for that
tick. Without this, a room-wide codec rollout fires a synchronized fake-degradation alert across
every participant at once.
Remote-RTP correlation (derived). During accept(), receiver/sender reports are linked
to the local streams by remoteId (fallback SSRC) and surfaced as fields:
- on
ObservedOutboundRtp:remoteRttInMs?,remoteFractionLost?,remoteJitter?,remotePacketsLost? - on
ObservedInboundRtp:remoteRttInMs?,remoteBytesSent?,remotePacketsSent?,remoteTimestamp?
These are reset each tick and only set when the matching remote report is present.
Schema types (ClientSample)
The shape of an accepted sample (re-exported from this package; identical to
@observertc/schemas). Only the top level is shown — each stat object mirrors the standard
WebRTC getStats() dictionaries plus a few extensions.
type ClientSample = {
timestamp: number; // client wall-clock (ms epoch)
callId?: string; // set by you or the library
clientId?: string; // set by you or the library
score?: number; // optional client-computed score (0..5)
attachments?: Record<string, unknown>;
peerConnections?: PeerConnectionSample[];
clientEvents?: ClientEvent[];
clientIssues?: ClientIssue[];
clientMetaItems?: ClientMetaData[];
extensionStats?: ExtensionStat[];
};
type PeerConnectionSample = {
peerConnectionId: string;
attachments?: Record<string, unknown>; // e.g. { direction: 'send'|'recv', producerId, consumerId, label }
score?: number;
inboundTracks?; outboundTracks?;
codecs?;
inboundRtps?; remoteInboundRtps?;
outboundRtps?; remoteOutboundRtps?;
mediaSources?; mediaPlayouts?;
peerConnectionTransports?; dataChannels?;
iceTransports?; iceCandidates?; iceCandidatePairs?;
certificates?;
};
type ClientEvent = { type: string; payload?: string; timestamp?: number; /* +ids */ };
type ClientIssue = { type: string; payload?: string; timestamp?: number };
type ClientMetaData = { type: string; payload?: string; timestamp?: number; /* +ids */ };
type ExtensionStat = { type: string; payload?: string };payload fields are JSON strings; the library parses the ones it understands.
ClientEventTypes (enum of known event.type values): CLIENT_JOINED, CLIENT_LEFT,
PEER_CONNECTION_OPENED/CLOSED/STATE_CHANGED, MEDIA_TRACK_ADDED/REMOVED/MUTED/UNMUTED/RESUMED,
ICE_GATHERING_STATE_CHANGED, ICE_CONNECTION_STATE_CHANGED, DATA_CHANNEL_OPEN/CLOSED/ERROR,
NEGOTIATION_NEEDED, SIGNALING_STATE_CHANGE, ICE_CANDIDATE, ICE_CANDIDATE_ERROR, and the
mediasoup set PRODUCER_* / CONSUMER_* / DATA_PRODUCER_* / DATA_CONSUMER_*.
ClientMetaTypes (enum of known meta type values): MEDIA_CONSTRAINT, MEDIA_DEVICE,
MEDIA_DEVICES_SUPPORTED_CONSTRAINTS, USER_MEDIA_ERROR, LOCAL_SDP, OPERATION_SYSTEM,
ENGINE, PLATFORM, BROWSER.
Worked example: a real ClientSample
Two consecutive samples from one participant ("Guest" in room qq0iwfnd) of an
edumeet/mediasoup call show what actually flows through accept(): a rich join snapshot,
then lean steady-state ticks.
Sample 1 — the join snapshot. Carries the one-off lifecycle clientEvents and device
clientMetaItems alongside the first stats. (Abbreviated; ids and times are from the real log.)
{
"timestamp": 1780572332518,
"callId": "d3dbf2f5-79be-4cb8-9d43-fb404f07ef27",
"clientId": "c926983c-4468-4046-ae8c-a9cabe1a1868",
"score": 0, // no quality measured yet on the join tick
"attachments": { "displayName": "Guest", "roomId": "qq0iwfnd", "actualSessionId": "d3dbf2f5-…" },
"clientEvents": [ // chronological lifecycle (12 in the real sample)
{ "type": "CLIENT_JOINED", "timestamp": 1780572324515 },
{ "type": "PEER_CONNECTION_OPENED", "timestamp": 1780572326790 }, // pc=b81c8d9d (media)
{ "type": "ICE_GATHERING_STATE_CHANGED", "timestamp": 1780572326811 }, // → gathering
{ "type": "PEER_CONNECTION_STATE_CHANGED", "timestamp": 1780572326812 }, // → connecting
{ "type": "PRODUCER_ADDED", "timestamp": 1780572326821 }, // producer=1abdaf82 (audio)
{ "type": "MEDIA_TRACK_ADDED", "timestamp": 1780572326821 }, // track=36ae42df (audio)
{ "type": "PEER_CONNECTION_STATE_CHANGED", "timestamp": 1780572326827 }, // → connected
{ "type": "PRODUCER_ADDED", "timestamp": 1780572326837 }, // producer=ba06a35b (video)
{ "type": "DATA_PRODUCER_CREATED", "timestamp": 1780572326853 }
],
"clientMetaItems": [ // environment & devices, one-off (10 in the real sample)
{ "type": "USER_AGENT_DATA", "payload": "{…Chrome 148 / macOS…}" },
{ "type": "MEDIA_DEVICE", "payload": "{…\"BRIO 4K Stream Edition\"…}" }
// …mic / camera / speaker devices…
],
"peerConnections": [
{
"peerConnectionId": "b81c8d9d-…", // the media PC — Guest publishes to the SFU
"outboundRtps": [ /* audio + video */ ],
"outboundTracks": [ /* mic + camera: label, settings, capabilities */ ],
"remoteInboundRtps": [ /* RTCP feedback from the SFU */ ],
"codecs": [ /* … */ ], "iceTransports": [ /* … */ ],
"iceCandidatePairs": [ /* … */ ], "dataChannels": [ /* … */ ]
},
{ "peerConnectionId": "8635acb7-…", "peerConnectionTransports": [ /* … */ ] } // signaling-only PC
]
}What accept() does with it, in order — each step emits on the bus with full ancestry:
- lazily creates the
ObservedCall→call-added; - creates the
ObservedClient→client-added, thenclient-joined(fromCLIENT_JOINED); - creates an
ObservedPeerConnectionper entry →peer-connection-added(×2 here); - creates an
ObservedOutboundTrackper track →outbound-track-added, plus the matchingoutbound-rtp-added; - replays the device list as
client-metadataevents and the lifecycle items asclient-event; and finallyclient-updatedfor the whole tick.
attachments.roomId lands on observedClient.attachments (read it on client-updated, not at
creation — see Ingestion).
Sample 2 — a steady-state tick (~8 s later): same callId / clientId, no new
clientEvents or clientMetaItems, just refreshed peerConnections stats. Each PC now scores 5
and the aggregate client score is 4.74 — a healthy call. This is the shape of nearly every
sample: each tick refreshes metrics and fires the *-updated events, while the heavy join
snapshot happens only once.
Detectors (server-side extension point)
observer-js ships ten detectors, each an opt-in extension you register
explicitly with addObserverDetector / addCallDetector / addDetector (see
Registering detectors) — none are created automatically. All of them
correlate across the clients of a call or the calls of a fleet, because that is the only thing a
server can do better than a browser: per-client signals — packet loss, jitter, RTT, freezes — are
already detected on the client and arrive on samples as clientIssues (surfaced via client-issue).
Findings are raised as CallIssue or ObserverIssue — both share IssueBase: { type,
timestamp, conclusion?, payload? } — and the payload is the object, not a JSON string. A
server-raised finding is delivered to an in-process handler, so there is nothing to serialise for:
observer.on('call-issue', ({ observedCall, issue }) => {
issue.payload; // the object; no JSON.parse
issue.conclusion?.faultDomain; // a first-class field, not payload.conclusion
issuePayloadAsString(issue); // only at an edge that needs text (log, HTTP, queue)
});(ClientIssue, the type on samples, keeps its string payload — that one really is a wire format.)
The registry is also an open extension point, on ObservedCall:
import { Observer, Detector } from '@observertc/observer-js';
class MyCrossClientDetector implements Detector {
readonly name = 'my-detector';
constructor(private readonly call /* : ObservedCall */) {}
update() { // called on every call.update()
// …inspect this.call.observedClients across participants…
if (/* condition only visible server-side */ false) {
this.call.addIssue({ type: this.name, payload: { /* … */ }, timestamp: Date.now() });
// → emitted on the bus as 'call-issue'
}
}
}
const observer = new Observer();
observer.on('call-added', ({ observedCall }) => {
observedCall.detectors.add(new MyCrossClientDetector(observedCall));
});
observer.on('call-issue', ({ observedCall, issue }) => { /* react */ });Client issues: the lifecycle, and the division of labour
The most important thing to understand about detection in this library is what it deliberately
does not do. A client running
client-monitor-js already ships ~20 detectors
that decide what is wrong with that endpoint — congestion, cpulimitation, audio-concealment,
freezed-video-track, keyframe-storm, video-decoder-overloaded, stuck-decoder,
ice-disconnected, and so on. Those verdicts are better than anything re-derived from raw counters
server-side, because they carry hysteresis and multi-signal confirmation: audio-concealment
subtracts silent concealment (raw concealedSamples rises during ordinary silence, so a naive
detector flags every quiet moment); audio-jitter-buffer-stress requires the buffer to be grown
and NetEQ to be time-stretching (a grown buffer alone means NetEQ is succeeding);
ice-disconnected only fires once disconnected has persisted, so the blips ICE heals on its own
never surface.
observer-js does not repeat that work. Its job is the question no browser can answer: who else is in this state right now, what do they have in common, and where in publisher → SFU → subscriber does the fault begin?
The wire format
From client-monitor-js 4.6.0 the whole issue lifecycle reaches the server. A stateful issue
arrives as two clientIssues[] entries sharing a key:
raise: { type: 'stuck-decoder', key, payload, timestamp: raisedAt }
resolution: { type: 'stuck-decoder-resolved', key, payload: { raisedAt, comment, …final }, timestamp: resolvedAt }The observer opens an entry in observedClient.activeIssues on the raise and closes it on the
matching key, emitting client-issue-resolved with the finished interval. Handled for you:
- the
-resolvedsuffix is stripped, so both entries share one logicaltype; - a re-raise of a live key refreshes the payload without restarting
raisedAt; - keyless entries are one-shot — reported via
client-issue, never tracked; - issues still open when a client closes are force-resolved (
resolvedBy: 'client-closed'), and the registry additionally expires stale entries, so a crashed participant can't leave an issue "active" forever.
Why intervals beat windows
This turns point-in-time symptom reports into intervals, and that is the whole game. "Several clients reported congestion in the last 10 seconds" is a heuristic that has to guess whether the symptoms are still happening. "Several clients are congested right now, simultaneously" is ground truth, because the client says when the episode ends. Overlapping intervals are far stronger evidence of a shared cause than near-in-time reports.
observer.on('client-issue', ({ observedClient, issue }) => { /* opened (or one-shot) */ });
observer.on('client-issue-resolved', ({ resolvedIssue }) => {
resolvedIssue.type; // 'stuck-decoder' — suffix stripped
resolvedIssue.durationInMs; // how long the episode lasted
resolvedIssue.resolvedBy; // 'client' | 'timeout' | 'client-closed'
});
// the live per-client mirror
observedClient.activeIssues; // ObservedClientIssueRegistry, keyed by issue.keyclient-monitor-js >= 4.6.0 is required for every issue-driven detector. There is no fallback path that infers these conditions from raw counters — the client decides better, and maintaining a worse second implementation to be polite to old clients is how both end up wrong. Issues without a
keyhave no lifecycle (nothing could ever close them), so they stay one-shot: reported onclient-issue, never registered.
ActiveIssuesRegistry — issues are pushed, not polled
A detector does not go looking for the issues it cares about. It implements ActiveIssueTracker and
registers for the types it consumes; the registry hands them over as they open and close.
observedCall.activeIssuesRegistry // this meeting
observer.activeIssuesRegistry // the fleet; every call's registry propagates into it
observer.activeIssuesRegistry.addIssueTracker('congestion', myDetector);
observer.activeIssuesRegistry.removeIssueTracker(myDetector);
registry.values(); // the open issues in this scope, oldest first
registry.size; // how manyThe cost of a detector is then proportional to the issues it actually receives, not to the number of participants: a healthy 500-client fleet does no per-tick work at all, because nothing was pushed.
There is no wildcard. A tracker names its types and sees nothing else. "Feed me everything and I'll work out what matters" moves the decision from the application — which knows its client build and its issue vocabulary — onto a detector that has to guess, and it makes the cost of a subscription unbounded and invisible. If a detector should watch five types, list five types.
Onset spread is measured on the observer clock, never the client's. raisedAt comes from each
participant's own machine, and comparing those across clients makes clock skew look like a
synchronized infrastructure event.
Observer-level detectors (cross-call / SFU-wide)
Some findings only exist above call scope — "many calls on the same SFU degraded at once" is far
more actionable than fifty individual client alerts. The same detector registry exists on the Observer,
runs on every observer.update(), and raises findings through observer.addIssue(...), surfaced on
the bus as observer-issue:
observer.detectors.add({
name: 'sfu-wide-degradation',
update: () => {
const degradedCalls = [ ...observer.observedCalls.values() ].filter(isDegraded);
if (observer.numberOfCalls > 3 && degradedCalls.length / observer.numberOfCalls > 0.6) {
observer.addIssue({ type: 'SFU_WIDE_QUALITY_DEGRADATION', timestamp: Date.now() });
}
},
});
observer.on('observer-issue', ({ issue }) => alert(issue));Publisher → subscribers: the resolver links
The question a single browser can never answer is "did everyone receiving Alice see the same
degradation?". The join is the publisher↔subscriber links maintained by a
RemoteTrackResolver, and detectors walk them directly:
outboundTrack.remoteInboundTracks; // Set<ObservedInboundTrack> — every subscriber of this source
inboundTrack.remoteOutboundTrack; // the publisher, or undefined if unlinked
inboundTrack.getInboundRtp(); // that receiver's RTP stats
observedCall.unconsumedOutboundTracks; // published tracks with no subscriber at allA
TrackDistributionAggregatorclass used to sit in front of these links and summarise every published track against all of its receivers, on every tick. It is gone. It scanned the majority (all published tracks) to find the interesting minority, which is the wrong axis — the detectors now start from the handful of affected tracks the issue registry pushed at them and resolve only those. The statistics helpers it used (percentile,median,summarize,counterDelta,robustZScore,SlidingWindow,TrendTester) are all still exported for building your own.
Call health: CallHealthAggregator
The client axis. Where the resolver links answer "how was this source delivered?", this asks "how is each participant doing, sending vs receiving?":
import { CallHealthAggregator } from '@observertc/observer-js';
const health = new CallHealthAggregator(observedCall).aggregate();
health.degradedRatio; // 0.82 — the number that distinguishes shared faults from individual ones
health.inboundDegradedRatio; // receiving side → egress/downstream suspicion
health.outboundDegradedRatio; // sending side → ingress suspicion
health.rttInMs?.median; // percentile rollups, never means
health.qualityLimitation; // { cpu, bandwidth, other } client counts
health.clients; // per-client entries with `reasons`, direction flags, TURN/TCPRegistering detectors
Nothing is created implicitly. A new Observer() has zero detectors. There is no detector
configuration in ObserverConfig and no default set — an application says what it wants to watch, or
it watches nothing.
const observer = new Observer({
createRemoteTrackResolver: createDefaultMediasoupRemoteTrackResolverFactory(),
});
// observer-scoped (cross-call) — built immediately onto `observer.detectors`
observer.addObserverDetector('observer-concurrent-issue-detector', {
issueTypes: [ 'congestion', 'ice-disconnected', 'ice-connection-failed' ],
minAffectedCalls: 3,
});
observer.addObserverDetector('turn-server-outage-detector', { minClientsAtPeak: 10 });
// call-scoped — recorded in `observer.callDetectorConfigs`, applied to every call created AFTER this
observer.addCallDetector('call-concurrent-issue-detector', {
issueTypes: [ 'congestion', 'ice-disconnected' ],
});
// one specific call
observedCall.addDetector('issue-fan-out-detector', { issueTypes: [ 'freezed-video-track' ] });Every add* is chainable — it returns the owning entity:
observer
.addObserverDetector('turn-server-health-detector')
.addObserverDetector('turn-server-outage-detector', { minClientsAtPeak: 10 })
.addValidator('remote-track-resolver');Removing them
By name, on the entity — which removes every instance under that name:
observer.removeObserverDetector('turn-server-outage-detector'); // → 1
observer.removeCallDetector('call-concurrent-issue-detector'); // stops it everywhere
observedCall.removeDetector('issue-fan-out-detector'); // this call onlyBy instance, through the registry — which is where instances live, since add* returns the
entity rather than the detector:
observer
.addObserverDetector('client-population-issue-detector', { issueTypes: [ 'cpulimitation' ], groupBy: 'browser' })
.addObserverDetector('client-population-issue-detector', { issueTypes: [ 'cpulimitation' ], groupBy: 'operationSystem' });
const [ byBrowser, byOs ] = observer.detectors.getAll('client-population-issue-detector');
observer.detectors.remove(byOs); // keeps the browser axis runningDetectors is a small collection: instances (a copy, in registration order), listOfNames,
size, get(name), getAll(name), has(name), add(detector), remove(detector),
removeByName(name), clear(), and it is iterable — for (const detector of call.detectors).
instances being a copy is deliberate: removing while iterating the live array would skip entries,
and "drop the ones that look like X" is the most natural thing to want to write.
Two things worth knowing:
- By name removes every instance under it, not the first. A name can legitimately be registered
more than once —
ClientPopulationIssueDetectoris meant to be added once pergroupByaxis — and "remove whichever is first in the array" is not something a caller can predict from a name. Go viadetectors.getAll(name)+detectors.remove(instance)when you mean one of them. removeCallDetectoraffects calls already open, by default. Otherwise whether a detector runs would depend on when a call happened to join, which is not a state anyone can reason about. Pass{ includeOpenCalls: false }to change only what future calls are built with.
Every removal path calls the detector's close(), so it unsubscribes from the issue registry and
drops any timers or bus listeners. A detector removed without closing would keep being fed matching
issues for the life of the call — invisible, unbounded, and it would still look healthy if you
inspected it.
Detectors are named by their kebab-case NAME, and the name types the config — an unknown name or a
key that belongs to a different detector will not compile. Each detector owns its defaults in its own
constructor, beside the doc explaining what each threshold means; there is no central table to keep
in sync.
Why no defaults? A detector nobody asked for is a detector nobody will act on. It costs time on every tick and raises findings into a handler that was not written to expect them. Earlier versions auto-created everything from a three-state config slot; the result was applications receiving finding types they had never heard of.
Every issue-driven detector takes an explicit, non-empty issueTypes (or
publisherIssueTypes/receiverIssueTypes). There is no "watch everything" option — see
the registry.
🔗 marks detectors that require a
RemoteTrackResolver. They reason about a published track and its subscribers, so without the publisher↔subscriber links they see nothing and stay silent forever — which looks exactly like "no problems found". ConfigureObserverConfig.createRemoteTrackResolver, and start theremote-track-resolvervalidator to prove it is wired.
Built-in detectors
They consume the verdicts client-monitor-js >= 4.6.0 already ships (raise + <type>-resolved) and
add only the cross-participant conclusion. None re-derives a per-endpoint verdict from raw
counters — that is the rule the whole design hangs on:
If a condition is detectable on the client, the client's issue is the source of truth.
| Detector | 🔗 | Scope | Raises |
|----------|:--:|-------|--------|
| CallConcurrentIssueDetector | | call | CONCURRENT_CLIENT_ISSUES, ISSUE_ONSET_BURST |
| ObserverConcurrentIssueDetector | | observer | CROSS_CALL_CONCURRENT_ISSUES, CROSS_CALL_ISSUE_ONSET_BURST |
| IssueFanOutDetector | 🔗 | call | PUBLISHED_TRACK_ISSUE_FAN_OUT, SINGLE_RECEIVER_ISSUE |
| PublisherFaultCorroborationDetector | 🔗 | call | CORROBORATED_PUBLISHER_FAULT |
| TrackDeliveryMismatchDetector | 🔗 | call | PUBLISHED_TRACK_NOT_DELIVERED, RECEIVER_TRACK_NOT_DELIVERED, PUBLISHER_TRACK_DRY |
| UnconsumedTrackDetector | 🔗 | call | UNCONSUMED_PUBLISHED_TRACK |
| ClientPopulationIssueDetector | | observer | CLIENT_POPULATION_ISSUE |
| SfuCongestionDetector | | observer | sfu-congestion |
| TurnServerHealthDetector | | observer | TURN_SERVER_DEGRADED |
| TurnServerOutageDetector | | observer | TURN_SERVER_OUTAGE |
What each adds that no endpoint can know:
CallConcurrentIssueDetector— who else in this meeting is in this state right now? The difference between "one person's Wi-Fi" and "this room is broken".ObserverConcurrentIssueDetector— is our infrastructure in trouble? A separate class, not the call one with a bigger denominator, because it is a different question with different gates. It requires the group to span at leastminAffectedCallsindependent calls (default2) and raises its ownCROSS_CALL_*types. Without that gate, one thirty-person meeting where everyone is congested clears every client threshold and pages you for a single bad room the call-scoped detector already reported. Clients in different calls share no room, no publisher and no host — only the servers, which is what makes the finding conclusive. Note there is deliberately no participant ratio at this scope: six broken calls out of forty is a small share of all clients, and a ratio gate would hide exactly the event you want.IssueFanOutDetector— does this issue follow one published source, or one receiver?PublisherFaultCorroborationDetector— do both ends of one track agree the source is at fault? Fan-out sees one end and infers; this sees the publisher reportingencoder-bottleneckabout its own send path while its subscribers reportfreezed-video-trackabout receiving it. Two independent parties, one conclusion, nothing left to deduce — hence the highest confidence in the library. Run both: fan-out is broader and catches the case where the publisher is fine and the SFU's forwarding is not.ClientPopulationIssueDetector— is this concentrated on one kind of client? The one correlation here that is neither per-call nor per-server. Every other observer-scoped detector reasons "clients in unrelated calls share only the infrastructure, so it must be us" — right for network symptoms, wrong for endpoint ones.cpulimitationacross six unrelated calls is not an SFU event; CPU is owned by the endpoint, so what those endpoints share is a browser version or a client release. Groups bybrowser/engine/platform/operationSystem/location, one axis per instance. The gate is relative risk, not share: "30% of Chrome 141 is unhappy" means nothing if 30% of everyone is, and a share-based rule simply indicts whichever browser is most popular. See thelocationaxis for the geographic form.SfuCongestionDetector— is congestion spiking across the fleet right now? Counts distinct clients reporting congestion in fixed wall-clock buckets and compares each bucket against a median+MAD baseline of the ones before it. Buckets rather than update ticks on purpose: the tick is unevenly spaced and shorter than a client's sampling period, so counting on it compares windows of different lengths and calls the difference a signal. Only add it when the observer's calls all come from the same SFU — the finding's meaning is "these clients share only that server".TrackDeliveryMismatchDetector— are the two ends of a track disagreeing?UnconsumedTrackDetector— is anyone actually subscribed? (reads the resolver's silence)TurnServerHealthDetector— does trouble cluster on one relay?TurnServerOutageDetector— covers the case the health detector structurally cannot. The health detector groups clients by the server relaying them and asks how many report issues — it needs clients on the server to ask. When a TURN server dies, allocation fails: existing sessions drop and new clients never obtain a relay candidate through it, so they are never attributed to it at all. Its population goes to zero and the health detector falls silent for the worst possible reason. Degradation makes clients unhappy; an outage makes them disappear. Absence is a dangerous signal, so the control group is the heart of the design: a call ending, everyone leaving at 6pm, and a fleet-wide network event all look identical to an outage. It refuses to blame a server unless clients not relayed through it are demonstrably still connected (requireControlGroup, on by default).
There is no ICE detector
ICE trouble is reported by client-monitor-js >= 4.6.0 as the keyed issues ice-disconnected,
ice-connection-failed, ice-transport-stalled and unstable-ice-path, each with hysteresis and
multi-signal confirmation behind it. An IceDisruptionDetector used to re-derive that server-side
from raw state transitions; it has been removed, because the server sees less and guesses more. The
client knows whether disconnected persisted or healed in 200 ms; the observer does not.
Correlating ICE trouble is now configuration, not a class:
observer.addObserverDetector('observer-concurrent-issue-detector', {
issueTypes: [ 'ice-disconnected', 'ice-connection-failed', 'ice-transport-stalled' ],
});Grouping by place: the location axis
If your clients report coordinates, ClientPopulationIssueDetector can group by where they are
instead of what they run — which is the grouping network symptoms actually cluster by:
observer.addObserverDetector('client-population-issue-detector', {
issueTypes: [ 'congestion', 'ice-disconnected' ],
groupBy: 'location',
locationPrecision: 3, // geohash chars: 3 ~156 km, 4 ~39 km, 5 ~5 km
resolveClientLocation: (client) => client.attachments?.geo as { latitude: number, longitude: number },
});The client still owns "RTT jumped". client-monitor's CongestionDetector compares each peer
connection's RTT against its own EWMA baseline and requires a bandwidth-limitation corroboration
before raising congestion. Absolute RTT is not comparable between clients — someone 200 ms away is
always 200 ms away, so the only signal is deviation from that client's own baseline, which is
exactly what the client measures. The observer's contribution is the part no endpoint can see: that
many of the affected clients are in the same place at the same time.
Three things to know:
- Cells, not radii. The population is a geohash prefix. "Within N km" is a clustering problem — order-dependent, no stable group name, pairwise cost — and a detector needs the same group key on every tick for its cooldown and control group to mean anything. The cost is that a cell boundary can split two adjacent clients, which biases towards missing a finding rather than inventing one.
- Only the cell key is reported.
payload.populationis the geohash; coordinates never enter the issue. These payloads get archived into call summaries, so that matters. - Geography is confounded with your topology. The control group is "everyone outside this cell", which
