@ai-matrx/desktop-protocol

The one wire contract between Matrx 2 (the desktop app) and everything that drives it — the phone console, the web app, the extension and the server. One zod source; a generated JSON Schema and Pydantic twin that CI keeps byte-identical (SPEC P11: contracts are generated, never written twice).

pnpm add @ai-matrx/desktop-protocol

| Entry | What it is | |---|---| | @ai-matrx/desktop-protocol | Schemas for the envelope and every cap.op, OPS / EVENTS registries, ErrorCode, close codes, DesktopProtocolError. Every schema name is also its inferred type. | | @ai-matrx/desktop-protocol/frame | The 16-byte binary frame codec. Zod-free (the relay Worker's hot path). | | @ai-matrx/desktop-protocol/client | createDesktopClient — reconnecting client with typed requests, credit-gated streams, transparent terminal reattach, heartbeat and relay reauth. | | @ai-matrx/desktop-protocol/react | DesktopClientProvider, useDesktopConnection, useDesktopRequest, useDesktopEvent, useDesktopWake. |

Client

import { createDesktopClient } from "@ai-matrx/desktop-protocol/client";

const client = createDesktopClient({
  url: `wss://relay.matrxserver.com/v1/devices/${deviceId}/connect`,
  // Re-read on every reconnect; tokens ride the subprotocol, never the URL.
  protocols: async () => ["matrx.v1", `bearer.${await getAccessToken()}`],
  getToken: getAccessToken, // answers relay.auth_expiring without dropping the socket
  clientType: "web",
  clientVersion: "1.0.0",
});

const listing = await client.request("fs.list", { path: "/Users/me" }); // typed by op name

const term = client.stream("exec.pty.start", { cols: 80, rows: 24 }, {
  onData: (bytes) => new Promise((done) => xterm.write(bytes, done)), // credit follows consumption
  onSnapshot: (bytes) => {
    xterm.reset();
    xterm.write(bytes);
  },
});
term.write("ls\n");

• Backpressure: credit for a stream is granted only after onData returns (or its promise settles), so a slow phone never makes the relay or the Mac buffer without bound.
• Reattach: a resource stream (terminal, watch) survives a dropped socket. On reconnect the client sends session.attach with since_seq = the last frame your onData consumed; the core replays the gap or sends gap + a screen snapshot. The same DesktopStream keeps working.
• Errors are DesktopProtocolError with code, retryable and data.reason — branch on those, never on messages. A dropped socket rejects unary work with DEVICE_OFFLINE (retryable).
• Terminal closes (4010 revoked, 4003 forbidden, 4004 not found, 4009 replaced, 4426 upgrade) end the client for good; everything else reconnects with the shared @ai-matrx/realtime backoff (stability reset + jitter) — including 1006 (a relay deploy drops every socket without a close frame) and 1012 (the relay's "the device reconnected": hello and reattach against the new core session).
• Coming back: client.wake() (or bindDesktopWake(client) / useDesktopWake(), which call it on visibilitychange, pageshow, online and focus) dials at once while reconnecting and probes an open socket with a ping, redialing if nothing answers within 4 s.
• Reload reattach: keep stream.resourceId and stream.lastSeq (e.g. in the URL); after a reload client.stream("session.attach", { resource_id, since_seq }) replays the gap and then behaves exactly like the original stream, including surviving later drops.
• Background problems (malformed messages, sequence gaps, heartbeat death, reconnect alarm) go to onDiagnostic — loud by default (console.warn).

Local transcription

import { createDesktopTranscriber } from "@ai-matrx/desktop-protocol/client";

const transcriber = createDesktopTranscriber(client, { model: "base", onProgress: (p) => show(p) });
const { text, segments } = await transcriber.transcribe(recordingBlob); // on the person's computer
// Or as a voice input's transport: new BrowserAudioKernel({ transport: transcriber })

React

<DesktopClientProvider client={client}>
  <Files />
</DesktopClientProvider>

function Files() {
  const { status } = useDesktopConnection();
  const { data, error, loading, refetch } = useDesktopRequest("fs.list", { path: "/Users/me" });
  useDesktopEvent("fs.changed", () => void refetch());
}

Python (aidream)

from aidream._generated.desktop_protocol import OPS, FsListParams, validate_wire

params = validate_wire(FsListParams, raw_json)  # same verdict as zod, always

validate_wire is the one Python entry point: strict (no "2" → 2) and JavaScript-faithful (2.0 is the integer 2). OPS maps every op to its params/result models, kind and errors.

Python services (matrx-desktop)

The desktop's supervised Python services generate their models from the installed package with the same generator, so the core and the service can never drift:

python3 node_modules/@ai-matrx/desktop-protocol/scripts/gen_pydantic.py \
  --schema node_modules/@ai-matrx/desktop-protocol/generated/desktop-protocol.schema.json --out <module>.py

The module carries SERVICES, a typed handler Protocol per service (MlServiceHandlers) and CONTRACT_SHA256, which must equal the core's (import { CONTRACT_SHA256 }) in the service hello.

Changing the protocol

Edit src/schema/*.ts, then pnpm generate and commit all three files. Rules (enforced by pnpm lint and by generate): z.strictObject only; no .refine/.transform/z.coerce/ .meta; never optional and nullable at once; a new named schema goes into NAMED_SCHEMAS; add a parity case to corpus/parity-corpus.json for any new wire shape.

MIT © AI Matrix Engine