@mnemopay/computer-use

Give an AI its own containerized Linux desktop — and govern every move it makes.

@mnemopay/computer-use runs a Claude computer-use agent inside a disposable Ubuntu/XFCE container and routes every action it takes through MnemoPay's governance spine before it executes:

screenshot → vision decision → budget precheck → policy + risk verdict
           → human-in-the-loop approval → reserve funds → execute
           → settle / release → debit budget → append to audit chain

The whole run is sealed into an EU AI Act Article 12 record bundle: a hash-chained event log with a detached Ed25519 signature over its head, so anyone holding the public key can verify — offline — that the recorded steps, spend, and approvals are exactly what happened and nothing was edited after the fact.

It's the auditable alternative to handing a raw computer-use loop the keys to a machine and a wallet.

Install

npm install @mnemopay/computer-use

The core (loop, providers, audit, signing, payments seam) has one runtime dependency (@modelcontextprotocol/sdk). Real policy/risk governance lives in @mnemopay/sdk, which is an optional peer dependency — you only need it if you import the /sdk-governance subpath. Everything else builds and runs without it.

Quickstart: boot a desktop

The container ships in this repo. Build and run it with Docker Compose:

docker compose up --build
curl localhost:8787/healthz            # confirm mnemopayd is live (returns geometry)

Watch — or take over — the desktop live in a browser:

http://localhost:6080/vnc.html         # noVNC web viewer
vnc://localhost:5900                    # or any native VNC client

Ports: 8787 mnemopayd (the DesktopAction HTTP contract), 5900 VNC, 6080 noVNC web viewer. Set MNEMOPAYD_TOKEN in production and pass the same value to the provider.

Run a governed task

import { desktopTask } from "@mnemopay/computer-use";
import { LocalDesktopProvider } from "@mnemopay/computer-use/providers";
import { ClaudeVisionDriver } from "@mnemopay/computer-use/agent";
import { createEd25519Signer, verifyArticle12Signature } from "@mnemopay/computer-use/audit";

const signer = createEd25519Signer(); // mints a fresh did:mp identity

const { run, audit } = await desktopTask(
  new LocalDesktopProvider({ baseUrl: "http://127.0.0.1:8787" }),
  {
    did: signer.did,
    task: "Open Firefox and search for the EU AI Act Article 12 text",
    budget_usd: 1.0,
    vision: new ClaudeVisionDriver({
      createMessage: myAnthropicEdge, // you own the only call to the model
      model: "claude-sonnet-4-6",
      costPerActionUsd: 0.01,
    }),
    signer,
  },
);

console.log(run.status, run.spent_usd);

// Verify the sealed bundle with nothing but the public key.
const check = verifyArticle12Signature(audit, signer.publicKeyHex);
console.log(check.ok); // true

desktopTask opens the session, runs the loop with the audit collector wired in, seals the Article 12 bundle, and always closes the session — even on error.

Provisioning a fresh container per run

LocalDesktopProvider fronts a daemon you started yourself. To get a brand-new disposable desktop per session and tear it down on close, use DockerDesktopProvider — it shells out to the Docker CLI (argv arrays, never a shell string, so nothing is injectable from a task prompt):

import { DockerDesktopProvider } from "@mnemopay/computer-use/providers";

const provider = new DockerDesktopProvider({
  image: "mnemopay/computer-desktop:dev",
  exposeNoVnc: true,          // publish 6080 on an ephemeral host port
  token: process.env.MNEMOPAYD_TOKEN,
});

const { run, audit } = await desktopTask(provider, { /* …same options… */ });
// `provider.novncUrl(sessionId)` → a browser-watchable URL while the run is live.

open runs the image, waits for mnemopayd to report healthy, and fronts it with an internal LocalDesktopProvider; close stops and removes the container. The governed loop never learns a container was spun up underneath it. Requires a working docker on PATH and the image already built (docker compose build or docker pull).

Use it as an MCP server

Expose the whole governed loop to any MCP client (Claude Code, Claude Desktop, Cursor, Windsurf, …) as a single high-value tool, desktop_task:

claude mcp add computer-use -s user -- npx @mnemopay/computer-use mcp

Tools exposed:

• desktop_task — { task, budget_usd?, max_steps?, model? }. Drives the desktop through the full governed loop and returns the run outcome plus the signed Article 12 audit summary (including the public key to verify it with).
• desktop_health — probes the daemon; returns reachability + geometry.

Configure via environment:

| Variable | Default | Purpose | | --- | --- | --- | | ANTHROPIC_API_KEY | (required) | Drives the computer-use vision model | | MNEMOPAYD_BASE_URL | http://127.0.0.1:8787 | Desktop daemon URL | | MNEMOPAYD_TOKEN | — | Bearer token, if the daemon requires one | | CLAUDE_MODEL | claude-sonnet-4-6 | Vision model | | MNEMOPAY_DID | (minted) | Agent DID to attribute runs to | | MNEMOPAY_APPROVAL_USD | 50 | Spend above this routes to approval | | MNEMOPAY_HARDCAP_USD | 5000 | Spend above this is blocked outright | | MNEMOPAY_DEFAULT_BUDGET | 1 | Default per-run USD ceiling | | MNEMOPAY_MAX_STEPS | 12 | Default loop step cap |

Governance, payments, and audit

Policy + risk + approval. Wire the real MnemoPay stack with one call:

import { createSdkGovernance } from "@mnemopay/computer-use/sdk-governance";

const gov = createSdkGovernance({ approvalThresholdUsd: 50, hardCapUsd: 5000 });
await desktopTask(provider, {
  /* … */ governance: gov.hooks, payments: gov.payments,
});
// Resolve a pending approval: gov.approvals.decide(id, "approve", who)

This is the only file that imports @mnemopay/sdk; it's behind the /sdk-governance subpath so the core stays SDK-free. Without it the loop defaults to allow-all governance and budget-only accounting (no money moves).

Payments seam. When an action carries an estimated_usd, the loop reserves funds before executing, settles on success, and releases the hold if the action fails to execute — so a crashed click never costs money. The PaymentsPort contract (reserve / settle / release) is byte-identical to the SDK's PaymentsAdapter, so a wallet-backed adapter (e.g. over a Stripe rail) drops straight in. Default is the SDK's MockPayments (in-memory holds).

Article 12 audit. Every governed event is appended to a SHA-256 hash chain. createEd25519Signer() seals the final head; verifyArticle12Signature(bundle, publicKeyHex) re-derives the head from the leaves and checks the signature, returning a specific error (chain_tampered, head_mismatch, did_pubkey_mismatch, …) if anything was altered. The signing format is wire-compatible with @mnemopay/sdk/identity, so bundles interoperate with the rest of MnemoPay.

Action vocabulary

DesktopAction covers: screenshot, cursor_position, move, click, double_click, mouse_down, mouse_up, drag, type, key, scroll, wait. Each may carry an optional estimated_usd (drives the payments seam) and label (audit-readable description).

Package layout

| Subpath | Contents | | --- | --- | | @mnemopay/computer-use | Everything: types, providers, loop, audit, fiscal-gate | | …/providers | MockDesktopProvider, LocalDesktopProvider, DockerDesktopProvider | | …/agent | desktopTask, the loop, vision drivers, governance + payments seams | | …/audit | Article 12 bundle + Ed25519 signer/verifier | | …/sdk-governance | Real @mnemopay/sdk governance binding (opts into the SDK) | | …/mcp | The MCP server entrypoint |

License

Apache-2.0 © J&B Enterprise LLC