@calltelemetry/ct-lab-mcp

Model Context Protocol (MCP) Server for Unified Call Telemetry Lab, Fleet, and Appliance Management across Proxmox VE and VMware ESXi

@calltelemetry/ct-lab-mcp is the single public MCP façade for Call Telemetry lab operations. It exposes high-level multi-hypervisor management (Proxmox VE & VMware ESXi), disposable VM workflows, and secure appliance diagnostics while delegating canonical asset identity, lease ownership, and release-line policy to the private lab-inventory CLI supplied by ct-meta.

Architecture

AI Clients (Claude Desktop / Cursor / Web / AI Agents)
                  │
                  ▼
       [MCP Server / Transports]
       ├── Stdio Transport (stdout JSON-RPC, stderr SafeLogger)
       └── Express SSE Transport (/sse, /messages, /health)
                  │
                  ▼
         [MCP Tool Registry]
  ├── list_hosts          (Proxmox + ESXi capacity, health & resource-derived 8GB unit estimates)
  ├── list_inventory      (Unified VM fleet inventory with Linear tags)
  ├── claim_slot          (8GB slot leases with TTL & Linear issue tracking)
  ├── release_slot        (Release lease + optional automated VM destruction)
  ├── provision_vm        (Disposable VM provisioning on Proxmox/ESXi)
  ├── proxmox_quick_provision (Size-driven randomized Proxmox fleet plan/apply)
  ├── pin_pool_slots      (Wraps pin-pool-slots.sh: pin web SHA tag on guest IPs)
  ├── vm_action           (Power lifecycle with non-bypassable safety guardrails)
  ├── host_maintenance_shutdown (Guarded whole-host drain and graceful shutdown)
  ├── lab_mission         (Canonical manifest-bound multi-phase mission)
  ├── lab_dev             (Canonical leased remote-development lifecycle)
  ├── list_appliance_claims (Kubernetes ApplianceClaim CRD fleet discovery)
  ├── create_appliance_claim (Declarative CRD provisioning with channel rules; profile=demo|bare)
  ├── renew_appliance_claim (Extend a claim TTL; never shortens)
  ├── claimed_import_qcow (Image to claimed guest: claim-first bare claim, then canonical vm provision --source published-qcow)
  ├── delete_appliance_claim (Operator teardown trigger via CRD finalizer)
  ├── sideload_appliance_runner (Isolated runner commands & container overrides)
  ├── get_appliance_id    (Hardware UUID -> CT-APP-XXXX-XXXX-XXXX derivation)
  ├── sideload_license    (Standardized calltelemetry-license-v2 envelope)
  ├── verify_appliance_demo (Automated 5-point demo verification gate)
  └── appliance_exec      (Mediated cli.sh commands over SSH:2222 with redaction)
                  │
  ┌───────────────┼───────────────┬───────────────────┐
  ▼               ▼               ▼                   ▼
[MultiHypervisor] [Slot Engine]   [Safety Guardrails] [Appliance SSH]
  ├── Proxmox VE   ├── ProxmoxLab  ├── Proxmox fixtures ├── ssh2 port 2222
  │   (2x8GB)      │   (2x8GB max) │   (100-104, 9000s) ├── cli.sh / db vacuum
  └── VMware ESXi  ├── ProxmoxMini └── ESXi fixtures    ├── migrate / logs
      (8x8GB)      │   (4x8GB max)     (CUCM, CUBE,     └── Redaction Engine
                   └── ESXi (8x8GB)     Runners, K8s)
                          │
                          ▼
            [Lab Credential Resolver]
            ├── Tier 1: Process Environment (LAB_PROXMOX_*, GOVC_*, LAB_VM_*)
            ├── Tier 2: Doppler REST API v3 (DOPPLER_TOKEN)
            └── Tier 3: In-Memory Mock Fallback / Offline Dev

In a governed ct-meta installation there is only one registered MCP server: @calltelemetry/ct-lab-mcp. The Python lab-inventory program remains a private canonical policy engine behind the Node façade; it is not registered as a second public MCP. Set CT_LAB_CANONICAL_REQUIRED=1 to fail closed rather than use the legacy Node VMID/name slot state when the bridge is unavailable.

Key Features

• Multi-Hypervisor Fleet Management: Polymorphic abstraction unifying Proxmox VE clusters (proxmox-lab, proxmox-mini) and VMware ESXi hosts (esxi-intel-192-168-123-176).

• Disposable VM Provisioning: Give proxmox_quick_provision one resource size (memoryMb, vcpus, diskGb). Live capacity filters the homogeneous Proxmox pool, then one eligible host is selected with system randomness. Apply repeats the reviewed explicit host, VMID, and operation ID.

• Thin-only guest disks (REL-1588, "never thick disks"): Every disk provision_vm, auto_provision_vm, proxmox_quick_provision, proxmox_import_qcow, claimed_import_qcow, host_migrate_vm and sandbox_clone creates, clones, migrates or imports on Proxmox must be on thin storage: lvmthin (the local-lvm pool on both lab hosts), or a zfspool proven sparse. Anything else (lvm, dir, nfs, iscsi, unknown) is refused with the stable error code THICK_STORAGE_REFUSED before any claim, journal, clone, import or other Proxmox change. The storage type is read through the same host capacity path as list_storage; if it cannot be determined the request fails closed. An omitted storage is checked against the default pool (local-lvm). importStorage (the dir pool that only stages a downloaded image) is not a guest-disk target and is not restricted. This server never builds Proxmox disk options itself, so discard=on is owned by the canonical transports and templates.

• Resource-based Lease Admission: There are no fixed per-host or global VM-count quotas. Fresh host health, physical free RAM, VM-image storage, and the requested guest size determine placement. Outstanding unprovisioned claims reserve their size under the claim mutex; backed running VMs are already reflected in physical usage. Unknown, offline, or insufficient resources fail closed.

  • Legacy slot tool names and numeric counts remain available. maxSlots is the observed lease count plus an estimate of additional standard-size capacity; it never gates admission.
  • ESXi leasing remains explicitly opt-in; platform authorization and protected assets remain separate from resource availability.

• Non-Bypassable Safety Guardrails: Hard protection preventing destructive actions (destroy, stop, reboot) against core infrastructure:

  • Cisco Unified Communications Manager (CUCM 14 / 15 publishers and subscribers)
  • Cisco CUBE Voice Gateways / Cat8000v
  • GitHub Actions CI Runners (GH-Runner)
  • Lab Kubernetes Clusters (LAB-K8S)
  • VM Templates (VMIDs 9000–9999 or template flags)
  • SPA-1103 on minis VMID 182, including cleanup of expired or untracked rows

• Physical Maintenance Shutdown: host_maintenance_shutdown accepts one exact canonical Proxmox or ESXi host. It plans by default; apply requires the reviewed plan digest and exact host confirmation, preflights credentials, gracefully drains catalog-matched unclaimed guests, shuts CUCM subscribers before publishers, and refuses active claims, partial discovery, identity drift, clustered Proxmox, or an ESXi maintenance-mode failure. Force host power-off is not implemented.

• Remote Appliance CLI Control: Mediated execution of cli.sh commands over SSH (port 2222) for status inspection, health diagnostics, Ecto migrations, table vacuums, and container logs.

• Leased Remote Development: lab_dev is a canonical-only façade for the ct-local-dev-v1 profile. It supports strict doctor, plan, prepare, start, status, sync, stop, and cleanup actions bound to one asset, lease, Linear issue, operation ID, and reviewed receipt chain. It accepts no guest commands, remote paths, or mock fallback.

• Physical-slot claim custody: Proxmox claim creation requires the actual canonical node, explicit VMID and VM name in ct-lab, with a finite TTL. A shared Kubernetes Lease mutex serializes the physical host/node/VMID; its durable reservation stays occupied through unknown, partial and retiring outcomes. The source stamps a nonce and effective creation-request hash, then independently reads the original CRD UID before binding. Whole effective immutable CRD fields and creation labels are rehashed on bind, retry and deletion; only the declared lifecycle fields and stale label can change. A matching name alone grants no custody.

• One configured Kubernetes claim backend: ApplianceClaim CRUD and physical-slot Lease operations share one backend resolver. Local stdio deployments must set an absolute LAB_CLAIM_KUBECONFIG, the exact LAB_CLAIM_K8S_CONTEXT, and an HTTPS-origin LAB_CLAIM_K8S_SERVER; the selected kubeconfig is validated against that server before a request. In-cluster deployments must set the context/server and provide the service-account token and CA for that same cluster. Cross-cluster deployments use the explicit kubeconfig path. The process never falls back to KUBECONFIG, ~/.kube/config, or an embedded server, and a missing or mismatched binding is refused before claim or reservation effects. The Flux-managed consumer manifest ct-infrastructure/clusters/doks-nyc1/apps/ct-lab/deploy-ct-lab-mcp.yaml is a companion rollout change: the image-roll workflow refuses to update it until all three backend variables are present, and the kubeconfig path must be mounted from the selected cluster's protected configuration.

• Original UID deletion: delete_appliance_claim requires expectedUid from the independently observed original claim. It sends UID/resourceVersion DeleteOptions preconditions and leaves the physical reservation retiring. The qualified operator/controller must independently prove original guest, owned-volume and original-UID absence before terminal release; an error or expiry-looking record never permits clearing protection.

• Guarded image import: claimed_import_qcow creates a bare claim, waits for an operator to set it Ready (default 120 s, CT_LAB_CLAIM_RECONCILE_TIMEOUT_MS; a claim no operator reconciles fails with CLAIM_NOT_RECONCILED before any image import, because the claim plane and the operator's cluster differ), then invokes the canonical vm provision --source published-qcow lifecycle. Its actual default bridge keeps host CPU semantics; portable first-power qualification uses the separate explicit canonical source route. Returned, thrown and uncertain import errors retain both claim and reservation protection and preserve diagnostics. This path requires the qualified controller and operator successors; a source pin alone does not establish deployed readiness.

• Legacy SDK effects: Direct SDK guest creation/clone/power/config/destruction, unguarded script guest writes and generic guest SSH authentication/commands are unavailable without the implemented canonical lifecycle. No caller flag or environment switch enables the retired effects. The historical purge daemon reports authority_required with zero releases/destruction and keeps all protection records. Provider inventory reads and implemented typed canonical operations remain available.

• Sideloaded Appliance Management Runner: sideload_appliance_runner executes isolated runner actions (execute, container_override, systemd_reset, diagnostic_bundle) over SSH (port 2222) using ephemeral runner payloads, eliminating brittleness and version drift from mutating /home/calltelemetry/cli.sh.

• Standardized 0.8.7+ Appliance Licensing: get_appliance_id derives canonical CT-APP-XXXX-XXXX-XXXX from the host hardware UUID. sideload_license delivers modern calltelemetry-license-v2 JSON envelopes directly to /etc/calltelemetry/calltelemetry.lic with strict chmod 600 permissions.

• Automated 5-Point Demo Verification Gate: verify_appliance_demo runs an end-to-end certification gate checking API session authentication, active license status, CUBE gateway connectivity, phone endpoint registrations, and processed CDR/call records before demo handoff.

• Dynamic Secret Redaction: Real-time masking prevents passwords, Doppler API tokens, and private keys from leaking into MCP responses or stderr logs.

• Dual Transports: Clean standard I/O transport (stdio) and Express HTTP/SSE transport (sse) with /health, /sse, and /messages endpoints.

• 3-Tier Credential Resolution:

  1. Tier 1: Direct Environment Variables (LAB_PROXMOX_*, LAB_VM_CREDENTIALS_*; ESXi GOVC_* is optional and retired)
  2. Tier 2: Doppler Secrets REST API v3 (DOPPLER_TOKEN)
  3. Tier 3: In-Memory Mock Hypervisor & Appliance Fallback (for offline test suites)

Installation

# Global installation
npm install -g @calltelemetry/ct-lab-mcp

# Or run directly with npx
npx @calltelemetry/ct-lab-mcp

Merges to main publish automatically. If package.json version is not on npm, that version is published, tagged, and released. If it is already on npm, CI bumps the patch version, publishes, tags, and creates the GitHub Release. --dry-run / docs-only work that should not ship must not land on main, or bump the version only when the package should ship.

# Current published line
npm view @calltelemetry/ct-lab-mcp version

CLI Usage

ct-lab-mcp [options]

Command-Line Options

| Flag | Description | Default | Environment Variable | |------|-------------|---------|----------------------| | -t, --transport <type> | Transport mode (stdio or sse) | stdio | CT_LAB_MCP_TRANSPORT | | -p, --port <number> | HTTP port for SSE transport | 3000 | PORT | | -H, --host <string> | Host interface to bind SSE server | 127.0.0.1 | HOST | | -c, --config <path> | Path to optional JSON configuration file | None | None | | --doppler-token <token> | Doppler config-scoped service token (prefer the environment variable) | None | DOPPLER_TOKEN | | --doppler-project <name>| Doppler project name | None | DOPPLER_PROJECT | | --doppler-config <name> | Doppler environment / config | None | DOPPLER_CONFIG | | None | Bound Doppler API resolution; invalid values use 8 seconds, valid values clamp to 250-30000 ms | 8000 | CT_LAB_MCP_RESOLVE_TIMEOUT_MS | | None | ct-dashboard base URL used to issue lab licenses | https://dash.calltelemetry.com | DASHBOARD_URL | | None | CRM API key with licenses:write, used to issue lab licenses (Doppler ai-workspace/prd) | None | DASHBOARD_API_KEY | | --force-mock | Force in-memory mock hypervisor & appliance mode | false | CT_LAB_MOCK_MODE | | --redact-secrets | Enforce strict secret redaction on all logs | true | None | | preflight --asset-id <id> --linear-issue <key> | Validate canonical inventory, exact VM identity, and release-line qualification before guest commands | n/a | CT_LAB_CANONICAL_CLI, LAB_INVENTORY_FILE, CT_LAB_CANONICAL_REQUIRED | | fleet versions --asset-id <id> | Qualify exact canonical assets against their cut release | n/a | CT_LAB_CANONICAL_CLI, CT_LAB_CANONICAL_REQUIRED | | fleet candidates --memory-mb <mb> --vcpus <n> --disk-gb <gb> | Rank reusable VMs and host capacity; all three capacity requirements are mandatory | n/a | CT_LAB_CANONICAL_CLI, CT_LAB_CANONICAL_REQUIRED | | fleet group <group-id> | Resolve a canonical fleet group without mutation | n/a | CT_LAB_CANONICAL_CLI, CT_LAB_CANONICAL_REQUIRED | | -v, --version | Output the current version | None | None | | -h, --help | Display CLI help text | None | None |

Examples

# Run over stdio (default for Claude Desktop and Cursor)
ct-lab-mcp

# Run SSE server on port 3001
ct-lab-mcp --transport sse --port 3001 --host 0.0.0.0

# Run with a config-scoped Doppler service token already exported by the caller
DOPPLER_PROJECT="ai-workspace" DOPPLER_CONFIG="dev" ct-lab-mcp

# Offline mock mode for development / unit testing
ct-lab-mcp --force-mock

# Governed fixture qualification (the release is cut by ct-release; the lab
# only qualifies the exact fixture against that release)
ct-lab-mcp preflight --asset-id proxmox-vm-uat-01 --linear-issue UAT-1202

# Rank only Proxmox candidates that satisfy the appliance's capacity floor
ct-lab-mcp fleet candidates --memory-mb 8192 --vcpus 4 --disk-gb 200 --hypervisor proxmox

Canonical lab mode

ct-meta starts the public MCP with these environment variables:

CT_LAB_CANONICAL_CLI=/path/to/lab-inventory
CT_LAB_CANONICAL_REQUIRED=1

In this mode lab_preflight, inventory and storage-capacity reads, claims, releases, reclaim, provisioning, VM lifecycle, and inventory-bound CUBE IOS-XE calls execute through the canonical CLI. The Node façade never creates a second lease store and never treats a VMID, name, IP address, Linear key, or operation ID as ownership. Mutation calls require the canonical asset identity, exact operation inputs, and the CLI's dry-run / apply receipt workflow. Legacy guest, storage, snapshot, CUBE lease, and appliance mutations that do not yet have an identity-bound bridge still return CANONICAL_OPERATION_REQUIRES_BRIDGE.

Canonical host reads dispatch one identity-bound CLI discovery per requested host. Multi-host reads keep successful host payloads and return partial: true with structured discovery_unavailable warnings for failed peers; an exact-host read, or a read where every requested host fails, remains fail-closed. list_inventory consumes the canonical calltelemetry-lab-vm-inventory/v1 partial-discovery contract and may project its rows by exact host or hypervisor without creating another inventory. Exact host reads invoke vm inventory --host-asset-id, hypervisor reads invoke vm inventory --hypervisor, and only all-scope reads invoke vm inventory --all. Warnings are preserved, with the current singleton ESXi warning bound to its canonical host identity. Status, template, text, and tag projection remains unsupported until those fields have an authoritative canonical CLI contract.

list_storage delegates to lab-inventory host capacity --asset-id once per requested exact host. Its default includeContent: false returns normalized pool/datastore capacity plus Proxmox pool content capabilities. host: all preserves healthy hosts and emits partial: true with structured unreachable warnings; an exact-host failure or all-host failure remains fail-closed. The canonical CLI does not currently prove individual ISO, VM-disk, or backup volume contents, so includeContent: true returns CANONICAL_DETAIL_UNSUPPORTED without consulting legacy Node credentials. ESXi datastore enabled, shared, and content-capability fields are returned as null because the current canonical capacity contract does not expose them.

lab_mission is the bounded façade for lab-inventory mission run. It accepts an immutable manifest path and canonical SHA-256, plans by default, and requires the exact asset confirmation plus a clean registered executor checkout for apply. Its internal mission deadline defaults to four hours and cannot exceed six hours. The canonical CLI receives that deadline. The Node watchdog waits an additional fixed 30 seconds for the canonical MissionRunner to terminate its executor group, write its terminal receipt, restore state, and release its lock. If that grace expires, Node terminates only the CLI-owned process group and reports a hard timeout; it does not claim to discover or clean up executor groups that deliberately created a separate session. Signal termination is always reported as a nonzero failure with signalCode preserved. It has no caller-selected discovery source, command name, or free-form argument input, and it stores no lease, mission, inventory, or receipt authority in the Node process. Package rollout therefore depends on the corresponding canonical MissionRunner deadline and cleanup implementation in ct-meta.

lab_dev is the bounded façade for the canonical dev lifecycle. Its only supported profile is ct-local-dev-v1; doctor is read-only and may omit the identity triplet, while plan requires an exact asset, lease, Linear issue, operation ID, workspace reference, source digest, and receipt path. prepare, start and sync additionally require the profile, source digest, separate reviewed plan and apply receipt paths, and exact asset confirmation for apply. status accepts only the identity triplet plus an optional operation ID. stop and cleanup require the identity triplet, operation ID, and separate plan/apply receipt paths. All actions delegate to canonical dev <action> argv; the Node façade never provisions, executes arbitrary guest commands, accepts caller-selected remote paths, or converts an unavailable/timeout result into success. The canonical authority owns workspace allowlists, source synchronization, process supervision, lease expiry, and durable receipts. plan is the explicit planning action. Lifecycle calls with apply: false are refused by the canonical controller; they are not alternative plan calls.

Development guest creation is a separate provision_vm request with explicit provisionProfile: "remote-dev-v1", source: "alma97-cloud", and the reviewed offline devArtifactDir. Canonical authority validates the artifact; apply requires confirmBootstrapManifestSha256 from the plan alongside the existing asset, host, VMID, name and cloud-image confirmations. Do not supply an appliance version, release digest, native CLI override or port-2222 readiness poll for this profile. auto_provision_vm rejects development inputs rather than silently treating them as an appliance request. This candidate facade does not enable an unfinished provider: canonical bootstrap checks still apply. OS/container bootstrap completion is not source synchronization, application readiness, browser HMR, or product/video proof.

Implementation status for REL-822: the candidate MCP delegates all eight actions to the canonical controller. The lifecycle controller journals prepare/start/sync/stop/cleanup and reconciles their observed state; it must report the remote adapter as unavailable when no supported adapter is registered. The production remote adapter and live HMR proof remain incomplete. Tool discovery, passing synthetic protocol tests, or a recorded plan is not a running preview, and this candidate has not been published as a new MCP version.

The response separates transportSuccess (the canonical command returned) from actionSuccess and ready/runtimeReady. diagnosed, planned, not_started, prepared, stopped, and cleaned can complete without application readiness. running is accepted but has actionSuccess: false and ready: false; poll status for the same operation. A ready response requires successful runtime execution, synchronized source, packaging and dependency digests, and positive private-transport, API, and SPA health observations. Even those checks do not prove actual browser HMR; hmr_browser_proof remains separate. Readiness must also match the caller's exact asset, lease, issue, operation, workspace, profile, and source; nested request identity/source must agree with the receipt, and the observation must be no older than 30 seconds (with at most five seconds of future clock skew). A status call without an operation ID may report the current operation, but must still match the requested identity.

prepare is an explicit development-only cache step. It shares the strict reviewed-plan/apply input and preserves operation identity after a timeout. prepared requires a fresh bound preparation-receipt digest, exact synchronized source/dependency/packaging, all three health checks false and no preview URL. It never authorizes capture. Start must verify prepared dependencies without hidden installs, builds or pulls. The first candidate consumes approved local immutable images only; missing images/private build authentication remains a separate explicit preparation failure, not an automatic acquisition fallback. The canonical source receipt must carry a valid runtimePackagingDigest and its fresh observation must match observed_runtime_packaging_digest. Missing, malformed or stale packaging identity is not readiness, even when transferred source bytes match. This digest binds audited omissions without exposing their private paths. Older canonical controllers must be updated before they can qualify a running preview; doctor and non-ready diagnostics remain available. The canonical controller alone opens and validates planReceiptFile; the facade checks the returned plan digest's format and directly available correlations, never opens arbitrary receipt paths, and checks output receipt paths against the caller's receiptFile when supplied.

Malformed receipts, contradictory nested health evidence, explicit canonical failure, and unknown outcomes never become ready. A controller timeout returns unknown-after-timeout with mutation outcome unknown and nextAction: status; do not resend a fresh mutation merely because the transport timed out. A required, configured canonical bridge validates credentials for each exact lab_dev action rather than requiring unrelated legacy aggregate credentials. Unknown actions and arbitrary command/source-root fields remain rejected. Tool discovery advertises the complete typed field envelope; the handler then enforces each action's exact required and forbidden fields before dispatch.

Run the real built-binary SDK/stdio contract tests with:

npm run build
npx vitest run test/tools/lab-dev.test.ts test/tools/lab-dev-stdio.test.ts test/tools/canonical-credential-readiness.test.ts

The stdio tests use an explicitly synthetic canonical authority, discover the actual built server's schema, and dynamically call all eight actions plus failure, timeout, redaction, and strict-input cases. They do not provision a VM, transfer source, or establish live product acceptance.

vm_repair is a separate canonical-only credential-recovery facade for an incomplete provision. It requires assetId, leaseId, linearIssue, the original provisionOperationId, a separate repair operationId, and the live confirmSmbiosUuid. Plan is the default; apply also requires an exact confirmAssetId. The canonical implementation must prove pending intent, live identity, unexpired ownership and shared locks before restoring that VM's credential. No host/IP, shell, secret, arbitrary receipt path or fixture file is accepted. Restored management access never implies application readiness or a completed provision; the original pending operation remains authoritative.

Serial-console retries use apply: true, reconcile: true and matching confirmAssetId with the same operation and retained material. An explicit restartConsole: true can recover a proven silent login prompt within an approved disposable recovery: canonical authority journals one graceful shutdown/start and verifies a fresh process before console observation. It never force-stops, blindly types into a terminal, repeats an uncertain power action or resets the password again. This requires a compatible canonical backend; successful tool discovery alone does not prove live recovery.

CUBE control is cube_list, cube_status, cube_exec, cube_config, cube_dialpeer_show, cube_dialpeer_apply, cube_xcc_enable, cube_xcc_status, and cube_ios_snapshot. Power stays vm_action / canonical vm_power. Claims stay vm_claim. Dial-peer apply is additive only. Wipe-first setup_cube_xcc is not registered.

Claiming policy is intentionally conservative: prefer a fresh unclaimed stopped candidate or a new disposable VM; reuse an active running VM only when the current run proves the exact lease, coordinator ref, and live managed metadata. Reclaim is permitted only for the oldest eligible expired claim when all eligible capacity is full. Protected and unknown assets are never reclaimed. A disposable claim that failed authentication may be destroyed and recreated only after preserving the failure receipt.

Canonical mutation parameters are deliberately explicit:

• claim_slot and provision_vm require an exact assetId, active linearIssue, and unique operationId; apply also requires matching confirmAssetId.
• For an interrupted remote-dev-v1 provision, only canonical provision_vm can resume the same alma97-cloud operation with recoverIncomplete: true; repairOperationId optionally binds the completed vm_repair receipt. Keep the original asset, host, name, provision operation, VMID, cloud image, and reviewed bootstrap manifest. Apply requires their exact confirmations, including confirmBootstrapManifestSha256 and confirmExistingSmbiosUuid. Facade validation rejects unsupported combinations before dispatch, and the canonical CLI rechecks retained intent, claim, and live identity before mutation. This does not imply development runtime readiness; verify that separately through lab_dev. auto_provision_vm, finalization, alternate sources, and caller-supplied fingerprints remain unsupported for this recovery path.
• For an already-installed alma97-cloud VM with an incomplete original provision operation, provision_vm and auto_provision_vm accept finalizeInstalled: true. This delegates --finalize-installed to the canonical CLI for exact pending-journal, claim, UUID, guest-health, and release readback; it does not rerun the installer. Use the original operation and reviewed VM inputs, not a replacement identity. It cannot be combined with recoverIncomplete, a template/ISO source, or legacy mode. Apply also requires confirmExistingSmbiosUuid from the reviewed plan. An uncertain timeout must be reconciled against the exact operation receipt before retry.
• release_slot requires the exact leaseId and an artifactReceipt. In canonical mode, pass destroyVm: true with a non-empty reason and apply: true to perform the single guarded release-and-destroy operation for an eligible disposable VM; the facade forwards that intent to vm release --destroy-vm --reason so runtime identity is retained through deletion.
• appliance_candidate_retire is a separate, dry-run-default cleanup path for an operator-approved failed disposable candidate. It requires the exact asset, lease, failed candidate operation and immutable failure receipt; canonical policy independently matches provisioning, live identity and shared quarantine. Apply additionally confirms the asset, VMID and SMBIOS UUID. Use separate unused plan/apply receipt paths. Failure records remain intact; an expired guest is never renewed or reused, and verified absence precedes ownership release. There is no force, host/IP selector or legacy fallback.
• create_snapshot and rollback_snapshot take a claim-scoped path when assetId is supplied (UAT-1767), forwarding to vm snapshot / vm rollback. They require linearIssue, leaseId, operatorClaimUid, operationId and snapshotName, are plan-only until apply: true, and apply additionally requires confirmAssetId, confirmSmbiosUuid, confirmSnapshotName and confirmPlanSha copied from the plan. Snapshots are disk-only; rollback accepts only the most recent snapshot this tooling created for the claim. The canonical CLI independently refuses fixtures, unclaimed guests and foreign claims. There is no VMID, name, host or RAM-state selector, no snapshot deletion, and no legacy fallback: a request without assetId keeps the legacy refusal. list_snapshots takes the same path (assetId, linearIssue, leaseId, operatorClaimUid) to vm snapshots and is advertised in canonical mode for that purpose only. Every refusal of these three tools tells the caller to pass assetId. Requires a lab-inventory bundle that includes ct-meta 067fa4cc or later (the CLI timeout-range help fix needs a later one).
• purge_slots delegates only to claims gc; age, force, orphan, and VM-name shortcuts are rejected.

This bridge change is delivered only by a reviewed @calltelemetry/ct-lab-mcp package release consumed through the host's approved version pin. Source edits do not mutate an installed MCP package; after release, advance the canonical launcher pin and restart/reconnect the host transport through the normal consumer update workflow.

• list_expired_vms classifies live VMs as legal, expiring, expired, dead leftover, or protected. It never deletes; terminal Linear-ticket VMs are classified (dead_running / dead_stopped) but ticket terminality alone never makes a claim delete-ready (ADR 0400 lease precedence).

• cleanup_expired_vms is read-only when dryRun: true (or apply: false): omit confirmAssets for the full classification or provide an exact comma-separated scope to preview selected and excluded rows. Mutation requires a non-empty confirmAssets list and acts only on selected delete-ready rows; a row is delete-ready only when its exact claim expiry has passed (or it is a fully unclaimed leftover), so a running expired guest receives an exact-identity graceful shutdown only after it is selected. Invalid, duplicate, unknown, stale, non-delete-ready, ambiguous, or identity-drift keys fail before the first mutation. deleteTerminalIssues: false remains available to exclude the terminal-ticket policy. Protected fixtures and non-terminal running guests are never candidates.

• vm_renew and vm_transfer preserve the exact asset and lease identity. Running-guest renewal and expired-running transfer are accepted only when the canonical CLI proves their narrow recovery conditions.

• vm_action maps only to canonical asset-bound status, guest reboot, power, and evidence-first delete operations.

• host_maintenance_shutdown is separate from vm_action. It is dry-run by default and routes only through canonical lab-inventory host maintenance-shutdown; the public MCP never sends provider host commands directly. Apply requires confirmHostAssetId and the latest planDigest. A partial transaction remains fail-closed for manual reconciliation and is never blindly redispatched with the same operation ID.

• proxmox_quick_provision wraps proxmox-quick-provision.sh. Pass one size object; hostAssetId defaults to auto, filters live Proxmox capacity, and randomly selects among eligible hosts. Production randomness is not seeded; placementSeed exists for reproducible tests. Apply requires the reviewed explicit hostAssetId, vmid, and operationId. 0.8.8-stable requires baseVersion plus webImage; install uses the GCS base. Never VMID 182.

• proxmox_import_qcow wraps canonical lab-inventory vm provision --source published-qcow (the vm import-qcow apply is retired); osVersion is required (plan by default; apply needs the exact asset, vm prefix, VMID, and confirmImageSha256). imageUrl is exactly one of two identity URLs on the DigitalOcean Spaces release host calltelemetry-releases-nyc3.nyc3.digitaloceanspaces.com (https only; no query string, fragment, userinfo, or port; the retired GCS host is refused, ADR 0735):

  • published: https://<host>/ovas/ct-<version>.qcow2
  • private CI candidate for an unpublished RC (ADR 0727): https://<host>/candidates/<version>/<run_id>/ct-<version>.qcow2, where <version> has four numeric parts with an optional -rc<N> and is identical in both path positions, and <run_id> is 6-20 digits.

  Candidates are private and digest-bound: the SHA-256 binding (imageSha256 + confirmImageSha256) is mandatory for both shapes and the public publish gate is unchanged. The tool never accepts a presigned URL; the lab operator supplies any download credential to the canonical CLI through its own environment.

• Expired leases with attached VMs remain occupied until the deletion transport succeeds. Protected or failed deletions retain their exact lease and VM identity, preventing phantom free capacity.

• pin_pool_slots wraps pin-pool-slots.sh by tag+host. Default is plan-only; apply: true pins calltelemetry/web:<line>-stable-<12sha> over SSH :2222.

0.8.8 is SHA-channel. Do not invent latest-0.8.8-stable.txt. Point CUBE XCC at http://<guest-ip>:80/api/xcc/<cube_id>.

MCP Tools Reference

The server registers 7 comprehensive tools conforming to the MCP JSON Schema specification:

1. list_hosts

Query capacity, health, CPU/RAM/Disk metrics, and resource-derived 8GB unit estimates across Proxmox VE (proxmox-lab, proxmox-mini) and VMware ESXi (esxi-intel-192-168-123-176).

{
  "host": "all",              // "proxmox-lab" | "proxmox-mini" | "esxi-intel-192-168-123-176" | "all"
  "hypervisorType": "all",    // "proxmox" | "esxi" | "all"
  "refresh": false            // boolean: force live hypervisor API poll
}

2. list_inventory

Query unified fleet inventory across Proxmox VE and VMware ESXi VMs with power status, IP addresses, VLAN, protected flags, release version, and Linear lease tags.

{
  "hypervisor": "all",        // "proxmox" | "esxi" | "all"
  "host": "proxmox-mini",     // optional host filter
  "status": "all",            // "all" | "running" | "stopped" | "claimed" | "disposable"
  "includeTemplates": false,  // boolean (default: false)
  "filter": "0.8.7-stable",   // text search across name, ID, IP, Linear issue, tags
  "tags": ["uat", "disposable"]
}

3. claim_slot

Lease an 8GB appliance slot on a specified host or randomly across eligible Proxmox peers, with lease TTL and Linear issue tracking. Use placementSeed only in deterministic tests.

{
  "tag": "CTUAT-693",         // required string: issue tag or session identifier
  "linearIssue": "CTUAT-693", // Linear issue key
  "host": "auto",             // "proxmox-lab" | "proxmox-mini" | "esxi-intel-192-168-123-176" | "auto"
  "hypervisorType": "any",    // "proxmox" | "esxi" | "any"
  "durationMinutes": 60,      // integer (1 - 1440 minutes / 24 hours, default: 60)
  "purpose": "UAT Testing"    // optional description
}

Claims can exceed the former 2/4/6 counts whenever fresh physical resources permit. Pass slotSizeMb and diskGb for a nonstandard lease size. Provisioning forwards the actual requested memory, vCPU, storage pool and disk requirements, then rechecks resources before public claim creation or cloning. Existing claims keep their identity, issue and TTL; the change does not trigger fleet cleanup.

4. release_slot

Release an active 8GB slot claim by claimId, tag, requester, or linearIssue, with optional automatic destruction of attached test VMs.

Canonical mode requires the complete identity/evidence fields below. Review the request with apply: false; then repeat the same request with apply: true and an exact confirmAssetId to execute the guarded release-and-destroy operation.

{
  "assetId": "ct-087-rc450-linear-uat-1695-appliance-uat",
  "linearIssue": "UAT-1695",
  "leaseId": "lease-f0c0e43e444cf7e95b121fd6",
  "operationId": "uat-1695-release-destroy-001",
  "artifactReceipt": "runs/UAT-1695/evidence.json",
  "destroyVm": true,
  "reason": "failed disposable UAT candidate",
  "apply": true,
  "confirmAssetId": "ct-087-rc450-linear-uat-1695-appliance-uat",
  "receiptFile": "runs/UAT-1695/release-destroy-receipt.json",
  "format": "json"
}

5. provision_vm

Provision a new disposable Call Telemetry appliance VM on Proxmox VE or VMware ESXi from a base template with IP/VLAN networking and auto-attached 8GB slot lease.

{
  "sourceVmIdOrName": 9000,                  // template VMID or template name
  "newVmName": "ct-appliance-uat-105",       // name for the new VM
  "targetHost": "proxmox-mini",              // optional host (auto-placed if omitted)
  "vlan": 124,                               // optional VLAN tag
  "ipAddress": "192.168.124.105",            // optional static IP
  "linearIssue": "CTUAT-693",                // Linear issue tag
  "startOnDeploy": true                      // power on immediately after cloning
}

Guest disks are thin-only: storage must name an lvmthin pool (for example local-lvm). A thick or undeterminable pool is refused with THICK_STORAGE_REFUSED before anything is created.

Size-driven Proxmox fleet plan

{
  "linearIssue": "REL-472",
  "version": "0.8.7-rc438",
  "purpose": "fleet-puppet",
  "hostAssetId": "auto",
  "size": {
    "memoryMb": 12288,
    "vcpus": 6,
    "diskGb": 160
  },
  "apply": false
}

The plan returns placement.selected_host_asset_id, the complete eligible host set, target.vmid, and operation_id. Copy those exact identities into apply; do not leave hostAssetId as auto for a mutating call.

6. vm_action

Execute power lifecycle operations across Proxmox and ESXi with non-bypassable guardrails protecting core infrastructure.

{
  "vmIdOrName": "ct-appliance-uat-105",     // VMID or VM name/path
  "host": "proxmox-mini",                   // target host name
  "action": "start",                        // "start" | "stop" | "reboot" | "shutdown" | "reset" | "status" | "destroy"
  "force": false                            // boolean: force immediate action (guardrails cannot be bypassed)
}

Destructive actions on protected VMs (CUCM, CUBE, base templates, K8s) are strictly denied with PROTECTED_VM_ACTION_DENIED.

host_maintenance_shutdown

First request the current plan. This performs live read-only discovery and returns plan_digest plus the exact required host confirmation:

{
  "hostAssetId": "proxmox-minis-lab",
  "reason": "Replace failed cooling fan",
  "operationId": "rel-476-maintenance-20260831-01"
}

After reviewing an eligible plan, apply the same immutable request:

{
  "hostAssetId": "proxmox-minis-lab",
  "reason": "Replace failed cooling fan",
  "operationId": "rel-476-maintenance-20260831-01",
  "apply": true,
  "confirmHostAssetId": "proxmox-minis-lab",
  "planDigest": "<64-character digest from the dry run>"
}

Proxmox apply verifies a standalone, fully drained node before requesting the supported node shutdown. ESXi apply verifies a fully drained host, enters maintenance mode without evacuation, then requests shutdown with force=false. Both require physical power-on for recovery; ESXi remains in maintenance mode until an operator verifies the host and exits it. Start CUCM publishers first, verify replication, then start subscribers.

7. appliance_exec

Execute mediated Call Telemetry appliance CLI operations over secure SSH on Port 2222 with output sanitization and secret redaction.

{
  "targetIp": "192.168.124.105",             // appliance IP or hostname
  "port": 2222,                              // SSH port (default: 2222)
  "action": "db_vacuum",                     // "status" | "health" | "diag" | "db_vacuum" | "migrate" | "docker" | "logs" | "version"
  "args": {
    "table": "cdr_records",
    "vacuumFull": true,
    "vacuumAnalyze": true
  },
  "timeoutMs": 30000
}

Client Configuration

Credentials are inherited from the process environment. Do not put a Doppler token, Proxmox credential, ESXi password, or appliance password in checked-in MCP JSON. A GitHub token authorizes repository fetches only; it is not a Doppler token and cannot authenticate a hypervisor.

For stdio clients, credential failure is diagnostic-first: MCP initialization, tool discovery, and credential_status remain available. When the canonical bridge is both required and configured, list_hosts and list_inventory resolve target credentials through that bridge even if legacy aggregate credential resolution failed. Canonical list_storage retains its existing bridge-owned credential check. These exceptions do not mark legacy credentials ready or permit other operations: those still return CREDENTIALS_UNAVAILABLE. The canonical-only vm_renew and vm_transfer claim tools likewise delegate credential resolution to the required, configured bridge. They remain dry-run by default and require exact asset/lease identity and explicit apply confirmation for changes. An unavailable bridge or failed target validation remains an error; this routing never substitutes mock credentials or claims a host is ready. All six Kubernetes ApplianceClaim lifecycle tools use their independent control plane when legacy fleet credentials are unavailable. Each request first verifies its exact Kubernetes verb/resource/namespace/name permission and then executes the real handler. A denied or unavailable Kubernetes permission cannot be bypassed by the lifecycle exemption. Pure create dry-run plans validate without requiring a control-plane connection. Unrelated hypervisor tools retain their credential gates. Canonical discovery still validates provider responses, reports partial-result warnings, and fails when the requested scope is unavailable. The server never invents a mock transport. Repair the direct provider credentials or the config-scoped DOPPLER_TOKEN, then restart the client. SSE startup remains fail-closed.

Governed Call Telemetry workspace

Use the ct-lab plugin and its launch-ct-lab-mcp.sh entrypoint. Cursor discovers the plugin-owned .mcp.json; do not add another ct-lab entry to the workspace .cursor/mcp.json, because that would start two servers. The launcher pins the reviewed package version and inherits credentials from the agent environment without invoking an interactive login.

Claude Desktop Configuration

For a standalone, non-plugin installation, add the server to claude_desktop_config.json and start Claude Desktop from an environment that already contains either direct lab credentials or a config-scoped DOPPLER_TOKEN:

{
  "mcpServers": {
    "ct-lab": {
      "command": "npx",
      "args": ["-y", "@calltelemetry/[email protected]"]
    }
  }
}

Cursor Configuration

For a standalone repository that does not install the Call Telemetry plugin, the equivalent project configuration is below. Start Cursor from a terminal or supervisor that already exports DOPPLER_TOKEN (or the complete direct lab credential set) so the npx child inherits it. If a desktop launcher does not inherit that environment, configure the environment in the operating-system launcher or supervisor—not in checked-in .cursor/mcp.json.

{
  "mcpServers": {
    "ct-lab": {
      "command": "npx",
      "args": ["-y", "@calltelemetry/[email protected]"]
    }
  }
}

Programmatic TypeScript Usage

import { createMcpServer, startStdioServer, SlotManager, MultiHypervisorManager } from "@calltelemetry/ct-lab-mcp";

const server = createMcpServer({
  name: "custom-lab-mcp",
  version: "1.0.0",
});

await startStdioServer(server);

Development & Verification

# Typecheck TypeScript source
npm run typecheck

# Run full Vitest test suite
npm test

# Build production distribution to dist/
npm run build

# Verify npm bundle package
npm pack --dry-run

Release flow

A package-affecting merge to main is released by the Publish to NPM workflow: it bumps the patch version when the current one is already released (a published version, or an existing v<version> tag), publishes to npm, tags v<version>, creates the GitHub release and then dispatches Docker Build and Publish on that tag. The image is therefore tagged with the same version as the npm release and the git tag, and the Docker workflow refuses a tag that differs from package.json. Container-only changes (Dockerfile, .dockerignore) build from the main push and get a v<version>-build.<sha8> tag instead. The image roll PR in ct-infrastructure is opened by the release App from that build.

The image installs with npm ci and must see the repository .npmrc (legacy-peer-deps=true, which resolves the agents zod peer): the Dockerfile copies it next to package*.json, and test/dockerfile-npm-install.test.ts guards that (REL-1723).

Appliance licensing

apply_license does not mint tokens. Appliances cannot sign their own (Cdrcisco.Token.trial/4 is compiled out of :onprem/:prod builds — cisco-cdr #5070 / #5071, API-3202), and ct-lab-mcp no longer mints either: the old local path carried the same shared HS256 secret and emitted a claim shape (tier, iss: calltelemetry_lab_mcp) that the appliance rejected, so the box quietly stayed Community while the tool reported success.

Licenses now come from ct-dashboard, the one issuing authority:

POST {DASHBOARD_URL}/api/crm/licenses/lab     Authorization: Bearer $DASHBOARD_API_KEY

Set DASHBOARD_API_KEY (Doppler ai-workspace/prd) before using apply_license or factory_reset_appliance. Without it the tool fails immediately and says so, rather than applying a token the appliance will reject.

The license is linked to the VM

Every issued key records the box it was issued for in its CRM notes:

[lab] ct-lab-mcp ip=192.168.124.229 vm=ct-uat-229 vmid=129 host=proxmox-mini claim=UAT-1234

Pass vmName, vmId, host, linearIssue, and purpose to apply_license — the values claim_slot / auto_provision_vm already returned. targetIp is always recorded.

That makes the link traversable in both directions:

• license → VM: GET /api/crm/licenses?org_id=<lab org> and read notes (parseVmLicenseNote in src/utils/dashboard-license.ts parses it back into fields).
• VM → license: the apply_license result carries licenseInfo.licenseId, keyPrefix, and expiresAt, and the same ids go out on the appliance.license.applied telemetry event.

Lab keys are tagged sku: "LAB" on the dashboard side, so they never mix with revenue licenses. Tiers are Demo (default), Beta, and Premium; commercial tiers are not issuable as lab keys. Community clears the license instead of issuing one.

License

MIT © 2026 Call Telemetry / Contributors

See resource admission and runtime activation for compatibility fields and release verification.