NexusOS Semantic

Universal Semantic Accessibility & Kinetic Substrate for AI Agents across Web, Native Desktop, and Mobile.
Stop clicking pixels. Give artificial intelligence direct semantic and accessibility-tree control via standard Model Context Protocol (MCP).
Developed and maintained by NexusOS Systems.

Developer Launch status. The npm package agent-web-graph is published to the public npm registry — install it with npm i -g agent-web-graph or run npx -y agent-web-graph. Both resolve whatever version the registry currently publishes (check with npm view agent-web-graph); this source tree and package declare version 0.3.2, whose shipped changes are listed in CHANGELOG.md → [0.3.2]. Release status (tags, GitHub Releases, registry versions) is authoritative from npm and from the repository's release history — never from this README. The GitHub source repository remains private: cloning the source and downloading the GitHub Release binaries require authorized access. The canonical public source location (github.com/nexusos-systems/nexusos-semantic) does not exist yet, so the dynamic CI/Pages badges were removed rather than left rendering "no status" (branding rules ratified in ED-07; docs/BRANDING_AND_NAMING_TAXONOMY.md was moved out of this repository in the 2026-09-20 release-boundary commit e2b0e1d — do not depend on it). Quality is enforced by branch protection requiring the Constitutional gates check: typecheck, full test suite, BiDi substrate probe, production build. Quality gates and release artifacts are verified in CI before every release (see docs/LIVE_STATE_AUDIT.md for the audit trail).

💡 Why NexusOS Semantic?

Vision-based agents fail in real-world automation:

• Context Bloat: Transmitting high-resolution screenshots wastes thousands of tokens per step.
• Coordinate Drift: Responsive reflows, high-DPI scaling, and animations cause vision models to hallucinate click coordinates.
• Silent Failures: Agents cannot verify if a button was actually clickable, disabled, or occluded.

NexusOS Semantic solves this by normalizing any interface into a deterministic, queryable Accessibility Graph (AxTreeNode):

• Compact semantic context: reduces screenshot-heavy context by exposing structured accessibility data — agents consume compact ARIA hierarchies instead of megabytes of vision pixels.
• Semantic targeting: elements are selected by exact semantic identifiers, roles (button, combobox, dialog), and computed names, reducing coordinate ambiguity rather than promising error-free results.
• Multi-Substrate Kinematics: One canonical schema spans Web browsers (WebDriver BiDi / CDP), Native Desktop windows (Windows UIA / macOS AX), and Mobile devices (Android ADB / iOS WDA).

   ┌────────────────────────────────────────────────────────┐
   │    AI Agent (Claude Desktop, Cursor, Claude Code)      │
   └───────────────────────────┬────────────────────────────┘
                               │ Standard Model Context Protocol (stdio / JSON-RPC)
                               ▼
   ┌────────────────────────────────────────────────────────┐
   │                   NEXUSOS SEMANTIC                     │
   │  • substrate_info— Substrate provenance & graph hash   │
   │  • graph_query   — Semantic element & state search     │
   │  • graph_path    — Graph navigation & shortest-path    │
   │  • graph_tool    — Discover executable capabilities    │
   │  • graph_act     — Execution decision (no execution)   │
   │  • graph_invoke  — Deterministic kinetic execution     │
   │  • graph_explain — Accessibility & neighborhood walk   │
   │  (+ visual / chrome / desktop / mobile tool groups)    │
   ├───────────────────────────┬────────────────────────────┤
   │                           │                            │
   ▼                           ▼                            ▼
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│  Web Substrate   │  │Desktop Substrate │  │ Mobile Substrate │
│ WebDriver BiDi   │  │   Windows UIA    │  │   Android ADB    │
│  & Chrome CDP    │  │   macOS AX API   │  │   iOS WDA API    │
└──────────────────┘  └──────────────────┘  └──────────────────┘

⚡ Quickstart

Distribution status (Developer Launch — published): the npm package agent-web-graph is published to the public npm registry under an immutable tag per release. Install it with npm i -g agent-web-graph or run npx -y agent-web-graph — no repository access required; the registry serves its current published version. Still access-gated: this server is not registered on Smithery (smithery.ai 404), and the GitHub source repository is private, so cloning source and downloading the GitHub Release SEA binaries require authorized access. The npx-based configs below run from the public npm package today.

1. Install from npm (works today for everyone)

npm i -g agent-web-graph        # global CLI: `nexus` / `awg`
# or, no install:
npx -y agent-web-graph doctor

2. Build from source (authorized repository users; local/offline)

git clone <your-accessible-origin-url> agent-web-graph
cd agent-web-graph
pnpm install && pnpm build

# register the compiled CLI with an MCP client, e.g. Claude Code:
claude mcp add nexus -- node /path/to/agent-web-graph/dist/src/cli/awg.js serve --graph /path/to/substrate-dir --no-bidi

serve requires an explicit --graph <dir> and refuses to start without one — there is no default substrate. The quickstart uses --no-bidi so a static graph loads without a browser driver; without --no-bidi, serve attempts to attach a live BiDi session and exits if no geckodriver is reachable (drop --no-bidi only when geckodriver is running).

3. Claude Desktop Integration

Add to your claude_desktop_config.json:

• Windows: %APPDATA%\Claude\claude_desktop_config.json
• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "nexus": {
      "command": "npx",
      "args": ["-y", "agent-web-graph", "serve", "--graph", "/path/to/substrate-dir", "--desktop", "--mobile"]
    }
  }
}

4. Cursor IDE

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "nexus": {
      "command": "npx",
      "args": ["-y", "agent-web-graph", "serve", "--graph", "/path/to/substrate-dir"]
    }
  }
}

5. Claude Code CLI

claude mcp add nexus -- npx -y agent-web-graph serve --graph /path/to/substrate-dir

Still gated: npx -y @smithery/cli run agent-web-graph requires a registered Smithery server (smithery.ai 404) — Smithery is not yet registered; smithery.yaml is committed in-repo for when it is. The GitHub Release SEA binaries live in a private repository and download only for authorized users; the public, unauthenticated path is the npm package above.

🛠️ MCP Tool Capabilities

The server's tool surface is not one number — it has five distinct figures (all counts locked by test/server/tool-surface-drift.test.ts, which proves each figure with runtime registration tests): bare server baseline: 18 (a programmatic server built with no options, or serve --graph <dir> --no-bidi); CLI serve with BiDi: 19 (the normal awg serve --graph <dir> attaches a live BiDi session by default, adding graph_live_read); CLI serve --no-bidi: 18; CLI maximum: 30 — 18 + graph_live_read (+1, BiDi attached) + async crawl jobs (+3 via --allow-crawl) + desktop UIA (+5 via --desktop) + mobile ADB (+3 via --mobile), with Chrome/CDP (+6) unreachable because there is no serve flag for it; programmatic maximum: 36 — all of the above plus Chrome/CDP (+6) via the library/server chrome option. Do not describe these simply as "the default": the bare baseline is 18, but an ordinary CLI serve session exposes 19. The 18-tool baseline is 6 core graph tools (graph_query, graph_path, graph_tool, graph_act, graph_invoke, graph_explain) + substrate_info + graph_diagnostics + 10 visual/token tools. Crawl tools are not part of any default surface — producing a substrate is a CLI/library job (nexus crawl); the server serves an already-built graph directory. graph_act is decision-only; graph_invoke is the execute path.

Opt-in substrate production (R5): with nexus serve --graph <dir> --allow-crawl, three more tools are registered — crawl_start, crawl_status, crawl_cancel — so an agent can trigger a crawl as a background job and receive a handle {jobId, outputDir, graphHash, producer, state, ...}. The substrate is published atomically (a failed or cancelled crawl never leaves a loadable partial directory), the graphHash on the completed handle is computed from the published graph.json bytes, and crawl_start is gated by the enforcement gateway as the crawl operation (impact modify).

| Tool | Category | Description | | --- | --- | --- | | substrate_info | Provenance | rootUrl, crawl timestamps, computed graphHash (sha256 of the served graph.json), build version. Assert on this before acting. | | graph_query | Semantic Search | Query nodes by ARIA role (button, combobox, searchbox), name, or coordinates. | | graph_path | Navigation | Traverse hierarchical tree relationships (children, parent, descendants, shortest-path). | | graph_tool | Discovery | Enumerate actionable tools and declared capabilities exposed by the active interface. | | graph_act | Decision | Decision-only — returns what would be executed (capability-tier gate); does not execute. | | graph_invoke | Kinematics | Dispatch verified synthetic clicks and text fills with safety validation. (The executor implements click + type only — there is no keyboard/hotkey path; with --no-bidi the tool always reports "no BiDi session attached".) | | graph_live_read | Live Web State | With BiDi attached, optionally navigate and safely snapshot document state plus one re-resolved CSS-selected element. | | graph_explain | Introspection | Retrieve accessibility context, parent bounding boxes, and neighborhood relations. | | graph_diagnostics | Health | Graph counts, health score, and substrate enablement (BiDi / desktop / mobile). (The availability strings come from the serve flags, not a probe — "online" means "enabled", not "verified reachable".) | | get_visual … export_dtcg_tokens | Visual (10 tools) | Visual intelligence: viewport diffs, layout mutations, spatial neighbors, occlusion, visual regression, design-token lint/export. | | chrome_targets … chrome_launch | Chrome/CDP (6 tools) | Attach to and read/act on Chrome via CDP, including cookie access. Reachable only programmatically (startMcpServer(graph, {chrome:true}) / buildMcpServer) — there is no --chrome CLI flag, so these are absent from nexus serve (CLI max 30 tools, programmatic max 36). | | desktop_list_windows | Desktop Substrate | Enumerate open top-level OS windows, process names, and window handles. Windows only. | | desktop_scrape_window | Desktop Substrate | Scrape native Windows UIA elements into canonical AxTreeNode. (macOS AX is not wired to this tool — see docs/CAPABILITY_AUDIT.md §3.) | | desktop_read_text | Desktop Substrate | Read text from a freshly resolved Windows Document/Edit element via UI Automation. | | desktop_replace_text | Desktop Substrate | Replace a Windows editor's full text and return its actual UI Automation read-back. | | desktop_kinetic_action | Desktop Substrate | Dispatch native OS clicks and keyboard shortcuts to target windows. (Windows UIA bridge.) | | mobile_list_devices | Mobile Substrate | List connected Android ADB devices/emulators plus the iOS WebDriverAgent endpoint status — probe-first: online: true only when a real WDA /status answers (WDA drives the single attached device/simulator; it does not enumerate). (Registered with --mobile; substrates you did not configure report configured: false and are never constructed.) | | mobile_scrape_device | Mobile Substrate | Dump and parse the active screen hierarchy into standard graph nodes. Android by default; pass deviceId: "ios" for the configured WebDriverAgent substrate (probe-first — an explicit "iOS substrate unavailable" error when no endpoint answers). | | mobile_touch_action | Mobile Substrate | Dispatch kinetic taps, swipes, text typing, and key events (Back, Home). Android whenever an Android substrate is configured; iOS when iOS is the only configured substrate ({ mobile: { ios: ... } }), probe-first — nothing is dispatched unless a real WDA endpoint answers. |

💻 CLI & Standalone Executable (local build; no Node.js needed at runtime)

NexusOS Semantic can be compiled into a zero-dependency Single Executable Application (SEA):

Availability truth (Developer Launch): GitHub Releases publish standalone SEA binaries (nexus/awg for linux-x64, windows-x64, macos-arm64) with per-artifact SHA-256 checksums, a CycloneDX SBOM, and a release manifest binding version, tag, candidate SHA, artifact hashes and SBOM hash; exact-SHA two-build reproducibility is proven in CI. GitHub artifact attestations and npm --provenance are unavailable for this repository's private-source configuration (the production release manifest records githubArtifactAttestation: unavailable_platform_private_repository and npmProvenance: unavailable_platform_private_repository); the launch release v0.3.0 (2026-09-26) did carry GitHub attestations, the current release does not. Because the source repository is private, those Release downloads require authorized access; the public, unauthenticated install path is the npm package (npm i -g agent-web-graph). The SEA build below reproduces those binaries locally for authorized repository users.

Known defect in the published v0.3.0 Windows binaries (fixed in 0.3.2 and later): desktop --list / --scrape failed from the released binary because uia-bridge.ps1 was neither shipped beside it nor embedded. The bridge is now a SEA embedded asset materialized at runtime, doctor gained a "Desktop UIA bridge script" check, and wave2:smoke-binary exercises desktop --list on Windows. Re-verified from a local pnpm build:sea on 2026-09-28: dist/bin/nexus.exe desktop --list → exit 0 with real windows, doctor → bridge script OK. The v0.3.0 Windows binaries still carry the old behaviour (doctor reports OK, desktop --list fails); install the currently published version (npm i -g agent-web-graph) for the corrected desktop route.

# Clone and build.
# NOTE: the repository is currently private. Clone from the origin you have
# been granted access to; the canonical public URL will be
# https://github.com/nexusos-systems/nexusos-semantic.git once the
# organisation exists (branding rules: ED-07).
git clone <your-accessible-origin-url> agent-web-graph
cd agent-web-graph
pnpm install
pnpm build:sea

# Standalone binary is ready in dist/bin/nexus (.exe on Windows)
./dist/bin/nexus doctor

Key CLI Commands

Note: The CLI responds to both nexus and awg interchangeably.

# 1. Environment and substrate diagnostics
nexus doctor

# 2. Crawl a web app YOU own and run (default posture is audit/read-only —
#    crawl is a modify operation and is denied unless you raise the posture
#    and authorize the exact target host; example targets a local app on :3000)
nexus crawl "http://localhost:3000" --out ./my-crawl --max-pages 10 --posture autopilot --authorize-targets localhost:3000

# 3. Inspect graph health, broken states, and ARIA coverage
nexus inspect --graph ./my-crawl --diagnostics

# 4. Query graph without launching a browser
nexus query graph_query '{"select":"ax-node","where":{"role":"button"}}' --graph ./my-crawl

# 5. Start the MCP server over stdio
nexus serve --graph ./my-crawl --desktop --mobile

# 6. Diff two graph snapshots for visual regressions
#    (layout shift, dimension changes, style drift, occlusion, token detachment)
nexus diff ./baseline-crawl ./candidate-crawl --page "page:/" --json

# 7. Inspect design tokens: lint token drift (styling debt)
nexus tokens --graph ./my-crawl --lint

# 8. Export a standard W3C DTCG design-token bundle
nexus tokens --graph ./my-crawl --export-dtcg ./tokens.dto.json

🧪 Validation Evidence — the Evidence Ledger

Every claim that NexusOS Semantic works against real applications is backed by machine-recorded evidence in verification/:

• verification/LEDGER.md — the summary table an auditor reads first: per-target verdicts, page/AX-node counts, health scores, and extractor diagnostics, regenerated deterministically by the runner from the committed per-target result files.
• verification/targets/ — the ratified target set, each with its recorded authorization basis: an owned synthetic app, self-hosted real OSS (OWASP Juice Shop, httpbin), and a hard-budgeted permissioned-public set (example.com, W3C ARIA APG).
• verification/results/ — the full per-target run record, including the robots.txt authorization snapshot, container image digest, threshold checks, and the capability-battery transcript.

Run it yourself:

node scripts/run-ledger.mjs --list                 # the ratified target table
node scripts/run-ledger.mjs --target demo-saas     # one target, end-to-end

There is no scheduled (nightly) ledger workflow in this repository — .github/workflows/nightly-ledger.yml was described in ED-09/ED-10 but was never committed (verified 2026-09-23). However, the ledger IS executed by CI: .github/workflows/wave3-certification.yml runs node scripts/run-ledger.mjs --strict on every pull_request (Job "Evidence Ledger (strict, Node 24 + live BiDi + Docker)") — a previous revision of this paragraph said the battery "must be run manually", which understated the actual coverage (full live-state audit, 2026-09-27). Outside CI, run it with node scripts/run-ledger.mjs. Hand-editing generated evidence is prohibited — the policy, including why bug bounty is not the test strategy and how design-partner testing is authorized, is binding in docs/EXECUTIVE_DECISIONS.md (ED-09, ED-10).

🌐 Interactive Demo & Visual Explorer

The interactive visual explorer ships in-repository and runs locally. site/ is configured to deploy through .github/workflows/deploy-pages.yml on a push to main touching site/**; the actual post-merge deployment and the resulting site availability/visibility must be verified after merge — repository privacy alone does not establish private Pages access. The canonical public URL under nexusos-systems does not exist yet:

pnpm site          # serves site/ locally; open the printed URL

The explorer provides:

• Live visual graph query explorer
• Element coordinate vs ARIA role comparison
• Ready-to-copy MCP client configurations

🔒 Security & Safe Execution Tiers

Every registered MCP tool and the side-effecting CLI commands route through the enforcement gateway + capability-tier gate (the read/execute/confirm decision itself lives in src/graph/security.ts):

• READ: Passive reads, accessibility scraping, and graph walks (allowed at the default AUDIT posture).
• EXECUTE: Reversible interactions like form field entry or tab switching.
• CONFIRM: Sensitive operations requiring explicit human approval.

Scope of that guarantee (corrected by the full live-state audit, 2026-09-27): all 36 MCP tool registrations are wrapped by withToolEnforcement, the TCP transport passes guardRpcRequest, the graph_invoke executor is gated in src/server/invoke.ts, and the CLI gates crawl, demo, mobile scrape and session-capture. The remaining CLI commands (query, inspect, doctor, view, diff, tokens, desktop, eval) do not traverse the gateway — they are read-only or write only under the caller's own working directory — so the claim "every operation enforces" was overstated: those commands produce no attempt/audit record.

See SECURITY.md for full vulnerability reporting guidelines and sandboxing rules.

📜 Open Source Governance & License

• License: Distributed under the Apache License 2.0.
• Branding & Taxonomy: Governed by ED-07 (docs/BRANDING_AND_NAMING_TAXONOMY.md was moved out of this repository in e2b0e1d).
• Contributing: Please read CONTRIBUTING.md before submitting pull requests.
• Constitution: Governed by the principles outlined in PROJECT_CONSTITUTION.md.