@brookesia/app-dev-plugin

Agent plugin for ESP-Brookesia app development. It connects to Cursor, Claude Code, Codex, and other Agent IDEs/CLIs via MCP and provides JSON UI design, CLI build/simulation, Figma export, screenshot-to-UI, and auto-test workflows.

Chinese README: README_CN.md

| npm package | Purpose | | -------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | @brookesia/app-dev-plugin | This plugin (MCP + Skills + installer) | | esp-brookesia-toolkit | brookesia CLI (build / simulate / pack; installed separately) |

Source tree: esp-brookesia-toolkit/plugins/brookesia-app-dev

Features

| Area | Description | | --------------------- | -------------------------------------------------------------------------------------- | | App projects | Init, build, pack, verify, and release .bpk packages | | WASM simulator | Run JSON UI in the browser with screenshot and reload support | | JSON UI | Schema validation, layout inspection, interactive widget enrichment | | Design import | Figma → JSON UI export sessions; reference-image icon extraction / primitive rendering | | Visual regression | Pixel-compare simulator screenshots against reference images | | Device services | MCP Resources exposing the @brookesia/service API catalog | | Agent skills | Guided workflows for setup, development, design, screenshot-to-UI, and auto-test |

The plugin is wired through MCP (Model Context Protocol). It is not a standalone extension in official IDE plugin marketplaces. Look for brookesia-app-dev under Settings → MCP (or /mcp, codex mcp list).

Requirements

• Node.js ≥ 20 with npm and npx on PATH (Windows, Linux, macOS)
• esp-brookesia-toolkit — brookesia CLI (installer prompts interactively); ≥ 0.1.13 recommended for canvas screenshots and visual parity
• Playwright Chromium — WASM mode=canvas screenshots / pixel compare (installer detects cache; ~150MB)
• curl — local simulator HTTP probes (/api/health, /api/screenshot); optional apt install on Linux
• Linux optional: playwright install-deps — Chromium system libraries if screenshots fail
• brookesia-usb-cli — Python 3.9+ host tool for installing .bpk to real hardware over USB Serial/JTAG (brookesia_deploy). Installer detects and offers it; manual: python3 -m pip install --user brookesia-usb-cli

The installer runs a preflight check (Node / npm / OS) then offers optional dependencies after MCP configuration.

Not installed by brookesia-plugin: npm canvas package (unused); ajv / pixelmatch / pngjs ship with the plugin; img2lv / playwright npm come with esp-brookesia-toolkit; @brookesia/service is per app project; Figma MCP is a separate Agent plugin; pnpm / Corepack only for toolkit monorepo development.

Platforms

| OS | Notes | | ----------- | ---------------------------------------------------------------------------------------------------------------- | | Linux | npx @brookesia/app-dev-plugin or npm i -g + brookesia-plugin | | Windows | Same commands in PowerShell or cmd; use where/npm from Node installer; reopen terminal after global installs | | macOS | Same as Linux; Playwright cache under ~/Library/Caches/ms-playwright |

Agent detection uses CLI (where on Windows, which on Unix) and config folders under %USERPROFILE% / $HOME.

Can I install without an official Marketplace?

Yes. This package is not yet listed in official Agent plugin marketplaces (Cursor, Claude, Codex, etc.), but you can use it in those IDEs today. The recommended path is MCP configuration, not searching the extension store.

Two integration modes

| Mode | Official Marketplace required? | Status | | ---------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------- | | MCP integration (32 tools + 9 resources) | No | Supported — primary path via npx @brookesia/app-dev-plugin | | Plugin package (Skills / Hooks / Commands) | Optional; local or npm load works too | Manifests ship in the npm package; installer writes MCP by default |

Core workflows (build, simulate, validate, screenshot compare, etc.) run through MCP and do not depend on marketplace listing.

Recommended: MCP install (no Marketplace)

npx @brookesia/app-dev-plugin          # interactive: wire detected Agents
npx @brookesia/app-dev-plugin cursor   # Cursor only
npx @brookesia/app-dev-plugin codex    # Codex only

Then Reload MCP in your IDE. You should see the brookesia-app-dev server in Agent chat.

Per-Agent setup without Marketplace listing

| Agent | How | | --------------------------------------------------------- | ---------------------------------------------------------------- | | Cursor | Installer writes ~/.cursor/mcp.json → Settings → MCP → Reload | | Claude Code / Desktop | Installer writes ~/.claude/mcp.json or ~/.claude.json | | Codex CLI | codex mcp add brookesia-app-dev -- … or ~/.codex/config.json | | OpenCode / Windsurf / Trae / Gemini / Copilot / ZCode | Installer writes each product's MCP config (see table below) |

Advanced: plugin package form (Skills / Hooks)

The npm package includes plugin manifests for future marketplace listing or local plugin loading:

| Path | Agent | | -------------------------------------- | -------------------------- | | .cursor-plugin/plugin.json | Cursor Plugin | | .claude-plugin/ + marketplace.json | Claude Code Skills / Hooks | | .codex-plugin/plugin.json | Codex | | .opencode/plugins/ | OpenCode | | gemini-extension.json | Gemini CLI | | skills/, commands/, hooks/ | Guided Agent workflows |

Example for Claude Code — add a local marketplace (no official store required):

claude plugin marketplace add <path-to-cloned-repo/plugins/brookesia-app-dev>

Skills/Hooks load only when the Agent's plugin install path is used. With MCP-only setup, all MCP tools remain available.

Relation to official Marketplaces

• Not listed → cannot one-click install from IDE plugin stores yet
• Published on npm → usable now via npx / npm i -g + the installer
• Future marketplace listing → mainly discoverability; MCP wiring continues to work

Installation

Option 1: npx (try without a global install)

npx @brookesia/app-dev-plugin

Two npm packages — different roles:

• @brookesia/app-dev-plugin — MCP installer (wires Agent IDE configs)
• esp-brookesia-toolkit — brookesia build/simulate CLI (required by brookesia_init and related MCP tools)

The installer checks both package versions before writing MCP. Upgrade commands: Upgrade plugin and toolkit. Playwright, brookesia-usb-cli, curl, and OS libraries are presence-only checks (no version compare).

In the interactive Agent menu:

• ✅ marks Agents detected on this machine
• Press Enter or type a → configure MCP for all detected Agents
• Type 1–10 → configure a single Agent

After MCP configuration, follow the presence-check prompts for Playwright Chromium (screenshots / visual compare, ~150MB) and other optional tools.

MCP launches via npx @brookesia/app-dev-plugin@latest by default. For routine updates see Upgrade plugin and toolkit.

Option 2: npm global install (recommended for daily use)

npm install -g @brookesia/app-dev-plugin@latest
brookesia-plugin

With a global install, MCP defaults to a local absolute path (faster startup). npm install -g @brookesia/app-dev-plugin@latest upgrades the plugin only. To bump the toolkit in the same command, see Upgrade plugin and toolkit.

Single Agent only

npx @brookesia/app-dev-plugin cursor
npx @brookesia/app-dev-plugin codex
brookesia-plugin claude

Installer options

brookesia-plugin --help

# Common flags
brookesia-plugin deps --yes          # Install/upgrade toolkit and missing deps (no MCP menu)
--yes              # Upgrade plugin/toolkit and install missing optional dependencies without prompting
--yes-plugin       # Auto-upgrade @brookesia/app-dev-plugin
--yes-toolkit      # Auto-install or upgrade esp-brookesia-toolkit
--yes-playwright   # Auto-install Playwright Chromium if cache is missing (does not upgrade an existing cache)
--yes-usb-cli      # Auto-install brookesia-usb-cli (Python; device install via USB)
--yes-docs-mcp     # Auto-configure Espressif Documentation MCP (esp-en-docs)
--yes-system-deps # Auto-install curl (apt) + Linux Playwright OS libs
--no-system-deps   # Skip curl / Playwright OS libs
--no-plugin        # Skip the plugin version check
--no-toolkit       # Skip toolkit install
--no-playwright    # Skip Playwright install
--no-usb-cli       # Skip brookesia-usb-cli install
--no-docs-mcp      # Skip Espressif Documentation MCP
--npx              # Force npx MCP launch (even with global install)
--local            # Force absolute server path (dev / offline)
--pin-version      # Pin npx to this plugin version instead of @latest

Supported Agent IDEs / CLIs

| ID | Agent | MCP config location (examples) | | ---------------- | ------------------ | ---------------------------------------- | | cursor | Cursor | ~/.cursor/mcp.json | | claude | Claude Code | ~/.claude/mcp.json | | claude-desktop | Claude Desktop | ~/.claude.json | | codex | Codex CLI / IDE | codex mcp add → ~/.codex/config.toml | | opencode | OpenCode | ~/.config/opencode/opencode.json | | windsurf | Windsurf | ~/.windsurf/mcp_config.json | | trae | Trae | ~/.trae/mcp.json | | gemini | Gemini CLI | ~/.gemini/settings.json | | copilot | GitHub Copilot CLI | ~/.copilot/config.json | | zcode | ZCode | ~/.zcode/cli/config.json, etc. |

After installation, restart the IDE or Reload MCP. You should see the brookesia-app-dev server in Agent chat.

Verify installation

Terminal:

brookesia --version
brookesia-plugin --help

In Agent chat:

List all MCP tools
→ Expect 32 MCP tools (including brookesia_board_capabilities and brookesia_visual_loop)

Read brookesia://schemas/service-catalog
→ Expect the device service API catalog

Read brookesia://schemas/board-capabilities
→ Extensible board catalog (do not pin a board in the app)

Quick MCP probe:

echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}' \
  | npx -y -p @brookesia/app-dev-plugin@latest brookesia-app-dev-mcp

Project environment (app root):

Call brookesia_doctor in Agent chat with your app projectDir and confirm capabilities.ok === true.

MCP tools (32)

Figma / JSON UI export

| Tool | Purpose | | --------------------------- | -------------------------------------------------------------------------- | | prepare_json_ui_export | Start a Figma → JSON UI export session; show the token and paste steps | | get_json_ui_export_status | Poll export progress | | cancel_json_ui_export | Cancel export | | inspect_json_ui_layout | Inspect JSON UI layout and hierarchy |

CLI / simulation

| Tool | Purpose | | ---------------------------------- | ---------------------------------------------------------------------- | | brookesia_init | Initialize an app project | | brookesia_build | Build | | brookesia_release | Release build artifacts | | brookesia_pack | Pack .bpk | | brookesia_verify | Verify package and manifest | | brookesia_deploy | Install a built .bpk onto a device via USB (brookesia-usb install) | | brookesia_device_status | Query device status via USB | | brookesia_board_capabilities | Hardware vs firmware service map (only with named board or USB info) | | brookesia_validate_service_usage | Scan @brookesia/service imports against bound hardware | | brookesia_simulate | Start WASM browser simulation | | brookesia_simulate_reload | Hot-reload simulator | | brookesia_simulate_stop | Stop simulator | | brookesia_simulator_probe | Probe simulator state | | brookesia_doctor | Environment / capability diagnostics |

JSON UI design and validation

| Tool | Purpose | | ---------------------------- | ------------------------------------------------- | | validate_gui_json | Validate GUI JSON against schema | | list_json_ui_elements | List UI elements and geometry | | enrich_interactive_widgets | Enrich interactive widgets (events, states, etc.) | | scaffold_app_js | Scaffold app.js from UI semantics | | validate_app_js_safety | Check app.js safety constraints | | get_safe_app_js_template | Return a safe app.js template |

Screenshots / visual / icons

| Tool | Purpose | | -------------------------------- | ------------------------------------------------- | | prepare_reference_screenshot | Normalize a reference screenshot | | brookesia_visual_loop | One-call prepare → build → simulate → compare | | compare_simulator_to_reference | Pixel-compare simulator vs reference | | extract_icons_from_reference | Extract icon regions from a reference image | | assess_icon_drawability | Assess whether icons can be drawn with primitives | | render_icons_from_primitives | Render icons from primitives | | verify_image_assets | Verify image assets | | convert_extracted_icons | Convert extracted icon assets |

MCP resources (9)

Read these in Agent chat via Resource URI (e.g. “Read brookesia://schemas/gui”):

| URI | Content | | ------------------------------------------ | -------------------------------------------------- | | brookesia://schemas/service-catalog | @brookesia/service device service APIs | | brookesia://schemas/board-capabilities | Board hardware catalog (no app-level board pin) | | brookesia://schemas/gui | JSON UI GUI schema | | brookesia://schemas/manifest | manifest.json schema | | brookesia://schemas/node-constraints | Node-type property whitelist | | brookesia://guides/json-ui-and-bpk | JSON UI + app package guide (links official docs) | | brookesia://skills/auto-test | Auto-test skill | | brookesia://skills/screenshot-to-json-ui | Screenshot-to-JSON-UI skill | | brookesia://skills/figma-to-json-ui | Figma plugin → JSON UI (plugins already installed) |

Official esp-brookesia documentation (human-readable source of truth):

• JSON UI: docs/zh_CN/gui/interface/json_ui
• App package / .bpk: docs/zh_CN/system/core/app_package.rst

Bundled skills

The plugin ships scenario-specific skills that Agents route to automatically:

| Skill | Use when | | ----------------------------- | ------------------------------------------------------- | | setup-brookesia-environment | Node / npm / toolkit setup and diagnostics | | develop-brookesia-app | CLI build, pack, simulate, verify | | design-brookesia-json-ui | Figma / JSON UI design and export | | figma-to-json-ui | Figma Community plugin → JSON UI (plugins already in) | | screenshot-to-json-ui | Generate JSON UI from screenshots and converge visually | | auto-test | Autonomous UI/logic testing with screenshot iteration | | using-brookesia-app-dev | Top-level router for the skills above |

Typical workflows

1. Create and run a Brookesia app

In Agent chat (natural language is fine):

1. brookesia_init — scaffold the project
2. Edit JSON UI under src/res/ and logic under src/app/
3. brookesia_build → brookesia_simulate — preview in the browser
4. brookesia_pack / brookesia_verify — produce .bpk

Example apps: esp-brookesia-apps

2. Screenshot / mockup → UI

1. Provide a reference screenshot
2. Agent uses the screenshot-to-json-ui skill
3. Prefer brookesia_visual_loop to converge; fall back to validate_gui_json → brookesia_simulate → compare_simulator_to_reference

3. Device services (Storage / Wi-Fi, etc.)

1. Read brookesia://schemas/service-catalog
2. If the user named a board or USB Device/GetServiceNames is available, call brookesia_board_capabilities then brookesia_validate_service_usage. Do not write a board into brookesia.config.js. Unlisted boards: Espressif Documentation MCP esp-en-docs.
3. Call APIs from app.js via @brookesia/service
4. Verify with brookesia_simulate (some services need device firmware). Pass boardId into brookesia_build only when hardware is bound.

4. Figma → JSON UI

Already installed: Agent brookesia-app-dev MCP + Figma desktop Community plugin ESP-Brookesia JSON UI. Do not reinstall. This is not Figma MCP and not screenshot-to-JSON.

You: In Agent chat say “把 Figma 导出到这个 app” (or /figma-export). Agent: Shows a one-time token (do not bury it). You in Figma: Export Package (must succeed) → expand Send to Brookesia Agent (optional · local only) → paste token → Send. Token is 10 minutes, one use. Agent cannot paste it. Do not use Copy Bundle.

If Agent MCP is not running, use the terminal instead (keep it open):

npx @brookesia/app-dev-plugin figma-token
# or: npx @brookesia/app-dev-plugin figma-token ./my-app

Do not run that command while Agent MCP is already on port 38473.

Then Agent polls import and validates (inspect_json_ui_layout). Full Agent instructions: brookesia://skills/figma-to-json-ui.

Upgrade and uninstall

Upgrade plugin and toolkit

Both packages can be updated together. Playwright Chromium is not version-upgraded on this path (see below).

One command (global npm packages only; no MCP wiring):

npm install -g @brookesia/app-dev-plugin@latest esp-brookesia-toolkit@latest

Via the installer (compares versions, then prompts or honors flags):

| Goal | Command | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Interactive: ask about plugin then toolkit | npx @brookesia/app-dev-plugin or brookesia-plugin if installed globally | | Non-interactive: upgrade both (also installs missing optional deps; may write MCP) | npx @brookesia/app-dev-plugin --yes | | Non-interactive, deps only (no MCP menu) | npx @brookesia/app-dev-plugin deps --yes | | Plugin + toolkit only; skip Playwright | npx @brookesia/app-dev-plugin deps --yes-plugin --yes-toolkit --no-playwright |

Installer behavior:

• Checks @brookesia/app-dev-plugin and global esp-brookesia-toolkit before any Agent MCP write; shows installed vs npm latest when outdated.
• An outdated npx run delegates to the exact latest version (avoids a stale @latest cache). A global install uses npm install -g @pkg@<exact-version>.
• A development tree (node bin/cli.js) does not bump npm packages.
• --yes implies --yes-plugin and --yes-toolkit. Use --no-plugin / --no-toolkit to skip one side.

After upgrading, Reload MCP in the Agent IDE/CLI. Reopen the terminal if brookesia is missing from PATH after a global install.

Playwright Chromium

The installer only looks for chromium-* (or chromium_headless_shell-*) in the browser cache. It does not compare browser versions.

| Flag | Chromium already present | Chromium missing | | ----------------------------- | ------------------------ | -------------------------------------- | | --no-playwright | Skip | Skip | | --yes-playwright or --yes | Skip (not an upgrade) | Runs npx playwright install chromium |

To refresh an existing cache:

npx playwright install chromium

Uninstall (recommended CLI)

brookesia-plugin uninstall              # Interactive: MCP configs + optional cleanup
brookesia-plugin uninstall --yes        # Non-interactive: MCP + global npm + npx cache
brookesia-plugin uninstall --dry-run    # Preview removals
brookesia-plugin uninstall --yes-all    # Also toolkit / Playwright cache

The uninstaller:

1. Removes brookesia-app-dev from all supported Agent MCP configs (including renamed entries that still launch this plugin)
2. Runs codex mcp remove brookesia-app-dev when Codex CLI is available
3. Backs up JSON configs under the OS temp directory before editing
4. Optionally: npm uninstall -g @brookesia/app-dev-plugin, clears npx cache, removes Playwright Chromium cache

Manual removal: delete the MCP entry in Agent settings, or edit the config files listed in the Agent table above.

npm only:

npm uninstall -g @brookesia/app-dev-plugin esp-brookesia-toolkit

Local development (monorepo)

In the esp-brookesia-toolkit repo:

cd plugins/brookesia-app-dev
npm test
node bin/cli.js --local cursor    # dev tree defaults to --local MCP

Manual testing: install a .bpk on real hardware

brookesia_deploy installs a built .bpk onto an ESP device via the official brookesia-usb CLI (USB Serial/JTAG). See the official USB CLI docs for device setup, brookesia-usb commands, and prerequisites (firmware with the USB service enabled, python3 -m pip install --user brookesia-usb-cli).

This toolkit tree pins simulator/components/esp-brookesia to a USB-enabled revision so PC simulator / firmware builds stay aligned with devices that expose the USB CDC service. Host-side install still uses PyPI brookesia-usb-cli.

Plugin-specific flow (in Agent chat, after brookesia build):

brookesia_deploy({ projectDir: "<app root>", port: "/dev/ttyACM0" })
  → ok:true, bytes, sha256 (installs dist/*.bpk)
brookesia_device_status({ projectDir, port: "/dev/ttyACM0", systemInfo: true })
  → device status + SuperOS version

Gotchas observed on real hardware (esp32p4 / SuperOS 0.8.3):

• Space deploy and status calls by ~1-2s — the USB control session is exclusive; an immediate follow-up call may return ok:false (busy). Wait and retry.
• StartApp prints "Brookesia app started" (non-JSON) on the CDC channel, so brookesia-usb may report invalid device response — the app actually started (confirm via ListApps → Running).

FAQ

Q: I cannot find it in Claude / Codex plugin marketplaces.
A: See Can I install without an official Marketplace? above. Use MCP settings for day-to-day work. For Claude Skills/Hooks as a plugin package, run claude plugin marketplace add against the local npm tree (see .claude-plugin/).

Q: MCP connects but build/simulate / brookesia_init fails.
A: esp-brookesia-toolkit is missing or was skipped during install. Run npm install -g esp-brookesia-toolkit@latest or npx @brookesia/app-dev-plugin deps --yes, reopen your terminal / Reload MCP, and check with brookesia_doctor.

Q: How do I upgrade the plugin and toolkit together?
A: See Upgrade plugin and toolkit. Fastest: npm install -g @brookesia/app-dev-plugin@latest esp-brookesia-toolkit@latest.

Q: Screenshot compare fails.
A: If Chromium was never installed, use --yes-playwright or npx playwright install chromium. --yes-playwright does not upgrade an existing cache; to refresh the browser, run npx playwright install chromium again.

Q: Windows notes?
A: Node 20+ required. In CMD, answer Y when prompted for toolkit; after global installs, reopen CMD or your IDE to refresh PATH.

Related links

• ESP-Brookesia Toolkit — CLI and SDK source
• esp-brookesia-apps — JavaScript app examples
• ESP-Brookesia — Device HMI framework
• @brookesia/service — JS device service SDK

License

Apache-2.0 — Espressif Systems (Shanghai) CO LTD