Interaction Tools MCP

A deliberately thin MCP server for raw Browser, PTY, and isolated X11 interaction. It gives an agent input and observation primitives; the agent or its Skill remains responsible for the user journey, waits, retries, evidence, diagnosis, assertions, and reports.

Published on npm as @wkj24/agentic-interactive-test.

Requirements and setup

• Node.js 22
• A C/C++ toolchain for node-pty
• Playwright's Chromium browser
• Linux X11 commands for the optional X11 surface

cd mcp/interaction-tools
npm install
npx playwright install chromium

# Debian/Ubuntu, for x11_* tools only
sudo apt install --no-install-recommends \
  xvfb openbox xauth xdotool ffmpeg kitty

npm run typecheck
npm test
npm run build

node-pty may require Python, make, and a C/C++ compiler on Linux; Xcode Command Line Tools on macOS; or Python and the Visual Studio C++ workload on Windows. Browser and PTY tools are portable where their dependencies support the platform. The first X11 implementation is Linux-only and checks every external command before allocating a desktop session.

Configure Codex with the absolute compiled entry path:

[mcp_servers.interaction_tools]
command = "node"
args = ["/absolute/path/to/mcp/interaction-tools/dist/src/server.js"]

The server reserves stdout for MCP protocol messages. It does not emit startup logs there.

For the published package, register the MCP server directly through npx:

codex mcp add interaction-tools -- npx -y @wkj24/agentic-interactive-test

Tool surface

The server exposes exactly 22 tools:

| Surface | Lifecycle and observation | Input | | --- | --- | --- | | Browser | web_open, web_snapshot, web_close | web_click, web_drag, web_scroll, web_type, web_key | | PTY | terminal_open, terminal_snapshot, terminal_close | terminal_text, terminal_key, terminal_mouse, terminal_resize | | X11 | x11_open, x11_snapshot, x11_close | x11_mouse, x11_key, x11_text, x11_window |

Successful open calls return a session ID. Input calls return only dispatch acceptance, not a semantic success verdict. Close calls release everything owned by that session. Driver failures are returned unchanged at the MCP boundary as one of:

{"code":"session_not_found","message":"..."}

The possible codes are session_not_found, target_not_found, process_exited, unsupported_action, and backend_error.

Browser

web_open launches Playwright and navigates to a URL. web_snapshot returns a PNG plus URL, title, and viewport metadata. Mouse coordinates are viewport pixels. Keyboard expressions use Playwright syntax. Playwright is imported only when the first Browser session opens.

PTY

terminal_open starts a real pseudoterminal and parses its bounded scrollback with a headless terminal. terminal_snapshot returns the visible text viewport, dimensions, cursor state, and raw process state. Mouse actions require the target to have enabled compatible SGR 1006 tracking; otherwise they return unsupported_action.

Closing owns and terminates the PTY process group. A target that deliberately daemonizes with setsid or otherwise leaves that group crosses the MCP's ownership boundary and must provide its own shutdown mechanism; the server does not scan and kill unrelated process trees.

A PTY/headless snapshot does not claim pixel-identical Kitty, XTerm, font, window-manager, compositor, or physical-terminal behavior. Use X11 with Kitty when those visible desktop interactions matter.

X11

x11_open creates a fresh authenticated Xvfb display, starts Openbox, then launches the requested target inside it. It uses -nolisten tcp, a private Xauthority file, private runtime/cache directories, and never attaches to the user's existing DISPLAY or WAYLAND_DISPLAY. x11_snapshot returns the framebuffer PNG, visible windows, client geometry, focus, and target process state. Window actions operate through Openbox/xdotool.

Closing a session terminates both the target process group and the complete Xvfb/Openbox group, then removes its private directory. Xvfb/Openbox coverage does not imply physical-display, GPU, compositor, input-device, accessibility, or performance coverage.

Scope and resource behavior

The server holds only active session resources. It stores no screenshot history, trace, evidence set, assertion, retry policy, or report. Playwright is loaded lazily, X11 processes start only for an X11 session, and terminal scrollback is capped at 1000 lines.

The resource smoke test prints informational RSS samples for idle, open, and closed states. It deliberately has no fixed RSS ceiling because allocators, browsers, fonts, native libraries, and platforms vary. Its hard assertions are resource ownership and cleanup: closed sessions become inaccessible, private directories disappear, and owned process groups exit.