@mzhub/portal-client

AI Agent's Eyes & Hands — Browser-side co-browsing infrastructure for AI agents.

Installation

npm install @mzhub/portal-client socket.io-client

Optional Dependencies

# Local LLM (in-browser inference)
npm install @xenova/transformers

# React integration
npm install react @mzhub/react

Quick Start

import { Portal } from "@mzhub/portal-client";

const portal = new Portal({
  serverUrl: "wss://your-server.com",
  overlay: { enabled: true },
  privacy: { mask: [".credit-card", ".ssn"] },
  onPermissionRequest: async (reason) => {
    return confirm(`AI wants to view your screen: ${reason}`);
  },
});

// Connect to server
await portal.connect();

// Create a semantic snapshot (only visible, actionable elements)
const snapshot = portal.snapshot("semantic");

// Highlight an element with tooltip
portal.highlight("#login-btn", "Click here to login");

// Fill a form with human-like typing
await portal.fillForm("#email", "[email protected]");

// Scroll to element
portal.scrollIntoView("#reviews-section");

// Show ghost cursor
portal.showGuidance({ x: 100, y: 200, label: "Click Here" });

HTTP Mode (Alternative to Socket)

For serverless or simple chat-style integrations:

import { Portal, executeAgentCommands } from "@mzhub/portal-client";
import axios from "axios";

const portal = new Portal({ privacy: { maskDefaults: true } });

async function chat(message: string) {
  const response = await axios.post("/api/chat", { message });

  const result = await executeAgentCommands(response.data, {
    portal,
    maxRetries: 2,
    onDomRequest: () => portal.snapshot("semantic"),
    onSendToolResult: (data) =>
      axios.post("/api/chat", data).then((r) => r.data),
  });

  return result.text; // Final reply
}

React Integration

import { useAgentPortal, PortalProvider } from "@mzhub/portal-client/react";

function App() {
  return (
    <PortalProvider options={{ serverUrl: "wss://..." }}>
      <ChatWidget />
    </PortalProvider>
  );
}

function ChatWidget() {
  const { portal, isConnected, isViewing } = useAgentPortal();

  return (
    <div>
      <p>{isConnected ? "🟢 Connected" : "⚪ Disconnected"}</p>
      {isViewing && <p>👁️ AI is viewing</p>}
    </div>
  );
}

Features

Semantic Snapshots

Creates a filtered DOM tree containing only visible, actionable elements:

const snapshot = portal.snapshot("semantic");
// Returns: { elements: [...], viewport: {...}, url, title }

Privacy Masking

Automatically redacts sensitive content before sending:

new Portal({
  privacy: {
    mask: [".credit-card", "[data-private]"],
    maskDefaults: true, // Auto-masks password inputs
  },
});

Developer Overlay

Shows when AI is actively viewing:

new Portal({
  overlay: {
    enabled: true,
    color: "rgba(128, 0, 255, 0.1)",
    message: "AI Assistant is viewing",
  },
});

Local LLM (transformers.js)

Run inference in-browser for privacy-sensitive tasks:

const result = await portal.runLocal(
  "text-classification",
  "Is this password strong?"
);

API Reference

Portal Class

| Method | Description | | ------------------------------- | --------------------------- | | connect() | Connect to WebSocket server | | disconnect() | Disconnect from server | | snapshot(type) | Create DOM snapshot | | highlight(selector, message?) | Highlight element | | scrollIntoView(selector) | Scroll to element | | fillForm(selector, value) | Fill form with typing | | click(selector) | Click element | | showGuidance(options) | Show ghost cursor | | trackScroll(callback) | Track scroll position | | trackFocus(callback) | Track focus events | | onNetworkError(callback) | Listen for network errors | | isConnected() | Check connection status | | isAIViewing() | Check if AI is viewing | | destroy() | Clean up resources |

Events

| Event | Description | | --------------------- | ------------------------ | | connected | Connected to server | | disconnected | Disconnected from server | | viewing | AI started viewing | | stopped | AI stopped viewing | | permissionRequested | Permission requested | | permissionGranted | Permission granted | | permissionDenied | Permission denied | | actionExecuted | Action completed |

License

MIT