agent-ui-annotation

Live Demo

A web page annotation toolbar for AI coding agents. Click on elements, add notes, and export structured markdown that helps AI assistants locate and modify specific UI components.

Overview

When working with AI coding agents, communicating which visual element needs modification can be challenging. Saying "fix the blue button in the sidebar" is ambiguous. agent-ui-annotation solves this by allowing you to:

1. Click any element on a web page
2. Add feedback or notes
3. Export structured markdown with element identifiers, CSS selectors, framework component paths, and context

The exported markdown gives AI agents the precise information needed to locate elements in code.

Installation

# npm
npm install agent-ui-annotation

# pnpm
pnpm add agent-ui-annotation

# yarn
yarn add agent-ui-annotation

Quick Start

Note: agent-ui-annotation is a development tool for communicating UI changes to AI coding agents. You should conditionally include it based on your environment to avoid shipping it to production.

Vanilla JavaScript

import { createAnnotation } from 'agent-ui-annotation';

const annotation = createAnnotation({
  theme: 'auto',
  // Add custom context to every annotation
  onBeforeAnnotationCreate: (data) => ({
    context: {
      route: window.location.pathname,
      timestamp: new Date().toISOString(),
    },
  }),
  onAnnotationCreate: (annotation) => console.log('Created:', annotation),
  onCopy: (markdown) => console.log('Copied:', markdown),
});

// Activate the toolbar
annotation.activate();

As a Web Component

<script type="module">
  import 'agent-ui-annotation';
</script>

<agent-ui-annotation theme="auto" output-level="standard"></agent-ui-annotation>

React

import { AgentUIAnnotation } from 'agent-ui-annotation/react';

function App() {
  return (
    <AgentUIAnnotation
      theme="auto"
      outputLevel="standard"
      onBeforeAnnotationCreate={(data) => ({
        context: { route: window.location.pathname },
      })}
      onAnnotationCreate={(annotation) => console.log('Created:', annotation)}
      onCopy={(markdown) => console.log('Copied:', markdown)}
    />
  );
}

Vue 3

<script setup lang="ts">
import { AgentUIAnnotation, useAgentUIAnnotation } from 'agent-ui-annotation/vue';
import type { Annotation, BeforeAnnotationCreateResult } from 'agent-ui-annotation/vue';

const { ref: annotationRef, activate, copyOutput } = useAgentUIAnnotation();

function handleBeforeCreate(): BeforeAnnotationCreateResult {
  return { context: { route: window.location.pathname } };
}

function handleCreate(annotation: Annotation) {
  console.log('Created:', annotation);
}

function handleCopy(markdown: string) {
  console.log('Copied:', markdown);
}
</script>

<template>
  <AgentUIAnnotation
    ref="annotationRef"
    theme="auto"
    output-level="standard"
    :on-before-annotation-create="handleBeforeCreate"
    @annotation-create="handleCreate"
    @copy="handleCopy"
  />
  <button @click="activate">Start Annotating</button>
  <button @click="copyOutput()">Copy Output</button>
</template>

Svelte 5

<script lang="ts">
  import { AgentUIAnnotation } from 'agent-ui-annotation/svelte';
  import type { Annotation, BeforeAnnotationCreateResult } from 'agent-ui-annotation/svelte';

  let annotation: AgentUIAnnotation;

  const onBeforeAnnotationCreate = (): BeforeAnnotationCreateResult => ({
    context: { route: window.location.pathname },
  });
  const onAnnotationCreate = (a: Annotation) => console.log('Created:', a);
  const onCopy = (content: string) => console.log('Copied:', content);
</script>

<AgentUIAnnotation
  bind:this={annotation}
  theme="auto"
  outputLevel="standard"
  {onBeforeAnnotationCreate}
  {onAnnotationCreate}
  {onCopy}
/>
<button onclick={() => annotation.activate()}>Start Annotating</button>
<button onclick={() => annotation.copyOutput()}>Copy Output</button>

Angular

import { Component, CUSTOM_ELEMENTS_SCHEMA, viewChild, ElementRef, afterNextRender } from '@angular/core';
import type { AnnotationElement } from 'agent-ui-annotation';
import 'agent-ui-annotation';

@Component({
  selector: 'app-root',
  schemas: [CUSTOM_ELEMENTS_SCHEMA],
  template: `
    <agent-ui-annotation #annotation theme="auto" output-level="standard"></agent-ui-annotation>
    <button (click)="activate()">Start Annotating</button>
  `,
})
export class App {
  annotationRef = viewChild<ElementRef<AnnotationElement>>('annotation');

  constructor() {
    afterNextRender(() => {
      const element = this.annotationRef()?.nativeElement;
      if (element) {
        // Add custom context to annotations
        element.setBeforeCreateHook((data) => ({
          context: { route: window.location.pathname },
        }));
        element.addEventListener('annotation:create', (e: Event) => {
          console.log('Created:', (e as CustomEvent).detail.annotation);
        });
      }
    });
  }

  activate() {
    this.annotationRef()?.nativeElement.activate();
  }
}

Features

Click-to-Annotate

Click any element to add feedback. The toolbar captures the element type, location, and your notes.

Smart Element Identification

Generates human-readable names like button "Save" or input [email] that AI agents can understand.

CSS Selector Paths

Creates CSS selectors (e.g., form > .actions > button) for precise element location.

Framework Component Paths

Automatically detects and displays the framework component hierarchy for annotated elements. Works with React, Vue, Svelte, and Angular in development mode.

| Framework | Example Output | |-----------|---------------| | React 18 | App (App.tsx:7) > LandingPage (App.tsx:54) > CTAButton (App.tsx:38) | | React 19 | App > LandingPage > CTAButton (names only, no file/line) | | Vue | App (App.vue) > LandingPage (LandingPage.vue) > CTAButton (CTAButton.vue) | | Svelte | LandingPage.svelte:5 > HeroSection.svelte:5 > CTAButton.svelte:5 | | Angular | _App > _LandingPageComponent > _CTAButtonComponent |

Component paths appear in the popup, hover tooltip, and markdown output (standard level and above).

Multi-Select

Hold and drag to select multiple elements at once.

Freeze Mode

Pause CSS animations and videos to capture specific states.

Block Page Interactions

When active, clicks are blocked from triggering buttons/links while annotating (configurable in settings).

Output Detail Levels

| Level | Description | |-------|-------------| | Compact | 1. **button "Save"**: Change color to blue | | Standard | Element + path + component path + comment with headers | | Detailed | + framework prefix, classes, position, nearby context, custom context | | Forensic | + full DOM path, computed styles, accessibility info, custom context |

Persistence

Annotations are saved to localStorage and persist across page reloads (7-day retention).

Custom Context Hook

The onBeforeAnnotationCreate hook lets you inject custom context (route info, user data, app state) into annotations before they're created. You can also modify the comment or cancel annotation creation.

// Add route and user context to every annotation
const annotation = createAnnotation({
  onBeforeAnnotationCreate: (data) => ({
    context: {
      route: window.location.pathname,
      params: Object.fromEntries(new URLSearchParams(window.location.search)),
      userId: getCurrentUserId(),
      timestamp: new Date().toISOString(),
    },
  }),
});

// Conditionally cancel annotation creation
const annotation = createAnnotation({
  onBeforeAnnotationCreate: (data) => {
    // Don't annotate elements inside specific containers
    if (data.element.closest('.no-annotate')) {
      return { cancel: true };
    }
    return { context: { route: window.location.pathname } };
  },
});

// Modify the comment
const annotation = createAnnotation({
  onBeforeAnnotationCreate: (data) => ({
    comment: `[${window.location.pathname}] ${data.comment}`,
  }),
});

The custom context appears in the markdown output at detailed and forensic levels.

Internationalization (i18n)

agent-ui-annotation supports multiple languages with built-in English and Simplified Chinese translations.

Basic Usage

Initialize i18n once at your app's entry point, before rendering any components:

import { initI18n } from 'agent-ui-annotation';

// Initialize at app startup (optional for English - it's the default)
initI18n({ locale: 'zh-CN' });

Note: If you're using English, you can skip calling initI18n() entirely - English is the default locale.

Framework Examples

React (in App.tsx or main.tsx):

import { initI18n } from 'agent-ui-annotation';
import { AgentUIAnnotation } from 'agent-ui-annotation/react';

// Initialize once at app startup
initI18n({ locale: 'zh-CN' });

function App() {
  return <AgentUIAnnotation theme="auto" />;
}

Vue (in main.ts):

import { initI18n } from 'agent-ui-annotation';

initI18n({ locale: 'zh-CN' });

// Then create your Vue app...

Svelte (in +layout.ts or main entry):

import { initI18n } from 'agent-ui-annotation';

initI18n({ locale: 'zh-CN' });

Vanilla JS:

import { createAnnotation, initI18n } from 'agent-ui-annotation/vanilla';

// Option 1: Use initI18n (recommended for consistency with other frameworks)
initI18n({ locale: 'zh-CN' });
const annotation = createAnnotation({ theme: 'auto' });

// Option 2: Pass locale directly to createAnnotation
const annotation = createAnnotation({ locale: 'zh-CN' });

Custom Translations

Override specific strings while keeping the rest:

import { initI18n } from 'agent-ui-annotation';

initI18n({
  locale: 'en',
  translations: {
    toolbar: {
      copiedFeedback: 'Successfully copied!',
    },
    popup: {
      addFeedback: 'Enter your notes...',
    },
  },
});

Output Translation

By default, markdown output stays in English for AI compatibility. To translate output:

initI18n({
  locale: 'zh-CN',
  translateOutput: true, // Translate markdown labels
});

Built-in Locales

| Locale | Language | |--------|----------| | en | English (default) | | zh-CN | Simplified Chinese |

Register Custom Locale

import { registerLocale, initI18n, en } from 'agent-ui-annotation';

const customLocale = {
  ...en,
  toolbar: {
    ...en.toolbar,
    settings: 'Einstellungen',
  },
};

registerLocale('de', customLocale);
initI18n({ locale: 'de' });

Contributing Translations

We welcome contributions for additional languages! To add a new locale:

1. Copy src/core/i18n/locales/en.ts as a template
2. Create a new file like src/core/i18n/locales/de.ts for your language
3. Translate all strings, keeping the same key structure
4. Export from src/core/i18n/index.ts
5. Submit a pull request

See the en.ts and zh-CN.ts files for examples.

API Reference

Options

interface AnnotationOptions {
  theme?: 'light' | 'dark' | 'auto';
  outputLevel?: 'compact' | 'standard' | 'detailed' | 'forensic';
  annotationColor?: string;
  onBeforeAnnotationCreate?: BeforeAnnotationCreateHook; // Hook to add context/modify/cancel
  onAnnotationCreate?: (annotation: Annotation) => void;
  onAnnotationUpdate?: (annotation: Annotation) => void;
  onAnnotationDelete?: (id: string) => void;
  onAnnotationsClear?: (annotations: Annotation[]) => void;
  onCopy?: (content: string, level: OutputLevel) => void;
}

// Hook types for onBeforeAnnotationCreate
interface BeforeAnnotationCreateData {
  element: Element;           // The DOM element being annotated
  elementInfo: ElementInfo;   // Collected element information
  comment: string;            // User's comment
  selectedText: string | null; // Selected text on page (if any)
  isMultiSelect: boolean;     // Whether part of multi-select batch
  clickX: number;             // Click X coordinate
  clickY: number;             // Click Y coordinate
}

interface BeforeAnnotationCreateResult {
  context?: Record<string, unknown>; // Custom data to attach
  comment?: string;                   // Override the comment
  cancel?: boolean;                   // Set true to cancel creation
}

type BeforeAnnotationCreateHook = (
  data: BeforeAnnotationCreateData
) => BeforeAnnotationCreateResult | Promise<BeforeAnnotationCreateResult> | void | Promise<void>;

// i18n is configured separately via initI18n()
interface I18nOptions {
  locale?: string;                    // e.g., 'en', 'zh-CN'
  translations?: PartialTranslations; // Custom translation overrides
  translateOutput?: boolean;          // Translate markdown output (default: false)
}

Instance Methods

interface AnnotationInstance {
  activate(): void;
  deactivate(): void;
  toggle(): void;
  copyOutput(level?: OutputLevel): Promise<boolean>;
  getOutput(level?: OutputLevel): string;
  clearAll(): void;
  destroy(): void;
}

Web Component Attributes

| Attribute | Type | Default | Description | |-----------|------|---------|-------------| | theme | light | dark | auto | auto | Color theme | | output-level | compact | standard | detailed | forensic | standard | Output detail level | | annotation-color | string | #AF52DE | Marker color | | disabled | boolean | false | Disable the toolbar |

Note: For i18n, call initI18n() at app startup rather than using component attributes.

Custom Events

| Event | Detail | Description | |-------|--------|-------------| | annotation:create | { annotation: Annotation } | Annotation created | | annotation:update | { annotation: Annotation } | Annotation updated | | annotation:delete | { id: string } | Annotation deleted | | annotation:clear | { annotations: Annotation[] } | All annotations cleared | | annotation:copy | { content: string, level: OutputLevel } | Output copied |

Output Examples

Compact

1. **button "Save"**: Change color to blue
2. **input [email]**: Add validation

Standard

## Page Feedback: /dashboard

### 1. button "Save"
**Location:** form > .actions > button
**Component:** App (App.tsx:12) > SaveForm (SaveForm.tsx:8)
**Feedback:** Change color to blue

---

### 2. input [email]
**Location:** form > .form-group > input
**Feedback:** Add validation

Forensic

Includes complete DOM path, computed styles, accessibility info, viewport dimensions, and timestamps.

Architecture

agent-ui-annotation uses a three-layer architecture:

1. Core - Framework-agnostic business logic
2. Web Components - Shadow DOM rendering
3. Framework Adapters - React, Vue 3, Svelte 5, Angular, Vanilla JS wrappers

Development

# Install dependencies
pnpm install

# Run development server
pnpm dev

# Build for production
pnpm build

# Run tests
pnpm test

# Type check
pnpm typecheck

Browser Support

• Chrome 90+
• Firefox 90+
• Safari 14+
• Edge 90+

License

MIT