@cristianglezm/live-commentary-widget

A React component library that adds a "Twitch-style" live chat overlay to your application. It uses AI Vision models to "watch" the screen (or a specific canvas) and generate humor, context-aware commentary, or useful insights in real-time.

✨ Features

• Plug & Play: Drop <LiveCommentary /> into any React app.
• Visual Context: Users can toggle "Show Context" on comments to verify exactly what the AI saw at that moment.
• Headless Hooks: Use useLiveCommentary and useScreenCapture to build your own custom UI.
• AI Middleware: Intercept raw AI responses to handle your own parsing, username assignment, or state management.
• AI Powered: Compatible with OpenAI's Vision API (gpt-4o) and local alternatives (e.g., llama.cpp server).
• Screen & Canvas Support: Capture the entire screen via the browser API or hook directly into a <canvas> for games.
• Sticky Scroll: Chat widget automatically scrolls to new messages but pauses when you scroll up.

📦 Installation

npm install @cristianglezm/live-commentary-widget

💻 Usage

1. Basic Usage (The "Easy" Way)

The LiveCommentary component wraps the logic and the UI into one standard widget.

import { LiveCommentary } from '@cristianglezm/live-commentary-widget';
import '@cristianglezm/live-commentary-widget/style.css';

function App() {
  return (
    <div>
      <LiveCommentary 
        config={{
          model: 'gpt-4o',
          apiKey: 'not_required', 
        }} 
      />
    </div>
  );
}

2. Middleware (Controlled Mode)

Want to control exactly who says what? Use responseTransform to intercept the raw text from the LLM and return your own message objects.

Pro Tip: You also receive the capturedImage snapshot, so you can attach it to your custom messages to enable the "Show Context" feature.

import { LiveCommentary, createChatMessage } from '@cristianglezm/live-commentary-widget';

<LiveCommentary 
  responseTransform={(rawText, capturedImage) => {
    // rawText: The raw string response from the AI
    // capturedImage: The base64 image used for this analysis
    
    // You can parse JSON, XML, or just return a hardcoded user
    return [
      // createChatMessage(text, username, color, customUsernamesList, attachment)
      createChatMessage(rawText, "MyCustomBot", "#ff0000", undefined, capturedImage)
    ];
  }}
/>

3. Headless Mode (Custom UI)

Build your own UI entirely using the hooks.

import { useLiveCommentary, useScreenCapture, ChatWidget } from '@cristianglezm/live-commentary-widget';

function MyCustomPage() {
  const { isCapturing, startCapture, captureFrame } = useScreenCapture({ mode: 'screen-capture' });
  
  const { messages, triggerEvaluation } = useLiveCommentary({
     isCapturing,
     captureFrame,
     config: { apiKey: '...' }
  });

  return (
    <div>
      <button onClick={startCapture}>Start Recording</button>
      <button onClick={() => triggerEvaluation()}>Force Comment</button>
      
      {/* You can use our widget or render your own list */}
      <div className="my-custom-chat-container">
        {messages.map(msg => (
            <div key={msg.id}>
                <b>{msg.username}:</b> {msg.text}
                {/* Access the visual context image via msg.attachment */}
                {msg.attachment && <img src={`data:image/jpeg;base64,${msg.attachment}`} />}
            </div>
        ))}
      </div>
    </div>
  );
}

📸 Visual Context

The widget automatically attaches a snapshot of the screen to every AI-generated comment. This allows users to understand why the AI said something.

1. Hover/Look at a message in the chat.
2. Click the "Show Context" button (small eye icon).
3. The image frame analyzed by the AI will appear below the text.

This is particularly useful for:

• Debugging prompts: See if the model is hallucinating or just saw something you missed.
• Verification: Proving the commentary is "live" and reacting to the actual video feed.

⚙️ Configuration Props (LiveCommentary)

| Prop | Type | Default | Description | |------|------|---------|-------------| | config | Partial<VlmSettings> | | Override API settings (url, key, model, temp). | | mode | 'screen-capture' \| 'external' | 'screen-capture' | Use 'external' for direct canvas/video hook. | | responseTransform | (raw: string, img?: string) => ChatMessage[] | | New: Middleware to handle raw AI text yourself. | | captureSource | () => string \| null | | Required if mode is external. Returns base64 image string. | | prompts | CommentaryPrompts | | Customize the system instructions and triggers. | | contextData | object | | Arbitrary JSON data sent to the AI for context. | | usernames | string[] | undefined | Custom list of usernames for random assignment. | | showBadges | boolean | true | Show Twitch-style badges (Broadcaster, Prime, etc). | | title | string | "Live Commentary" | Header title of the widget. | | overlay | boolean | true | If true, fixed positioning. If false, fills parent. |

🎨 Customization

The widget uses CSS variables for theming. Import the CSS and override variables in your :root.

import '@cristianglezm/live-commentary-widget/style.css';

:root {
  --color-lc-bg: #18181b;
  --color-lc-accent: #9147ff;
}

📄 License

MIT © Cristian Gonzalez