@cogstream/copilotkit

CopilotKit / AG-UI bridge for CogStream.

@cogstream/copilotkit connects CogStream sensing, interpretation, actions, voice primitives, and dynamic surface rendering to CopilotKit and AG-UI experiences.

It is a bridge, not a replacement for CopilotKit.

Status

Alpha. APIs may change before 1.0.

This package remains backward-compatible with the existing @cogstream/copilotkit API. Surface support is additive.

What this package provides

• CogStreamProvider and session context
• default CogStream actions for CopilotKit
• voice control primitives
• VAD, input, output, and interruption helpers
• AG-UI event dispatch
• generative component registry
• CogStream surface rendering bridges
• chat, sidebar, popup, inline, and workspace surfaces

Relationship to @cogstream/surfaces

@cogstream/surfaces owns the framework-neutral dynamic surface contracts:

| Contract | Meaning | | --- | --- | | SurfaceGenerationContext | What is happening in the journey | | SurfaceGenerationControl | What is allowed to appear or happen | | SurfacePayload | What should be rendered | | GeneratedSurfaceContext | What surface appeared | | SurfaceOutcome | Whether the surface helped |

@cogstream/copilotkit adapts those contracts to CopilotKit / AG-UI experiences.

CogStream Behavioral Intelligence
  → @cogstream/surfaces
  → @cogstream/copilotkit
  → CopilotKit / AG-UI surfaces
  → user interaction
  → SurfaceOutcome

Installation

npm install @cogstream/copilotkit

@cogstream/surfaces is installed as a dependency.

If your application imports surface types or creates SurfacePayload objects directly, install it explicitly:

npm install @cogstream/copilotkit @cogstream/surfaces

Peer dependencies:

next >=14
react >=18
react-dom >=18

Provider

Place <CopilotKit> above <CogStreamProvider> in your tree.

import { CopilotKit } from "@copilotkit/react-core";
import { CogStreamProvider } from "@cogstream/copilotkit";

export default function Layout({ children }: { children: React.ReactNode }) {
  return (
    <CopilotKit runtimeUrl="/api/copilotkit">
      <CogStreamProvider
        apiKey={process.env.NEXT_PUBLIC_COGSTREAM_API_KEY!}
        sessionId={crypto.randomUUID()}
      >
        {children}
      </CogStreamProvider>
    </CopilotKit>
  );
}

CogStreamProvider does not replace or wrap CopilotKit. It expects CopilotKit context to already exist.

Existing component rendering

Existing generative component rendering still works:

import {
  registerComponent,
  GenerativeRenderer,
} from "@cogstream/copilotkit";

registerComponent("HelpCard", HelpCard);

export function HelpSurface() {
  return <GenerativeRenderer name="HelpCard" props={{ title: "Need help?" }} />;
}

CogStream surface rendering

CogStream surfaces use SurfacePayload objects from @cogstream/surfaces.

import { CogStreamInlineSurface } from "@cogstream/copilotkit";
import type {
  SurfaceGenerationContext,
  SurfaceGenerationControl,
  SurfacePayload,
} from "@cogstream/surfaces";

export function InlineHelpSurface({
  payload,
  context,
  control,
}: {
  payload: SurfacePayload;
  context: SurfaceGenerationContext;
  control: SurfaceGenerationControl;
}) {
  return (
    <CogStreamInlineSurface
      payload={payload}
      context={context}
      control={control}
      onOutcome={(outcome) => {
        console.log("surface outcome", outcome);
      }}
    />
  );
}

Declarative surface renderers

Register custom renderers for declarative surface payloads.

import { registerSurfaceRenderer } from "@cogstream/copilotkit";

registerSurfaceRenderer("checklist", ChecklistSurface);
registerSurfaceRenderer("confirmation", ConfirmationSurface);
registerSurfaceRenderer("handoff_panel", HandoffPanelSurface);

If no renderer is registered, GenerativeRenderer renders a minimal fallback surface.

Surface modes

@cogstream/copilotkit supports three surface modes through the GenerativeRenderer bridge:

| Mode | Meaning | | --- | --- | | static | Registered component selected by name | | declarative | Structured SurfacePayload rendered by a surface renderer | | open_ended | Sandboxed open-ended surface content |

High-stakes journeys should prefer static or declarative surfaces. Open-ended surfaces should be sandboxed and policy-gated.

Chat / Chat+ / Chatless surfaces

import {
  CogStreamChat,
  CogStreamSidebar,
  CogStreamPopup,
  CogStreamInlineSurface,
  CogStreamWorkspace,
} from "@cogstream/copilotkit";

• CogStreamChat, CogStreamSidebar, and CogStreamPopup support CopilotKit-style conversational surfaces.
• CogStreamInlineSurface supports chatless in-product dynamic surfaces.
• CogStreamWorkspace supports larger side-by-side or Chat+ style surfaces.

Render policy

Use render policy helpers to validate a proposed surface before rendering.

import { validateSurfacePayloadForRender } from "@cogstream/copilotkit";

const decision = validateSurfacePayloadForRender({
  payload,
  context,
  control,
});

if (!decision.allowed) {
  console.warn(decision.reason);
}

Voice

import {
  useVoiceControl,
  useVoiceSurfaceSync,
} from "@cogstream/copilotkit";

useVoiceSurfaceSync lets the host app coordinate spoken agent output with newly rendered surfaces.

Pass a semanticHintProvider to give the voice agent structured meaning about the element currently in focus. This is optional — voice works without it.

import { addSemanticHint } from '@cogstream/sensing';

const { isListening } = useVoiceControl({
  semanticHintProvider: async (elementId) => {
    // Return the current hint for this element, or undefined
    return getActiveHintForElement(elementId);
  },
});

Testing helpers

import {
  createMockUserStateModel,
  createMockSurfaceGenerationContext,
  createMockSurfaceGenerationControl,
  createMockSurfacePayload,
} from "@cogstream/copilotkit";

These helpers are intended for demos, examples, and smoke tests.

Design rule

Adapters bridge CogStream functionality into native libraries. They do not become the native libraries.

@cogstream/copilotkit adapts CogStream’s surface contract into CopilotKit / AG-UI. The canonical surface model belongs to @cogstream/surfaces.