@cogstream/sensing

Browser SDK for CogStream — captures UI and voice signals, segments them into behavioral episodes, and streams them to the CogStream interpretation service.

Raw signals (pointer events, keystrokes, audio) never leave the browser. Only the derived EpisodeV2 object is transmitted.

Installation

npm install @cogstream/sensing

Quick start

import { SensingRuntime } from '@cogstream/sensing';

const sensing = new SensingRuntime({
  session_id: 'user-abc-123',
  endpoint: 'https://your-cogstream-instance/episode',
  apiKey: 'csk_yourapp_...',
});

sensing.start();

// Stop and flush on page unload
window.addEventListener('beforeunload', () => sensing.stop());

Get an API key by calling POST /admin/tenant on your CogStream instance (requires COGSTREAM_ADMIN_KEY).

Local-only mode (no server)

Process episodes in the browser without sending them anywhere:

const sensing = new SensingRuntime({
  session_id: 'user-abc-123',
  onEpisode: (episode) => {
    console.log('Episode:', episode.episode_type, episode.patterns);
  },
});

sensing.start();

Voice

const sensing = new SensingRuntime({
  session_id: 'user-abc-123',
  voice_enabled: true,
  endpoint: 'https://your-cogstream-instance/episode',
  apiKey: 'csk_yourapp_...',
});

sensing.start();
await sensing.enableVoice(); // requests microphone permission

Swap the default STT adapter:

import { WebSpeechSTTAdapter } from '@cogstream/sensing';

sensing.setSTTAdapter(new WebSpeechSTTAdapter());

API

SensingRuntime

new SensingRuntime(config: SensingRuntimeConfig)

| Option | Type | Required | Description | |--------|------|----------|-------------| | session_id | string | yes | Stable identifier for this user session | | user_id | string | no | Optional authenticated user ID | | voice_enabled | boolean | no | Start with voice capture enabled (default: false) | | flushIntervalMs | number | no | Episode emission interval in ms (default: 500) | | onEpisode | (episode: EpisodeV2) => void | one of | Local callback for each completed episode | | endpoint | string | one of | CogStream service URL for remote episode submission | | apiKey | string | if endpoint set | Bearer token for the remote endpoint | | onPartialEpisode | (partial: PartialEpisode) => void | no | Called with in-flight state on each flush |

At least one of onEpisode or endpoint must be provided.

Methods:

| Method | Description | |--------|-------------| | start() | Begin signal capture | | stop() | Stop capture and flush the final episode | | enableVoice() | Start voice capture (requests mic permission) | | disableVoice() | Stop voice capture | | setSTTAdapter(adapter) | Replace the STT implementation | | getCurrentEpisode() | Returns the in-progress PartialEpisode | | getSessionId() | Returns the configured session_id |

Lightweight primitives (advanced)

For integration with custom pipelines:

import { createSignalCollector, createWindowingEngine } from '@cogstream/sensing';

const collector = createSignalCollector();
collector.attach(document.body);

const engine = createWindowingEngine();
const signals = collector.drain();
engine.feed(signals);

const ep = engine.buildEpisodeV2([], 'session-id');

Semantic hints

Semantic hints let you give the CogStream agent structured meaning about specific elements — for example, labelling a dropdown as a high-stakes "Plan selection" step or marking a billing address block as sensitive.

import { addSemanticHint, clearSemanticHint } from '@cogstream/sensing';
import type { SemanticHint } from '@cogstream/types';

// Label a high-stakes step (content safe to transmit)
addSemanticHint('plan-selector', {
  type: 'label',
  value: 'Plan selection',
  category: 'journey-step',
  complexity: 'high',
});

// Mark a sensitive field — only metadata is transmitted, never the value
addSemanticHint('billing-address', {
  type: 'content',
  value: '123 Main St',   // captured locally for complexity scoring; never leaves browser
  category: 'personal-info',
  sensitive: true,        // required for content hints
});

// Remove a hint before its duration expires (e.g. when the condition changes)
clearSemanticHint('plan-selector');

The hint is automatically cleared after durationMs (default: 5000 ms). Elements without hints are still fully tracked — hints are optional enrichment.

addSemanticHint(elementId, hint, durationMs?)

| Parameter | Type | Description | |-----------|------|-------------| | elementId | string | The element's HTML id attribute | | hint | SemanticHint | A SemanticLabel or SemanticContent object | | durationMs | number | How long to apply the hint (default: 5000) |

Must be called after sensing.start().

clearSemanticHint(elementId)

Removes the hint for the given element immediately. Safe to call even if no hint is set.

SemanticHint types

// SemanticLabel — value is a descriptive label (safe to transmit)
type SemanticLabel = {
  type: 'label';
  value: string;           // e.g. "Plan selection"
  category?: string;       // e.g. "journey-step", "form-field"
  complexity?: 'low' | 'medium' | 'high';
};

// SemanticContent — value is actual content (requires sensitive: true)
type SemanticContent = {
  type: 'content';
  value: string;
  category?: string;
  sensitive: true;          // required — prevents value from leaving the browser
};

Element-level tracking

When the sensing SDK detects behavioral patterns (hesitation, field struggle, etc.), it automatically records which element was involved using the element's id, name, and class attributes. No configuration needed — this happens passively as long as elements have stable id attributes.

Captured signals

The SDK attaches passive listeners for: click, focus, blur, scroll, input (change), field validation errors, form submit errors, pointer move/down/up, and idle detection (3 s debounce).

Links

• cogstream.ai
• Full specification