videocontext

Browser-based video content recognition with IAB content taxonomy classification.

Recognizes what's playing in HTML5 video players and outputs standardized IAB Content Taxonomy categories — ready for programmatic advertising (OpenRTB, VAST, Prebid.js) or any system that needs content classification.

Features

• 3 recognition engines — use solo or in combination:
  • Metadata — page context, meta tags, schema.org, URL patterns, player metadata
  • Cloud AI — Claude Vision, Google Vision, AWS Rekognition (via server proxy)
  • On-device ML — TensorFlow.js or ONNX Runtime Web (fully client-side)
• 5 player adapters — HTML5 <video>, Video.js, HLS.js, Shaka Player, JW Player
• IAB taxonomy 2.0 & 3.0 — selectable per instance, with cross-version mapping
• Ad-stack ready — OpenRTB and VAST formatted output, global window.__videocontext
• Privacy-aware — on-device ML needs no external API; battery-aware frame capture

Install

npm install videocontext

Optional peer dependencies (only needed if using those engines):

npm install @tensorflow/tfjs    # for on-device TF.js engine
npm install onnxruntime-web      # for on-device ONNX engine

Quick Start

import { createVideoContext } from 'videocontext';

const vc = createVideoContext({
  target: '#my-video',          // CSS selector or HTMLVideoElement
  taxonomyVersion: '3.0',
  engines: [
    { type: 'metadata', weight: 1.0 }
  ],
  exposeGlobal: true,           // sets window.__videocontext
});

vc.on('classification', (result) => {
  console.log(result.categories);
  // [{ id: '1000', name: 'Sports', path: 'Sports', confidence: 0.85, tier: 1 }, ...]
});

await vc.init();

Configuration

interface VideoContextConfig {
  // Required
  target: string | HTMLVideoElement;

  // Taxonomy
  taxonomyVersion?: '2.0' | '3.0';           // default: '3.0'
  crossVersionMapping?: boolean;               // default: false

  // Engines (at least one required)
  engines?: EngineConfig[];                    // default: [{ type: 'metadata', weight: 1.0 }]
  confidenceThreshold?: number;                // default: 0.5 (0-1)
  maxCategories?: number;                      // default: 5

  // Frame capture (for cloud/ondevice engines)
  capture?: {
    fps?: number;                              // default: 0.2 (one frame per 5 seconds)
    maxDimension?: number;                     // default: 320
    format?: 'image/jpeg' | 'image/png' | 'image/webp';  // default: 'image/jpeg'
    quality?: number;                          // default: 0.7
    batteryAware?: boolean;                    // default: true
  };

  // Player
  adapter?: 'html5' | 'videojs' | 'hlsjs' | 'shaka' | 'jwplayer';  // default: auto-detect

  // Output
  exposeGlobal?: boolean;                      // default: false
  debounceMs?: number;                         // default: 5000
}

Engines

Metadata (no API, instant)

Extracts keywords from page context and matches them to IAB categories.

engines: [
  {
    type: 'metadata',
    weight: 1.0,
    options: {
      useSchemaOrg: true,    // scan JSON-LD / schema.org
      useMetaTags: true,     // scan <meta> tags and Open Graph
      useUrlPatterns: true,  // classify based on URL patterns
    }
  }
]

Cloud AI (requires server proxy)

Captures video frames and sends them to a cloud vision API for classification.

Security: API keys must never be in browser code. Use proxyUrl to route requests through your server.

engines: [
  {
    type: 'cloud',
    weight: 0.7,
    options: {
      provider: 'claude',                    // 'claude' | 'google-vision' | 'aws-rekognition'
      proxyUrl: '/api/classify-frame',       // your server endpoint
      maxFramesPerRequest: 3,
    }
  }
]

Claude Vision is prompted to return IAB categories directly — no label-to-taxonomy mapping needed.

Google Vision / AWS Rekognition return object labels that are fuzzy-matched against the IAB taxonomy.

On-Device ML (fully client-side)

Runs a classification model in the browser. No API calls, fully private.

engines: [
  {
    type: 'ondevice',
    weight: 0.8,
    options: {
      runtime: 'tfjs',                       // 'tfjs' | 'onnx'
      modelUrl: '/models/classifier/model.json',
      preferGpu: true,
    }
  }
]

Models are cached in IndexedDB after the first download. Place a labels.json file next to the model for human-readable class names.

Combining Engines

Use multiple engines with weights to get the best results:

engines: [
  { type: 'metadata', weight: 0.3 },
  { type: 'cloud', weight: 0.7, options: { provider: 'claude', proxyUrl: '/api/classify' } },
]

The classifier normalizes scores per engine, applies weights, groups by category, sums weighted scores, filters by threshold, and returns the top N.

Events

vc.on('classification', (result) => { ... });
vc.on('classification:change', (result, previous) => { ... });
vc.on('engine:result', (engineType, categories) => { ... });
vc.on('engine:error', (engineType, error) => { ... });
vc.on('adapter:detected', (adapterName) => { ... });
vc.on('capture:frame', (frameIndex) => { ... });
vc.on('ready', () => { ... });
vc.on('destroy', () => { ... });

Output Formats

Instance API

const result = vc.getResult();
// { timestamp, taxonomyVersion, categories, sources, confidence }

Global Object

When exposeGlobal: true:

window.__videocontext.result    // ClassificationResult
window.__videocontext.openrtb   // { cat: ['1000', '1028'], cattax: 6 }
window.__videocontext.vast      // [{ authority: '...', value: '1000' }, ...]

OpenRTB Integration (Prebid.js)

import { formatOpenRTB } from 'videocontext';

vc.on('classification:change', (result) => {
  const ortb = formatOpenRTB(result);
  // ortb = { cat: ['1000', '1028'], cattax: 6 }

  pbjs.mergeConfig({
    ortb2: {
      site: { content: ortb },
    },
  });
});

cattax values follow the OpenRTB spec: 2 = IAB 2.0, 6 = IAB 3.0.

VAST Integration

import { formatVAST } from 'videocontext';

const vast = formatVAST(result);
// [{ authority: 'https://iabtechlab.com/standards/content-taxonomy-3', value: '1000' }]

Player Adapters

Adapters are auto-detected by default. To force a specific adapter:

const vc = createVideoContext({
  target: '#player',
  adapter: 'videojs',  // 'html5' | 'videojs' | 'hlsjs' | 'shaka' | 'jwplayer'
});

Custom Adapter

Implement the PlayerAdapter interface:

import type { PlayerAdapter } from 'videocontext';

class MyPlayerAdapter implements PlayerAdapter {
  readonly name = 'my-player';
  canHandle(target) { ... }
  attach(target) { ... }
  detach() { ... }
  getVideoElement() { ... }
  getPlaybackState() { ... }
  getMetadata() { ... }
  on(event, handler) { ... }
  off(event, handler) { ... }
}

const vc = createVideoContext({
  target: '#player',
  adapter: new MyPlayerAdapter(),
});

API Reference

createVideoContext(config)

Factory function. Returns a new VideoContext instance.

VideoContext

| Method | Description | |--------|-------------| | init(): Promise<void> | Initialize adapters, engines, and start classification | | classify(): Promise<ClassificationResult> | Force an immediate classification cycle | | getResult(): ClassificationResult \| null | Get the latest classification result | | on(event, handler) | Subscribe to events | | off(event, handler) | Unsubscribe from events | | updateConfig(partial) | Update configuration at runtime | | destroy(): Promise<void> | Stop all engines and clean up resources |

ClassificationResult

{
  timestamp: string;              // ISO 8601
  taxonomyVersion: '2.0' | '3.0';
  categories: CategoryMatch[];
  mappedCategories?: CategoryMatch[];  // when crossVersionMapping is true
  sources: ('metadata' | 'cloud' | 'ondevice')[];
  confidence: number;             // 0-1 average confidence
}

CategoryMatch

{
  id: string;       // e.g., 'IAB17' (2.0) or '1000' (3.0)
  name: string;     // e.g., 'Sports'
  path: string;     // e.g., 'Sports > Soccer'
  confidence: number; // 0-1
  tier: number;      // 1-4
}

Edge Cases

| Scenario | Behavior | |----------|----------| | Cross-origin video | Canvas becomes tainted — frame capture disabled, falls back to metadata engine. Logs a warning. Fix: add crossorigin="anonymous" to <video>. | | DRM/EME content | Hardware DRM produces blank frames. Detected via pixel sampling, frame is discarded. | | Cloud API failure | Network errors return empty categories. Other engines continue. Error emitted via engine:error. | | Model load failure | On-device engine fails gracefully. Error emitted via engine:error. Other engines continue. | | Low battery | When batteryAware: true and battery < 20%, frame capture rate is halved. | | No engines initialize | init() throws an error. | | Missing taxonomy data | Logs a warning. Run npm run build:taxonomy. |

Building from Source

git clone <repo>
cd videocontext
npm install
npm run build:taxonomy   # generate IAB taxonomy JSON from TSV
npm run build            # compile ESM + CJS + types
npm test                 # run test suite
npm run typecheck        # TypeScript type check

License

MIT