@framers/agentos-ext-topicality

Embedding-based on/off-topic detection guardrail for @framers/agentos, with optional LLM fallback for ambiguous cases.

What it does

Compares incoming user messages and agent outputs against a configured topic policy using sentence-embedding cosine similarity. If similarity falls below the configured threshold, the guardrail flags the content as off-topic. An optional LLM fallback adjudicates borderline cases.

Use it to:

• Keep customer-support agents on the company's product domain
• Prevent topic drift in long multi-turn conversations
• Audit and report drift incidents per session

Install

npm install @framers/agentos-ext-topicality

This package is an AgentOS extension pack. It expects @framers/agentos as a peer dependency.

Quickstart

import { AgentOS } from '@framers/agentos';
import { createTopicalityGuardrail } from '@framers/agentos-ext-topicality';

const agentos = new AgentOS();
await agentos.initialize({
  extensionManifest: {
    packs: [
      {
        factory: () =>
          createTopicalityGuardrail({
            allowedTopics: ['product support', 'billing', 'account access'],
            similarityThreshold: 0.62,
          }),
        enabled: true,
      },
    ],
  },
});

Public API

• TopicalityGuardrail — guardrail class implementing the AgentOS guardrail contract
• CheckTopicTool — tool agents can invoke to ask the guardrail for an explicit decision
• createTopicalityGuardrail(options) — factory returning an ExtensionPack
• createExtensionPack(context) — auto-discoverable factory used by AgentOS extension auto-pickup
• cosineSimilarity, clearEmbeddingCache — utilities for direct use

See src/types.ts for the full options shape (TopicalityOptions).

Examples

• test/ — unit tests demonstrating threshold tuning and fallback behavior

Lazy loading and optional install

This package is an optional dependency of @framers/agentos-extensions-registry. The registry ships catalog metadata; createCuratedManifest() calls import.meta.resolve() per entry and silently skips anything not installed. npm install @framers/agentos-ext-topicality is the gate.

The embedding model and TopicDriftTracker do not load at activation. The pack registers a factory under the topicality:embedder key in SharedServiceRegistry, and the embedding library enters the module graph only on the first similarity check. Embeddings are cached per-topic-string via clearEmbeddingCache() (exported for tests and re-warm scenarios).

The optional LLM fallback is gated by an optional requiredSecrets entry. Without the provider key, the fallback descriptor is skipped and the regex/embedding tiers still run.

The guardrail registers with config.evaluateStreamingChunks = true and runs in Phase 2 of the two-phase dispatcher (parallel classifiers). Below-threshold similarity returns FLAG; the optional LLM fallback can escalate to BLOCK when configured. Drift incidents accumulate per-session for downstream reporting.

For the full DI model and end-to-end walkthrough, see How extensions stay optional and lazy and the auto-loading guide.

License

Apache 2.0 — see the repo root LICENSE.