@molecule/api-content-moderation

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Content moderation core interface for molecule.dev.

Defines the abstract contract for AI-powered content moderation and user report management. Bond a concrete provider (e.g., one backed by @molecule/api-ai) to enable moderation in your application.

Quick Start

import { setProvider, requireProvider } from '@molecule/api-content-moderation'
import type { ContentModerationProvider } from '@molecule/api-content-moderation'

// Bond a provider at startup
setProvider(myModerationProvider)

// Use anywhere in the app
const moderation = requireProvider()
const result = await moderation.check('some user content')
if (result.flagged) {
  console.log('Content flagged:', result.categories)
}

Scoring only? Use a classifier. Classifier bonds score content and store nothing; bond one with setClassifier and call requireClassifier():

import { setClassifier, requireClassifier } from '@molecule/api-content-moderation'
import { classifier } from '@molecule/api-content-moderation-openai' // or -molecule

setClassifier(classifier)

const verdict = await requireClassifier().check(comment.body)
if (verdict.flagged) throw new Error('This comment breaks the community rules.')

Type

core

Installation

npm install @molecule/api-content-moderation @molecule/api-bond @molecule/api-i18n

API

Interfaces

ContentClassifierProvider

A content CLASSIFIER — scores text (and optionally images) against moderation categories and nothing else. No report storage.

This is the part of moderation a vendor or model actually provides (OpenAI's moderation endpoint, a self-hosted model, molecule.dev's hosted service). It is bonded separately (setClassifier) so a classifier bond does not have to be a report database too. An app's {@link ContentModerationProvider} can delegate its check() / checkImage() to requireClassifier() and keep its reports in the app's own DataStore.

Category names are the provider's own (e.g. harassment/threatening); every result carries a 0–1 score per category and an overall flagged.

interface ContentClassifierProvider {
  /** Provider name (e.g. 'openai', 'molecule'). */
  readonly name: string

  /**
   * Classify text.
   *
   * @param content - The text to classify.
   * @param options - `threshold` overrides the provider's own flag decision:
   *   when set, a category is flagged iff its score ≥ threshold. `categories`
   *   limits the returned (and flag-deciding) categories.
   * @returns Per-category scores and the overall decision.
   */
  check(content: string, options?: ModerationOptions): Promise<ModerationResult>

  /**
   * Classify an image. Optional — absent when the classifier is text-only.
   *
   * @param image - The image bytes.
   * @param options - As for `check`, plus the image's MIME type.
   * @returns Per-category scores and the overall decision.
   */
  checkImage?(image: Uint8Array, options?: ImageModerationOptions): Promise<ModerationResult>
}

ContentModerationConfig

Configuration for the content moderation provider.

interface ContentModerationConfig {
  /** Default score threshold above which content is flagged (0–1). */
  threshold?: number
  /** Categories to check by default. */
  defaultCategories?: string[]
  /** Whether to include reasoning in moderation results. */
  includeReasoning?: boolean
}

ContentModerationProvider

Content moderation provider interface.

Implement this interface in a bond package to provide concrete content moderation (e.g., via @molecule/api-ai) and report management (e.g., via @molecule/api-database).

interface ContentModerationProvider {
  /** Provider name (e.g. 'ai-moderation', 'perspective-api'). */
  readonly name: string

  /**
   * Checks text content against moderation rules.
   *
   * @param content - The text content to check.
   * @param options - Optional moderation configuration.
   * @returns The moderation result with per-category scores.
   */
  check(content: string, options?: ModerationOptions): Promise<ModerationResult>

  /**
   * Checks image content against moderation rules.
   *
   * @param image - The image data as a byte array.
   * @param options - Optional moderation configuration.
   * @returns The moderation result with per-category scores.
   */
  checkImage(image: Uint8Array, options?: ModerationOptions): Promise<ModerationResult>

  /**
   * Submits a user report against a resource.
   *
   * @param input - The report creation input.
   * @returns The created report.
   */
  report(input: CreateReportInput): Promise<Report>

  /**
   * Retrieves reports with optional filtering and pagination.
   *
   * @param options - Query and pagination options.
   * @returns A paginated result of reports.
   */
  getReports(options?: ReportQuery): Promise<PaginatedResult<Report>>

  /**
   * Resolves a pending report with a moderation decision.
   *
   * @param reportId - The ID of the report to resolve.
   * @param resolution - The resolution action and details.
   */
  resolveReport(reportId: string, resolution: Resolution): Promise<void>
}

CreateReportInput

Input for creating a new report.

interface CreateReportInput {
  /** The type of resource being reported. */
  resourceType: string
  /** The ID of the resource being reported. */
  resourceId: string
  /** The ID of the user submitting the report. */
  reporterId: string
  /** The reason for the report. */
  reason: string
}

ImageModerationOptions

Options for classifying an image.

interface ImageModerationOptions extends ModerationOptions {
  /** The image's MIME type, e.g. `image/png`. Defaults to `image/jpeg`. */
  mimeType?: string
}

ModerationCategoryResult

Per-category result from a moderation check.

interface ModerationCategoryResult {
  /** The category that was evaluated. */
  category: string
  /** Whether the content was flagged in this category. */
  flagged: boolean
  /** Confidence score between 0 and 1. */
  score: number
}

ModerationOptions

Options for configuring a moderation check.

interface ModerationOptions {
  /** Limit checking to specific categories. Defaults to all categories. */
  categories?: string[]
  /** Score threshold above which content is flagged (0–1). Defaults to provider-specific value. */
  threshold?: number
  /** Additional context to help the moderation model make a decision. */
  context?: string
}

ModerationResult

Result of checking content against moderation rules.

interface ModerationResult {
  /** Whether the content was flagged by any category. */
  flagged: boolean
  /** Per-category breakdown of the moderation result. */
  categories: ModerationCategoryResult[]
  /** Optional reasoning explaining why content was or was not flagged. */
  reasoning?: string
}

PaginatedResult

A paginated result set.

interface PaginatedResult<T> {
  /** The result items for the current page. */
  data: T[]
  /** Total number of matching items across all pages. */
  total: number
  /** Maximum number of results per page. */
  limit: number
  /** Number of results skipped. */
  offset: number
}

PaginationOptions

Options for paginated queries.

interface PaginationOptions {
  /** Maximum number of results to return. */
  limit?: number
  /** Number of results to skip. */
  offset?: number
}

Report

A user-submitted report against a resource.

interface Report {
  /** Unique report identifier. */
  id: string
  /** The type of resource being reported (e.g. 'comment', 'post'). */
  resourceType: string
  /** The ID of the resource being reported. */
  resourceId: string
  /** The ID of the user who submitted the report. */
  reporterId: string
  /** The reason for the report. */
  reason: string
  /** Current status of the report. */
  status: ReportStatus
  /** Resolution details, if resolved. */
  resolution?: string
  /** ID of the moderator who resolved the report. */
  resolvedBy?: string
  /** When the report was created (ISO 8601). */
  createdAt: string
  /** When the report was last updated (ISO 8601). */
  updatedAt: string
}

ReportQuery

Query options for fetching reports.

interface ReportQuery {
  /** Maximum number of results to return. */
  limit?: number
  /** Number of results to skip. */
  offset?: number
  /** Filter by report status. */
  status?: ReportStatus
  /** Filter by resource type. */
  resourceType?: string
}

Resolution

Resolution action for a moderation report.

interface Resolution {
  /** The action taken on the report. */
  action: 'approve' | 'reject' | 'dismiss'
  /** Optional reason for the resolution. */
  reason?: string
  /** ID of the moderator who resolved the report. */
  resolvedBy: string
}

Types

ModerationCategory

Category of content violation detected during moderation.

type ModerationCategory =
  'hate' | 'violence' | 'sexual' | 'self-harm' | 'harassment' | 'dangerous' | 'spam' | 'custom'

ReportStatus

Status of a user-submitted report.

type ReportStatus = 'pending' | 'reviewing' | 'resolved' | 'dismissed'

Functions

getClassifier()

Retrieves the bonded content classifier, or null if none is bonded.

function getClassifier(): ContentClassifierProvider | null

Returns: The bonded classifier, or null.

getProvider()

Retrieves the bonded content moderation provider, or null if none is bonded.

function getProvider(): ContentModerationProvider | null

Returns: The bonded provider, or null.

hasClassifier()

Checks whether a content classifier is currently bonded.

function hasClassifier(): boolean

Returns: true if a classifier is bonded.

hasProvider()

Checks whether a content moderation provider is currently bonded.

function hasProvider(): boolean

Returns: true if a provider is bonded.

requireClassifier()

Retrieves the bonded content classifier, throwing if none is bonded.

function requireClassifier(): ContentClassifierProvider

Returns: The bonded content classifier.

requireProvider()

Retrieves the bonded content moderation provider, throwing if none is bonded. Use this when moderation functionality is required.

function requireProvider(): ContentModerationProvider

Returns: The bonded content moderation provider.

setClassifier(classifier)

Registers a content classifier (text/image scoring only, no reports).

function setClassifier(classifier: ContentClassifierProvider): void

• classifier — The classifier to bond.

setProvider(provider)

Registers a content moderation provider.

function setProvider(provider: ContentModerationProvider): void

• provider — The content moderation provider to bond.

Available Providers

| Provider | Package | | ---------- | ------------------------------------------- | | Moderation | @molecule/api-content-moderation-molecule | | Moderation | @molecule/api-content-moderation-openai |

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-bond ^1.0.1
• @molecule/api-i18n ^1.0.1

Runtime Dependencies

• @molecule/api-bond

• @molecule/api-i18n

• Two bond slots. setClassifier / requireClassifier hold a ContentClassifierProvider (scores only — OpenAI, molecule.dev hosted). The full ContentModerationProvider (setProvider) adds user reports and a moderator queue; no package ships one, so implement it in the app: delegate check() / checkImage() to requireClassifier() and store reports with the DataStore. Never setProvider(classifier) — it lacks the report methods.

• Classifier category names are the provider's own (harassment/threatening, sexual/minors, …), not the ModerationCategory union — filter with options.categories using the names the provider returns.

• Unlike most cores, there are NO module-level convenience delegates. Call methods on requireProvider() (throws when unbonded). Note getProvider() returns null rather than throwing — don't optional-chain into silently skipping moderation.

• Moderate SERVER-SIDE, before persisting or publishing. Run check() inside the create/update handler and block or quarantine flagged content there — a client-side check is decoration, not enforcement.

• Choose the failure mode explicitly. If the moderation call itself fails (AI backend down), decide fail-open (publish + log) or fail-closed (hold for review) per surface — don't let the exception 500 the request.

• Report workflows are privileged. report() is for authenticated end users; getReports() / resolveReport() power a moderator surface — gate those routes with an admin authorizer.

• Thresholds and category coverage are provider-specific — pass ModerationOptions.threshold / categories rather than assuming defaults.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip:

• [ ] Every surface that accepts user-generated content (post, comment, image upload, bio — whatever this app has) runs it through check() / checkImage() SERVER-SIDE in the create/update handler, before persisting. Confirm the moderation call is on the write path, not client-side or skipped.
• [ ] Assert BOTH directions with real samples: clearly-violating content returns flagged: true and the UI rejects it with a visible reason; benign content returns flagged: false and publishes normally. A moderator that flags everything or nothing is broken.
• [ ] The decision gates persistence: a blocked item is NOT stored and is absent when you view the feed as a second user. Tighten ModerationOptions.threshold and a borderline item flips allowed → blocked.
• [ ] The failure mode is deliberate: when the moderation/AI call errors the request does NOT 500 — content is either published + logged (fail-open) or held for review (fail-closed) per the surface's intent.
• [ ] If the app has reporting, the round-trip works: a user's report() creates a 'pending' Report, it appears in the moderator queue via getReports(), and resolveReport() (approve/reject/dismiss) visibly changes its status and clears it from the pending queue.
• [ ] Moderator surfaces are privileged: a non-admin can't reach getReports() / resolveReport() (403) and can't see other users' flagged or pending content.