Semantic-Similarity Rating (SSR) — TypeScript

A TypeScript port of the Python Semantic-Similarity Rating library for converting LLM textual responses into Likert-scale probability distributions using semantic similarity.

Inspiration

This package is a direct port of the Python SSR library by PyMC Labs. The original implements the methodology described in:

Maier, B. F., Aslak, U., Fiaschi, L., Pappas, K., Wiecki, T. (2025). Measuring Synthetic Consumer Purchase Intent Using Semantic-Similarity Ratings.

Rather than bridging to Python at runtime (child processes, HTTP servers, etc.), this package reimplements the core algorithms in pure TypeScript so they can be consumed natively in any JavaScript or TypeScript project.

Installation

npm install @sirmews/semantic-similarity-rating

Text mode (auto-computes embeddings)

If you want the library to compute sentence embeddings automatically, install the optional peer dependency:

npm install @huggingface/transformers

Embedding mode (pre-computed embeddings)

If you supply your own embeddings, no additional dependencies are needed.

Quick Start

import { ResponseRater } from "@sirmews/semantic-similarity-rating";
import type { ReferenceSentence } from "@sirmews/semantic-similarity-rating";

// Define reference sentences for a Likert scale (any N ≥ 2)
const refs: ReferenceSentence[] = [
  { id: "set1", intResponse: 1, sentence: "Strongly disagree" },
  { id: "set1", intResponse: 2, sentence: "Disagree" },
  { id: "set1", intResponse: 3, sentence: "Neutral" },
  { id: "set1", intResponse: 4, sentence: "Agree" },
  { id: "set1", intResponse: 5, sentence: "Strongly agree" },
];

// Text mode — embeddings are computed automatically
const rater = await ResponseRater.create(refs);

// Convert LLM responses to probability distributions
const pmfs = await rater.getResponsePmfs("set1", [
  "I totally agree",
  "Not sure about this",
  "Completely disagree",
]);
// pmfs[0] → high probability on "Agree" / "Strongly agree"
// pmfs[1] → peaked around "Neutral"
// pmfs[2] → high probability on "Strongly disagree"

// Aggregate into a single survey-level PMF
const surveyPmf = rater.getSurveyResponsePmf(pmfs);

OpenAI embeddings

Use the OpenAI embeddings API instead of the local model — no extra dependencies required (uses native fetch):

import {
  ResponseRater,
  OpenAIEmbeddingProvider,
} from "@sirmews/semantic-similarity-rating";

const provider = new OpenAIEmbeddingProvider({
  apiKey: process.env.OPENAI_API_KEY!,
  // model: "text-embedding-3-large",         // optional, defaults to "text-embedding-3-small"
  // baseURL: "https://my-proxy.example.com/v1", // optional, for OpenAI-compatible APIs
});

const rater = await ResponseRater.create(refs, { embeddingProvider: provider });

Embedding mode

// Provide pre-computed embeddings (e.g. from your own pipeline)
const refsWithEmbeddings: ReferenceSentence[] = refs.map((r) => ({
  ...r,
  embedding: myEmbeddingFunction(r.sentence), // number[]
}));

const rater = await ResponseRater.create(refsWithEmbeddings);

// Pass pre-computed response embeddings instead of text
const pmfs = await rater.getResponsePmfs("set1", responseEmbeddingMatrix);

API

ResponseRater.create(refs, options?)

Async factory — creates and initialises a rater.

| Parameter | Type | Description | |-----------|------|-------------| | refs | ReferenceSentence[] | Reference sentences (with or without embeddings). Supports any N-point scale (N ≥ 2) with contiguous integer intResponse values. | | options.modelName | string | HuggingFace model id (default "Xenova/all-MiniLM-L6-v2") | | options.embeddingProvider | EmbeddingProvider | Custom/mock embedding provider for DI |

rater.getResponsePmfs(refSetId, responses, temperature?, epsilon?)

Convert LLM responses into PMFs over the Likert scale.

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | refSetId | string | — | Reference set id, or "mean" to average across all sets | | responses | string[] or number[][] | — | Text (text mode) or embeddings (embedding mode) | | temperature | number | 1.0 | Temperature scaling (0 = one-hot, 1 = identity, >1 = smooth) | | epsilon | number | 0.0 | Regularisation to prevent exact-zero probabilities |

rater.getSurveyResponsePmf(pmfs)

Average individual response PMFs into a single survey-level PMF.

rater.getSurveyResponsePmfByReferenceSetId(refSetId, responses, temperature?, epsilon?)

Convenience: getResponsePmfs + getSurveyResponsePmf in one call.

Pure functions

The core algorithm is also available as standalone pure functions:

import { responseEmbeddingsToPmf, scalePmf } from "@sirmews/semantic-similarity-rating";

Integrity & Cross-Validation

This port was built with strict behavioural fidelity to the Python original:

• Line-by-line algorithm port — compute.ts and response-rater.ts mirror their Python counterparts step by step, with inline comments referencing the original NumPy operations.
• Cross-validation test suite — Exact inputs generated by NumPy (np.random.seed(42), np.random.seed(43)) are fed to both the Python and TypeScript implementations. Outputs are compared to full float64 precision with a tolerance of 1×10⁻¹² (well below the 1×10⁻⁶ acceptance criterion).
• 120 tests, 6 suites — covering linear algebra helpers, PMF computation, temperature scaling, validation rules, the full ResponseRater lifecycle, and N-point Likert scale support.
• Zero external numeric dependencies — all linear algebra (linalg.ts) is implemented with plain number[] arrays, avoiding any third-party matrix library that could introduce behavioural drift.
• Deterministic test infrastructure — a seeded PRNG (Mulberry32 + Box-Muller) replaces np.random so tests are reproducible without NumPy.

Caveats

• Floating-point differences — JavaScript uses IEEE 754 float64 like NumPy, so results are identical to within rounding. Cross-validation tests confirm agreement to 1×10⁻¹² or better. In practice, differences are negligible for SSR use cases.
• Model download on first use — In text mode, the first call downloads the ONNX model (~50 MB) via @huggingface/transformers. Subsequent runs use the local cache.
• @huggingface/transformers is an optional peer dependency — It is only required for text mode. If you only use embedding mode (pre-computed vectors), you do not need to install it.

Future Improvements

• In-Memory Batching: The current engine processes text arrays synchronously in memory, which may cause memory spikes for excessively large datasets. Implementing an automated chunking/batching parameter within the response processing pipeline would safely distribute computation without altering the external API.

Architecture

| Module | Purpose | Pure? | |--------|---------|-------| | types | All type definitions and interfaces | N/A | | linalg | Plain-array linear algebra helpers | ✅ | | compute | Core SSR algorithm (PMF conversion & scaling) | ✅ | | validation | Reference sentence validation | ✅ | | embeddings | Embedding provider (wraps @huggingface/transformers) | Async | | openai-embeddings | Embedding provider (OpenAI API via fetch) | Async | | response-rater | High-level ResponseRater class | Async |

Python ↔ TypeScript mapping

| Python | TypeScript | |--------|------------| | ResponseRater(df) | ResponseRater.create(refs) | | rater.get_response_pmfs(…) | rater.getResponsePmfs(…) | | rater.get_survey_response_pmf(pmfs) | rater.getSurveyResponsePmf(pmfs) | | rater.encode_texts(texts) | rater.encodeTexts(texts) | | scale_pmf(pmf, T) | scalePmf(pmf, T) | | response_embeddings_to_pmf(…) | responseEmbeddingsToPmf(…) |

Citation

Maier, B. F., Aslak, U., Fiaschi, L., Pappas, K., Wiecki, T. (2025).
Measuring Synthetic Consumer Purchase Intent Using Semantic-Similarity Ratings.

License

Apache License 2.0 — see LICENSE.

Built with the help of Amp.