@framers/agentos-ext-grounding-guard

NLI-based factual grounding verification for @framers/agentos. Extracts claims from agent outputs and verifies each against the supplied source documents using natural-language-inference scoring.

What it does

Extracts each factual claim from an agent response, runs it against retrieval context (or supplied passages) using an NLI model, and reports per-claim entailment scores. Claims below the configured entailment threshold are flagged as ungrounded.

Use it to:

• Reduce hallucination in RAG agents
• Enforce a minimum-grounded-claims policy before responses ship
• Build hallucination-rate dashboards from per-claim scores

Install

npm install @framers/agentos-ext-grounding-guard

Peer dependency: @framers/agentos.

Quickstart

import { AgentOS } from '@framers/agentos';
import { createGroundingGuardrail } from '@framers/agentos-ext-grounding-guard';

const agentos = new AgentOS();
await agentos.initialize({
  extensionManifest: {
    packs: [
      {
        factory: () =>
          createGroundingGuardrail({
            entailmentThreshold: 0.7,
            minGroundedClaims: 0.8,
          }),
        enabled: true,
      },
    ],
  },
});

Public API

• ClaimExtractor — pulls discrete factual claims from a response
• GroundingChecker — runs each claim through NLI scoring
• GroundingGuardrail — guardrail wiring both for the AgentOS contract
• createGroundingGuardrail(options?) — factory returning an ExtensionPack
• createExtensionPack(context) — auto-discoverable factory used by AgentOS extension auto-pickup

See src/types.ts for GroundingGuardOptions. ValueAgreementChecker and the value reader have their own section, Value agreement, a claim with no number in it is under A stated fact without a number, and the rule for a phrase a public source is searched with, phraseProblem, under A search phrase; verifyQuotes is under Verbatim quotes, checkSentence and checkLabel under Sentence check, and createNliFunction with the model's options under NLI.

Value agreement

Embeddings and NLI grade topic and entailment, not whether 1918 equals 1919. ValueAgreementChecker decides that by code:

import { ValueAgreementChecker } from '@framers/agentos-ext-grounding-guard';

const checker = new ValueAgreementChecker();
const source = { text: 'The Treaty of Versailles was a peace treaty signed on 28 June 1919.', group: 'Treaty_of_Versailles' };
const decision = checker.decide({
  heardWords: 'The Treaty of Versailles was signed in 1918.',
  heardSentence: 'The Treaty of Versailles was signed in 1918.',
  proposed: 'differs',
  named: [source],
  found: [source],
});
// decision.relation === 'differs'; decision.marks[0] marks "1919" in the source's sentence

Each value the claim states (a year, a date, a month, or a number with its unit) pairs with the one value of the same kind in the source's sentence, the nearest shared word deciding when the sentence holds several; the sentence matches when every pair is equal and both sides count the same negations, and differs when a pair is unequal. A "differs" also needs every content word of the claim in the sentence and a name the speaker said, so a source sentence is never shown as differing on a name or a wording alone. A number in a name ("Apollo 17") or a Roman numeral of one or two letters after a capitalised word ("World War I") must also stand in the source's sentence as written (nameMarks reads them) before a "matches", a "differs" or a "disagree" is shown, so a claim about Apollo 17 is never shown as matching or differing from a sentence about Apollo 11.

decide returns null when nothing may be shown: a range, an approximation ("around 1918"), an ordinal, a decade or a number that is part of a name ("Apollo 12") is never compared, names on or off, and a "differs" is withheld when another passage found agrees with the claim's values. Two sentences of one source that pair a value of the claim with different values return disagree. passages picks the sentences worth grading: at most 40 words and 600 characters, at least two stems shared with the claim, and a comparable value of a kind the claim holds; top is the first three.

The value reader (values, valuesEqual), the gates (checkable, and checkableStatement with nameWords), the sentence cuts (sentenceSpans), the verbatim span (exactSpan) and the sentence check (checkSentence, checkLabel) are also the entry @framers/agentos-ext-grounding-guard/values, which imports nothing and runs in a browser. The checker stems with the Porter stemmer of @framers/agentos/nlp; pass { stem } to the constructor to use another.

A stated fact without a number

"Joe Biden is the president." holds no number or date for the value check to compare. checkableStatement, claimProblem with names and a checker with names take such a claim, and the grader's quote gives code the words to compare:

import { ValueAgreementChecker } from '@framers/agentos-ext-grounding-guard';
import { checkableStatement, claimProblem } from '@framers/agentos-ext-grounding-guard/values';

checkableStatement('Joe Biden is the president.'); // true: "Biden" names something
checkableStatement('Coffee is great.'); // false

const fields = { focus: 'Joe Biden is the president.', heard: 'Joe Biden is the president', search: 'Joe Biden' };
claimProblem({ ...fields, statement: 'Donald Trump is the president.' }, { names: true }); // 'statement_names'

const checker = new ValueAgreementChecker({ names: true });
const source = {
  text: 'Joe Biden is an American politician who served as the 46th president of the United States from 2021 to 2025.',
  group: 'Joe_Biden',
};
const decision = checker.decide({
  heardWords: 'Joe Biden is the president.',
  heardSentence: 'Joe Biden is the president.',
  proposed: 'differs',
  named: [source],
  found: [source],
  quote: 'served as the 46th president of the United States from 2021 to 2025',
});
// decision.relation === 'differs'; decision.marks[0] marks the quote in the source's sentence

checkableStatement passes a sentence that may be checked and holds a number in digits, a month's name, or a word after its first that names something (nameWords: it starts with a capital and is not "I", a month or a stop word). A sentence capitalises its first word whatever it is, so "Einstein developed relativity." does not pass. checkable stays the gate for a number or a date.

claimProblem(fields, { names: true }) takes heard words that hold no value at all in place of a value when they hold a name the checker reads and a word of their own. claimNames reads the names: the words after the first that name something, with three letters or more, so "US", "UK", "EU" and "Xi" are not read and "the president of the US" is read as "the president". ownWords reads the claim's own words: its content words that are not capitalised ("president" in "Joe Biden is the president"), or, when it has none, a capitalised word standing alone right after "the" ("the President", as speech-to-text writes it), which is then a name as well. Heard words whose only values are never compared ("signed around 1918", "the 46th president"), and a claim the checker could never decide ("London is the capital of the UK", "Kamala Harris is the Vice President"), are refused with heard_no_value before any source is searched. A statement that drops or swaps a name heard is refused with statement_names, as a statement that changes a number is.

new ValueAgreementChecker({ names: true }) finds and decides the claim. passages keeps a sentence that shares two stems with the claim and holds every name heard after the first word. decide reads the grader's quote, copied from the named sentence:

| Relation | What code asks | |---|---| | Both | The heard words hold no value at all. The sentence holds every name heard after the first word, at least half the claim's content words and as many negations as the heard sentence. The quote stands in it word for word and holds one of the claim's own words. | | matches | The quote holds every one of the claim's own words, no number or date of any kind and no full name the claim lacks, so a quote that dates the fact ("from 2021 to 2025") never matches a fact said in the present, and "a large domesticated cat breed" never matches "a small cat breed". | | differs | The quote holds a number or date the value reader compares, or a full name the claim lacks: two capitalised words in a row ("United States"). The claim is said in the present: "was", "were", "former", "served" or another past form puts it in the past, which a dated quote agrees with, and a fact said in the past is never shown as different on its names. No other passage found holds every content word of the claim with no number, date or full name the claim lacks. Code marks the quote. |

A disagree is never decided without a value, and a claim with a value, compared or not, is decided on its values whether names are on or off: "The Treaty of Versailles was signed around 1918." stays undecided. Many real differences and matches stay undecided: a one-word name ("Canberra" for "Sydney"), a place given after a preposition alone, a verb whose forms stem apart ("wrote" and "written"), a title of two capitalised words ("the Vice President"), and the long form "Joe Biden is the president of the United States.", whose quote then names nothing the claim lacks, while "Joe Biden is the president." and "Joe Biden is the president of the US." are shown as different.

Two outcomes rest on the grader's proposal, since code reads neither whether a full name in the quote bears on the claim nor the quote's tense. A differs stands on any full name the quote holds that the claim lacks, one named in passing among them: "Paris is the capital of France." against the quote "the capital of France and its largest city, home to the Eiffel Tower" is shown as different when the grader proposes differs, and a quote that names "the United States" in passing does the same for a claim about an office. "Joe Biden is the president." against the quote "Joe Biden was the president" matches when the grader proposes matches (row N24). The quote's tense alone never shows a claim said in the present as different (row N24b): a fact that held only in the past is shown as different only when the quote also holds a number or date the reader compares, or a full name the claim lacks, as row N1's quote does through "United States". The value-agreement vectors pin these outcomes, so a change to any of them is made on purpose. Since 0.7.2 every decision says which branch decided it, basis: 'values' | 'quote': the rows of the names path, these two outcomes among them, come back quote, and a caller that shows a relation to anyone but the speaker can hold a quote decision back.

A search phrase

phraseProblem holds a phrase a public source is searched with to the caller's limits:

import { phraseProblem } from '@framers/agentos-ext-grounding-guard/values';

const phrase = 'when did the city of lisbon first host the web summit conference';
phraseProblem(phrase, { maxWords: 12, maxChars: 120 }); // null: it may be sent
phraseProblem(phrase, { maxWords: 6, maxChars: 80 }); // 'search_too_long'
phraseProblem('Treaty/Versailles', { maxWords: 6, maxChars: 80 }); // 'search_characters'

A phrase may be sent when it holds 1 to maxWords words, counted as words counts them (every character that is part of no word separates two words, so "O'Brien" is two), at most maxChars characters, and letters, digits, spaces and .,'()-:& alone. Otherwise the answer is the first rule it breaks, in that order: search_empty, search_too_long or search_characters. claimProblem holds a claim's topic phrase to it at 6 words and 80 characters.

Verbatim quotes

verifyQuotes holds each quote to a verbatim span of the source it names:

import { fold, verifyQuotes } from '@framers/agentos-ext-grounding-guard';

const sources = [
  'The Treaty of Versailles was a peace treaty signed on 28 June 1919.',
  { content: 'Latency is 1.25 ms, and the treaty’s terms were “harsh”.' },
];

verifyQuotes([{ text: 'signed on 28 June 1919', sourceIndex: 0 }], sources);
// [{ found: true, sourceIndex: 0, start: 44, end: 66 }]

verifyQuotes([{ text: 'signed on 28 Ju', sourceIndex: 0 }], sources)[0].found;
// false: the quote cuts a word

const quote = { text: 'the treaty\'s terms were "harsh"', sourceIndex: 1 };
verifyQuotes([quote], sources)[0].found; // false: the source's quotation marks are typographic
verifyQuotes([quote], sources, { normalize: fold })[0].found; // true

The comparison is character for character and case-sensitive, in the source's text with its white space collapsed, and a quote that cuts a word or a number is not found. start and end are offsets into the collapsed source, both -1 when the quote is not found. A quote is looked for only in the source its sourceIndex names, and an index the list does not hold is not found. A source is a string or an object with a content string, such as a retrieved chunk. normalize reads both sides before they are compared and must keep one character for one, so the offsets hold; fold maps typographic quotation marks to their plain forms.

A line within a span

lineWithinSpan(line, span) checks whether a line may stand beside a quoted span using its own words:

import { lineWithinSpan } from '@framers/agentos-ext-grounding-guard/values';

const span = 'Q3 revenue was 1.2M, up 18 percent.';
lineWithinSpan('Revenue was 1.2M, up 18 percent.', span); // true
lineWithinSpan('Revenue was not up 18 percent.', span); // false
lineWithinSpan('Revenue was up 18 points.', span); // false
lineWithinSpan('Revenue was $1.2M.', span); // false

The line needs at least one word, and each word must occur in the span, compared in lower case. Negations must match by form and count. Each number must occur with the same sign and, when the line has a following word in the next 60 code points, the same following word. A number at the end of a line may omit that word. Numbers use NUMBER and numberOf, with ASCII digits; word membership still applies to their written digit groups.

Every %‰°+−±$€£¥×÷<>=≤≥≠ symbol and every other currency sign in the line must occur in the span. Words, numbers and symbols may be reordered or repeated, except that negation counts must stay equal. This is a lexical check, so it does not establish who did what; show the line beside its span.

Sentence check

checkSentence holds a line, such as a summary's sentence or an answer's, to the words it quotes; checkLabel holds a title or a tag to the same terms:

import { checkLabel, checkSentence } from '@framers/agentos-ext-grounding-guard/values';

const turn = 'Yes, we grow the budget by ten percent next quarter.';
const quotes = [{ quote: 'we grow the budget by ten percent', source: turn }];

checkSentence('We grow the budget by ten percent.', quotes); // null: the line stands on its quote
checkSentence('Cut the budget by 20 percent.', quotes); // 'beyond_quotes': "cut" is not the quote's
checkSentence('We do not grow the budget by ten percent.', quotes); // 'negation_differs'
checkSentence('We will not grow the budget by ten percent.', quotes); // 'qualifier_not_quoted': "will"

checkLabel('Budget', quotes); // null
checkLabel('Acme budget review', quotes); // 'name_not_quoted'

Each quote is at least minQuoteChars characters long (20 when not given) and stands in its source: its words in order and next to each other, in any letter case and spacing, with the source's own symbols (<, =, % and the like) between them, while quotation marks, emphasis marks, commas, stops, brackets and dashes may differ. A quote also keeps every negation of the sentence it stands in: "the gateway supports IPv6" cut out of "The team never confirmed that the gateway supports IPv6." is quote_drops_negation. Where a quote stands more than once, its first stand that keeps them is read. A blank line ends a source's sentence; a single line break does not, since wrapped text breaks mid-sentence.

Each sentence of the line then needs content words, and at least support of them (0.7 when not given) are words of the quotes, compared without their endings ("-ing" past five characters, "-ed" and "-es" past four, "-s" past three but not "-ss") or, in a script written without spaces (isUnspacedScript), by neighbouring pairs of characters (letterPairs). The term checks come last, in this order:

| Problem | The line holds | |---|---| | name_not_quoted | a capitalised word of two characters or more that is not one of STOPWORDS, or an acronym ("US", "U.K."), that no quote holds | | symbol_not_quoted | a comparison, arrow, currency, percent or arithmetic symbol the quotes lack in that order; quotes that overlap in one source lend a symbol they share once | | number_not_quoted | a number, read whole with its sign, that no quote holds: 0.5 is not 5, and -20 is not 20 | | qualifier_not_quoted | a word of QUALIFIERS (direction, comparison, modality, quantity, order, numbers written as words) no quote holds, or a verb of QUALIFIER_FAMILIES whose family no quote holds, so "increases" never stands on "decreases" | | negation_differs | negations that, counted, are not the quotes' taken together; quotes that overlap in one source share theirs once |

checkSentence answers the first problem found, a SentenceProblem, or null when the line stands. checkLabel runs the term checks alone: it asks no share of content words and no length of a quote, and each quote still stands in its source and keeps its sentence's negations. A label's negations equal its quotes', so "approved" never stands on "not approved"; hold a label to the quotes it rests on, since a negation of any quote passed counts against it. With no quotes, a label passes only when it holds no name, symbol, number, qualifier or negation.

The checks are lexical: a line that swaps who did what can pass them, so show each line with its quotes.

NLI

createNliFunction() scores one pair, premise first, on a local model, in the shape CitationVerifier's nliFn takes:

import { CitationVerifier } from '@framers/agentos';
import { createNliFunction } from '@framers/agentos-ext-grounding-guard';

const nli = createNliFunction();

const answer = await nli('A man is eating pizza', 'A man eats something');
// answer.label: 'entailment', 'contradiction' or 'neutral'
// answer.score: that label's score; answer.scores: all three, which sum to 1

const verifier = new CitationVerifier({ embedFn, nliFn: nli }); // embedFn: your batch embedding function

await nli.dispose();

The premise is the source and the hypothesis is the claim, the order the model card's examples use. The pair goes to the model as the tokenizer's text and text_pair, and the labels come from the model's own id2label. A premise that takes the pair past the model's window (512 tokens for the default model, the pair's special tokens counted) is cut by tokens at its end, so the whole hypothesis stays in the pair: transformers.js cuts a pair that is too long at its tail, which would drop the claim. transformers.js (@huggingface/transformers, an optional dependency) is imported and the model loaded when the first pair is asked: the first load downloads the model's files, and later loads read them from transformers.js's cache. A load that fails is not kept, so the next pair loads again. dispose releases the model, and resolves with nothing to release after a failed load. GroundingChecker and the guardrail score every claim through the same function, with the source chunk as the premise.

| Option | Default | What it chooses | |---|---|---| | modelId (nliModelId in the guardrail's and GroundingChecker's options) | Xenova/nli-deberta-v3-small | The model: a sequence-classification model with ONNX weights that transformers.js can load. The default is the ONNX export of cross-encoder/nli-deberta-v3-small. | | dtype | q8 | Which weight file loads: q8 is the 8-bit quantised file (onnx/model_quantized.onnx, about 172 MB for the default model), fp32 the full-precision one (onnx/model.onnx, about 568 MB). A model repository without onnx/model_quantized.onnx needs dtype: 'fp32': cross-encoder/nli-deberta-v3-small, for one, holds its 8-bit weights as onnx/model_qint8_*.onnx and onnx/model_quint8_avx2.onnx, which no dtype names. | | quantized (the guardrail's and GroundingChecker's options, not createNliFunction's) | | Deprecated: use dtype. quantized: false is dtype: 'fp32'. |

Examples

• test/ — claim-extraction and entailment scoring fixtures

Lazy loading and optional install

This package is an optional dependency of @framers/agentos-extensions-registry. The registry ships catalog metadata; createCuratedManifest() calls import.meta.resolve() per entry and silently skips anything not installed. npm install @framers/agentos-ext-grounding-guard is the gate.

The NLI model does not load at activation. The checker keeps one NLI function under the agentos:grounding:nli-pipeline key in SharedServiceRegistry, and transformers.js and the model file load only when the first claim is checked against a source. The claim extractor runs first against extracted claims; only claims that pass extraction reach NLI.

GuardrailOutputPayload carries ragSources?: RagRetrievedChunk[] when the response was generated from RAG context. The grounding guardrail uses these as the entailment-source set; without them, the claim extractor still runs but per-claim entailment falls back to whatever passages the host provides.

Streaming checks are on by default (enableStreamingChecks): the guardrail registers with config.evaluateStreamingChunks = true, checks each completed factual sentence against the sources as the response streams, and checks every claim of the whole response at the end. Set enableStreamingChecks: false to check the final response alone. It runs in Phase 2 of the two-phase dispatcher. Below-threshold claims return FLAG by default; configure contradictionAction: 'block' to escalate to BLOCK.

For the full DI model and end-to-end walkthrough, see How extensions stay optional and lazy and the auto-loading guide.

License

Apache 2.0 — see the repo root LICENSE.