@framers/agentos-ext-public-sources

Public sources for AgentOS citation checks. AgentOS's CitationVerifier grades a claim against the passages it is given; this library fetches those passages from a public source and keeps their attribution. It ships English Wikipedia's provider behind a PublicSourceProvider interface: a search by a short topic phrase, a read of a hit's page HTML by its key, and the article as blocks of plain text with the page's own title, the reference markers, maintenance tags, tables, figures, quotations, code, mathematics and navigation removed.

citationSources(document, cut) turns a fetched article into passages, one for each sentence your cut finds in each block. A passage is a VerificationSource (content, title, url) that also carries the site, the article's key, its revision, the time it was read, its licence, and the sentence's block and offsets. The library holds no sentence rule of its own: pass the one the rest of your pipeline uses, so the claims and the passages are cut the same way.

Installation

npm install @framers/agentos-ext-public-sources

It needs Node.js 22 or later, as @framers/agentos does. Its one runtime dependency is htmlparser2; @framers/agentos is a peer.

The example below takes its sentence rule, sentenceSpans, from the values entry of the grounding guard, a package of its own. Install it beside this one to run the example as written, or pass any function that answers a block's sentences as offsets:

npm install @framers/agentos-ext-grounding-guard

Example

import { citationSources, SourceLimiter, WikipediaSourceProvider } from '@framers/agentos-ext-public-sources';
import { sentenceSpans } from '@framers/agentos-ext-grounding-guard/values';

const wikipedia = new WikipediaSourceProvider({
  client: { name: 'MyApp', version: '1.2.0', contact: 'https://example.org/contact' },
  limiter: new SourceLimiter(), // one for every caller on this host: Wikimedia's limits apply per client
});
const found = await wikipedia.search('Treaty of Versailles', { limit: 2 });
if (found.kind === 'hits' && found.hits[0]) {
  const page = await wikipedia.read(found.hits[0]);
  if (page.kind === 'document') {
    const passages = citationSources(page.document, sentenceSpans); // VerificationSource[] with attribution
  }
}

Reading any page

htmlToBlocks(html, { mode: 'page' }) reads a page that is not an encyclopedia article, such as a page a citation names. It keeps the tables, figures, quotations, code and notes the page shows, and makes each table cell, caption, quotation and code block a block of its own, so two cells never read as one sentence. It still drops the page's own title, scripts, styles, templates, inline SVG, mathematics, elements with the hidden attribute, and Wikipedia's reference markers, maintenance tags, reference lists and navigation boxes. The article mode, the default, is the one the Wikipedia provider reads with.

import { htmlToBlocks } from '@framers/agentos-ext-public-sources';

const html = '<table><tr><th>Plan</th><td>$20 per month</td></tr></table>';
htmlToBlocks(html, { mode: 'page' }); // ['Plan', '$20 per month']
htmlToBlocks(html); // []: the article mode drops tables

Outcomes

search answers one of:

• { kind: 'hits', hits }: at most limit hits (2 by default), each with a key and a title;
• { kind: 'limited' }: the limiter refused, or your onSend did, and nothing was sent;
• { kind: 'failed', status }: the HTTP status when an answer came, null for a transport failure, the deadline, or a phrase no address can carry (one holding half of a surrogate pair), for which nothing is sent.

read answers { kind: 'document', document }, { kind: 'limited' }, or { kind: 'skipped', status } for an answer that is not 200 HTML, one over the size caps, a transport failure, the deadline, or a key no address can carry. Neither method throws for a refusal or a failure; aborting your own signal throws its reason.

onSend runs once a search has its slot and before it is sent. Throw new LimiterRefused('caller') from it to hold searches to a quota of your own; the search then answers limited with nothing sent and none of the limiter's starts spent. The limiter stamps a search's start when the search is sent, after onSend has settled, so an onSend that takes a while never brings two requests closer than the spacing. A search can still answer limited after onSend has run, with nothing sent: when a 429 or a 503 blocked the limiter meanwhile, or when the spacing would take its wait past the 2 seconds (waitMs); the time onSend takes is not counted as waiting.

Wikimedia's limits and the User-Agent

Every request runs through a SourceLimiter: at most 3 at once, starts at least 250 ms apart (under 5 a second), at most 200 starts in any 60 seconds, a wait of at most 2 seconds for a slot or the spacing, and, after a 429 or a 503, a block until Retry-After has passed (5 seconds when the header is missing, an hour at most). The 3 at once, the 5 a second, the 200 a minute and the 5 seconds are what Wikimedia's rate limits and robot policy give an unauthenticated client with a compliant User-Agent; the 2-second wait and the hour's ceiling are this library's own.

The limits apply per client, and a limiter counts only the requests of its own process. A provider makes a limiter of its own when none is passed, so give every provider in a process the same one, and when several processes on one host read Wikipedia, send their requests through one of them.

The provider sends the User-Agent <name>/<version> (<contact>), the form of Wikimedia's User-Agent policy with its library part left out, as the policy allows. Give your client's own name and version and a contact the policy accepts: a website, an email address or a wiki user. Every request also sends Accept-Encoding: gzip, as the robot policy asks, follows no redirect and sends no cookie.

Sizes and the deadline

| Option | Default | What it bounds | |---|---|---| | timeoutMs | 3,000 | each request, from sending it to reading its body | | searchMaxBytes | 65,536 | a search answer's body | | pageMaxSentBytes | 1,500,000 | the content-length a page answer declares | | pageMaxBytes | 6,000,000 | a page's body once unzipped | | textBudget | 300,000 | the characters of text kept from a page |

A page answer may declare no length (a chunked one never does), so pageMaxBytes is the cap that bounds every read.

Attribution

Wikipedia's text is under CC BY-SA 4.0, and every document and passage carries it as licence (WIKIPEDIA_LICENCE). Wherever you show a passage, show with it:

• the sentence, as content holds it;
• a link to the article (url), named by its title;
• "CC BY-SA 4.0", linked to licence.url;
• that the text was modified: the library removes reference markers and maintenance tags and collapses white space, and CC BY-SA 4.0 asks a reuser to "indicate if You modified the Licensed Material" (section 3(a)(1)(B) of its legal code).

Wikipedia's page on reusing its content describes the attribution in full. revision and readAt name the exact text a passage came from.

The page path

The provider reads page HTML from /api/rest_v1/page/html/{key} (PAGE_PATH), the cached interface the robot policy names. RESTBase, the service behind it, is being deprecated: pass pagePath: '/w/rest.php/v1/page/{key}/html' to read the same HTML from MediaWiki's own REST API. A read uses the search hit's key, so it follows no redirect; a title that redirects answers 307 at both paths, and the read answers skipped.

API

| Export | What it is | |---|---| | PublicSourceProvider | The interface a public source implements: site, search(phrase, options), read(hit, options) | | WikipediaSourceProvider | English Wikipedia's provider, or another wiki's through origin | | SourceLimiter, LimiterRefused, retryAfterMs | The limiter, its refusal and the Retry-After reading | | htmlToBlocks(html, { budget, mode }) | HTML as blocks of plain text, white space collapsed, in NFC: an article's prose (article, the default) or everything a page shows (page) | | citationSources(document, cut) | A fetched article as passages with their attribution and place | | wikipediaUserAgent, articleUrl, WIKIPEDIA_LICENCE, SEARCH_PATH, PAGE_PATH | The User-Agent, the article's address, the licence and the paths the provider uses | | SourceHit, SourceDocument, CitationSource, SourceLicence, SearchOutcome, ReadOutcome, CallOptions, WikipediaOptions, WikipediaClient, LimiterOptions, BlockMode, HtmlToBlocksOptions | The types of the hits, documents, passages, outcomes and options |

License

Apache-2.0