@vertana/context-web

Web context gathering for Vertana — fetch and extract content from linked pages to provide additional context for translation.

Features

• fetchWebPage: A passive context source that fetches a single URL and extracts the main content using Mozilla's Readability algorithm.
• fetchLinkedPages: A required context source factory that extracts all links from the source text and fetches their content.
• searchWeb: A passive context source that performs a web search (DuckDuckGo Lite) and returns a list of results (title, URL, snippet).
• extractLinks: A utility function to extract URLs from text in various formats (plain text, Markdown, HTML).

Installation

Deno

deno add jsr:@vertana/context-web

npm

npm add @vertana/context-web

pnpm

pnpm add @vertana/context-web

Usage

import { translate } from "@vertana/facade";
import { fetchLinkedPages, fetchWebPage, searchWeb } from "@vertana/context-web";
import { openai } from "@ai-sdk/openai";

const text = `
Check out this article: https://example.com/article
It explains the concept in detail.
`;

const result = await translate(openai("gpt-4o"), "ko", text, {
  contextSources: [
    // Automatically fetch all links in the text
    fetchLinkedPages({ text, mediaType: "text/plain" }),
    // Allow LLM to search the web and fetch URLs on demand
    searchWeb,
    fetchWebPage,
  ],
});

License

MIT License