semantic-stream

semantic-stream is a Node.js ESM module for building circular text streams from Wikipedia pages, news articles, and a YP/comment stream.

The package is now structured as a normal npm module:

• canonical source lives in src/
• public entry point is src/index.js
• tests live in test/
• live smoke tests are opt-in and gated by environment variables

3.0.5

• initStreams can skip titles containing configured filter words

3.0.4

• serializes Wikipedia requests per language and spaces them 250ms apart

3.0.2

• identifies requests with a configurable Wikipedia user agent
• retries Wikipedia 429 and 503 responses with exponential backoff
• honors Retry-After when the HTTP client exposes it
• avoids amplifying a rate limit with fallback searches
• initializes multiple streams sequentially by default
• supports normalized News API request options

What It Does

The module exposes stream classes that:

• fetch content from a source
• extract links or articles
• build circular link pools
• return the next item from the pool

The main use case is semantic text generation or exploration where the next result is derived from linked content rather than random selection.

Install

npm install semantic-stream

Requirements

• Node.js >= 22
• ESM-compatible runtime

Environment

Create a .env file if you want live news support:

NEWS_API_KEY=your-newsapi-key
SEMANTIC_STREAM_USER_AGENT=your-app/1.0 (https://example.com/contact)

The live news test and NewsStream both read NEWS_API_KEY from the environment. SEMANTIC_STREAM_USER_AGENT is optional. Set it to identify your application and provide contact information to Wikimedia. A package-specific default is used when the variable is absent.

Package Entry

Import from the package root:

import pkg from 'semantic-stream';

const { WordStream, NewsStream, YPStream, initStreams } = pkg;

You can also use named imports:

import { WordStream, NewsStream, YPStream, initStreams } from 'semantic-stream';

Public API

WordStream

Wikipedia-backed stream.

const stream = new WordStream('mami', 'de');
await stream.start();
const next = await stream.getNext();

Constructor:

new WordStream(word = 'maus', lang = 'en', options = {})

Supported options:

• circularLinkType: link selection method used by CircularLinks
• userAgent: overrides SEMANTIC_STREAM_USER_AGENT
• requestPolicy: overrides retry settings such as maxAttempts, baseDelayMs, maxDelayMs, and minIntervalMs (defaults to 250)

Methods:

• start()
• getArticle(title)
• getNext()
• searchArticle(str)
• check(wikijsResult)

NewsStream

News-backed stream.

const stream = new NewsStream({});
await stream.start();
const next = await stream.getNext();

Constructor:

new NewsStream(options = {})

Supported options for the News API request:

• q
• searchIn
• sources
• domains
• excludeDomains
• from
• to
• language
• sortBy
• pageSize
• page
• noCache

Accepted shapes:

• searchIn, sources, domains, excludeDomains can be a string or an array of strings
• from and to can be ISO strings or Date objects
• sortBy accepts relevancy, popularity, or publishedAt

Defaults:

• language: en
• pageSize: 100
• page: 1
• noCache: true for live requests
• all other request fields are optional and may be omitted

Methods:

• start()
• getNews()
• getSources()
• getNext()
• check(articles)

YPStream

YP/comment stream.

const stream = new YPStream({ startWord: 'example' });
await stream.start();
const next = await stream.getNext();

Constructor:

new YPStream(options = {})

Methods:

• start()
• getComment()
• getNext()
• check(comments)

initStreams

Initializes multiple streams in one call.

import { initStreams } from 'semantic-stream';

const streams = await initStreams([
  ['mami', 'de'],
  [':NEWS', {}],
  [':YP', { startWord: 'Feedback' }]
], {
  filter: ['DOI', 'ISBN']
});

Input format:

• [word, lang] creates a WordStream
• [':NEWS', options] creates a NewsStream
• [':YP', options] creates a YPStream

Optional initStreams options:

• circularLinksGetNext: custom getNext() implementation bound onto each stream’s CircularLinks instance
• filter: array of case-insensitive words; a matching title is consumed and skipped, then getNext() advances to the next item
• wordStreamOptions: options forwarded to every WordStream
• streamStartDelayMs: delay between sequential stream starts; defaults to 1000
• initializeSequentially: set to false only when parallel startup is intentional
• registerExitHandlers: set to false when the caller owns process lifecycle handling

Important Behavior

• WordStream.getNext() may reload more content when the pool gets small.
• NewsStream.getNews() caches identical in-memory requests for the current process.
• CircularLinks controls which item is returned next.
• getNextClassic returns items in insertion order.
• getNextUnique returns each title once.

Tests

Run the deterministic suite:

npm test

This covers the local, stubbed behavior and does not require external API access.

Live Smoke Tests

The repo also has opt-in live tests.

Live News

Requires NEWS_API_KEY and RUN_LIVE_TESTS=1:

RUN_LIVE_TESTS=1 NEWS_API_KEY=your-key node --test test/news.live.test.js

Live Wikipedia

Requires RUN_LIVE_TESTS=1:

RUN_LIVE_TESTS=1 node --test test/streams.live.test.js

The live Wikipedia smoke test fetches 10 live getNext() results for mami/de.

Repository Layout

src/
  index.js
  core/CircularLinks.js
  streams/WordStream.js
  streams/NewsStream.js
  streams/YPStream.js
  streams/onExit.js
  utils/wiki-parse-fkt.js
  utils/wiki-page-links.js
  utils/news-article-groups.js
  utils/news-request.js
  utils/wikipedia-request-policy.js
  utils/word-stream-context.js
  utils/word-stream-reload.js
test/
  *.test.js
  *.live.test.js

Development Notes

The test suite is designed to pin current legacy behavior before refactoring.

If you change stream logic, update or add tests first where possible.

If you add a new source or stream:

1. put the implementation under src/
2. expose it from src/index.js
3. add a deterministic test
4. add a live test only if the live path is useful and safe

Agent Workflow

If a bot is working from this README, use this order:

1. Read package.json and src/index.js
2. Read the relevant file under src/
3. Check the corresponding test/*.test.js
4. Make the smallest change that preserves the current tests
5. Run npm test
6. If the change touches live behavior, run the matching live test with RUN_LIVE_TESTS=1
7. Keep the published surface in src/ only

License

Beerware. If you like the project, buy the author a beer sometime.