semantic-stream

semantic-stream is a Node.js ESM module for building circular text streams from Wikipedia pages, news articles, and a YP/comment stream.

The package is now structured as a normal npm module:

• canonical source lives in src/
• public entry point is src/index.js
• tests live in test/
• live smoke tests are opt-in and gated by environment variables

What It Does

The module exposes stream classes that:

• fetch content from a source
• extract links or articles
• build circular link pools
• return the next item from the pool

The main use case is semantic text generation or exploration where the next result is derived from linked content rather than random selection.

Install

npm install semantic-stream

Requirements

• Node.js >= 22
• ESM-compatible runtime

Environment

Create a .env file if you want live news support:

NEWS_API_KEY=your-newsapi-key

The live news test and NewsStream both read NEWS_API_KEY from the environment.

Package Entry

Import from the package root:

import pkg from 'semantic-stream';

const { WordStream, NewsStream, YPStream, initStreams } = pkg;

You can also use named imports:

import { WordStream, NewsStream, YPStream, initStreams } from 'semantic-stream';

Public API

WordStream

Wikipedia-backed stream.

const stream = new WordStream('mami', 'de');
await stream.start();
const next = await stream.getNext();

Constructor:

new WordStream(word = 'maus', lang = 'en', options = {})

Methods:

• start()
• getArticle(title)
• getNext()
• searchArticle(str)
• check(wikijsResult)

NewsStream

News-backed stream.

const stream = new NewsStream({});
await stream.start();
const next = await stream.getNext();

Constructor:

new NewsStream(options = {})

Supported options for the News API request:

• q
• searchIn
• sources
• domains
• excludeDomains
• from
• to
• language
• sortBy
• pageSize
• page
• noCache

Accepted shapes:

• searchIn, sources, domains, excludeDomains can be a string or an array of strings
• from and to can be ISO strings or Date objects
• sortBy accepts relevancy, popularity, or publishedAt

Defaults:

• language: en
• pageSize: 100
• page: 1
• noCache: true for live requests
• all other request fields are optional and may be omitted

Methods:

• start()
• getNews()
• getSources()
• getNext()
• check(articles)

YPStream

YP/comment stream.

const stream = new YPStream({ startWord: 'example' });
await stream.start();
const next = await stream.getNext();

Constructor:

new YPStream(options = {})

Methods:

• start()
• getComment()
• getNext()
• check(comments)

initStreams

Initializes multiple streams in one call.

import { initStreams } from 'semantic-stream';

const streams = await initStreams([
  ['mami', 'de'],
  [':NEWS', {}],
  [':YP', { startWord: 'Feedback' }]
]);

Input format:

• [word, lang] creates a WordStream
• [':NEWS', options] creates a NewsStream
• [':YP', options] creates a YPStream

Optional initStreams options:

• circularLinksGetNext: custom getNext() implementation bound onto each stream’s CircularLinks instance

Important Behavior

• WordStream.getNext() may reload more content when the pool gets small.
• NewsStream.getNews() caches identical in-memory requests for the current process.
• CircularLinks controls which item is returned next.
• getNextClassic returns items in insertion order.
• getNextUnique returns each title once.

Tests

Run the deterministic suite:

npm test

This covers the local, stubbed behavior and does not require external API access.

Live Smoke Tests

The repo also has opt-in live tests.

Live News

Requires NEWS_API_KEY and RUN_LIVE_TESTS=1:

RUN_LIVE_TESTS=1 NEWS_API_KEY=your-key node --test test/news.live.test.js

Live Wikipedia

Requires RUN_LIVE_TESTS=1:

RUN_LIVE_TESTS=1 node --test test/streams.live.test.js

The live Wikipedia smoke test fetches 10 live getNext() results for mami/de.

Repository Layout

src/
  index.js
  core/CircularLinks.js
  streams/WordStream.js
  streams/NewsStream.js
  streams/YPStream.js
  streams/onExit.js
  utils/wiki-parse-fkt.js
  utils/wiki-page-links.js
  utils/news-article-groups.js
  utils/word-stream-context.js
  utils/word-stream-reload.js
test/
  *.test.js
  *.live.test.js

Development Notes

The test suite is designed to pin current legacy behavior before refactoring.

If you change stream logic, update or add tests first where possible.

If you add a new source or stream:

1. put the implementation under src/
2. expose it from src/index.js
3. add a deterministic test
4. add a live test only if the live path is useful and safe

Agent Workflow

If a bot is working from this README, use this order:

1. Read package.json and src/index.js
2. Read the relevant file under src/
3. Check the corresponding test/*.test.js
4. Make the smallest change that preserves the current tests
5. Run npm test
6. If the change touches live behavior, run the matching live test with RUN_LIVE_TESTS=1
7. Keep the published surface in src/ only

License

Beerware. If you like the project, buy the author a beer sometime.