@theloremi/toolkit

Web scraping, RSS parsing, markdown processing, SEO audit, JSON-LD structured data, and URL utilities. A zero-dependency-on-frameworks toolkit used across The Loremi product ecosystem.

Install

npm install @theloremi/toolkit

Quick Start

Scrape metadata from a URL

import { scrape } from '@theloremi/toolkit'

const meta = await scrape('https://example.com/article')
console.log(meta.title)       // page title
console.log(meta.og.image)    // Open Graph image
console.log(meta.twitter.card) // Twitter card type

Extract article content

import { extract } from '@theloremi/toolkit'

const article = await extract('https://example.com/article')
console.log(article.title)       // cleaned article title
console.log(article.textContent) // plain text body
console.log(article.excerpt)     // short excerpt

Parse RSS feeds

import { parseFeeds } from '@theloremi/toolkit'

const result = await parseFeeds(
  [
    { name: 'TechCrunch', url: 'https://techcrunch.com/feed/', category: 'tech' },
    { name: 'Ars Technica', url: 'https://feeds.arstechnica.com/arstechnica/index', category: 'tech' },
  ],
  { limit: 10, page: 1, search: 'AI' }
)

console.log(result.articles)          // FeedArticle[]
console.log(result.pagination.total)  // total matching articles

API Reference

Scrape & Extract

scrape(url, options?): Promise<ScrapeResult>

Fetch a URL and extract metadata using cheerio. Returns title, description, keywords, author, Open Graph tags, Twitter Card tags, canonical URL, favicon, publish/modified dates, first image, and first paragraph.

Options (ScrapeOptions):

| Option | Type | Default | Description | |--------|------|---------|-------------| | timeout | number | 10000 | Request timeout in ms | | retries | number | 3 | Number of retry attempts | | userAgent | string | random | Custom User-Agent header | | maxSize | number | 5MB | Maximum response size in bytes |

extract(url, options?): Promise<ExtractResult>

Fetch a URL and extract the main article content using Mozilla Readability + JSDOM. Returns title, content (HTML), textContent (plain text), excerpt, byline, length, and siteName. Throws if extraction fails.

Takes the same ScrapeOptions as scrape().

Feeds

parseFeeds(feeds, options?): Promise<FeedResult>

Parse multiple RSS/Atom feeds in parallel with per-feed error isolation.

Parameters:

• feeds -- FeedSource[] with { name, url, category } per feed
• options -- FeedOptions:

| Option | Type | Default | Description | |--------|------|---------|-------------| | limit | number | 20 | Articles per page | | page | number | 1 | Page number | | search | string | -- | Filter by keyword in title/content | | language | string | 'en' | Language filter | | timeout | number | -- | Per-feed timeout in ms |

Returns FeedResult with articles, lastUpdated, sources, and pagination.

Markdown

extractHeadings(markdown, options?): TocHeading[]

Extract TOC headings from markdown/HTML. Handles both # Heading and <h1>Heading</h1> syntax. Ignores headings inside code fences. Deduplicates IDs automatically.

Options: { minLevel?: number, maxLevel?: number } (defaults: 1-6).

extractHeadingIds(markdown): string[]

Shorthand that returns just the ID strings from extractHeadings.

injectHeadingIds(markdown, options?): InjectHeadingIdsResult

Rewrite markdown so every heading becomes <hN id="slug">. Returns { content, headings }.

splitMarkdownForAds(markdown, chunkCount?, options?): string[]

Split markdown at safe paragraph boundaries into chunkCount chunks. Useful for inserting ads or CTAs between sections.

slugify(text): string

Convert any text to a URL-safe slug.

stripInlineMarkdown(input): string

Strip bold, italic, code, links, strikethrough, and images to plain text.

mdInlineToHtml(input): string

Convert markdown inline formatting (bold, italic, code, links, strikethrough) to HTML.

SEO

checkSEOIssues(): SEOIssue[]

Browser-only. Audits the current DOM for common SEO issues: missing title, description, canonical, H1, viewport, lang attribute, empty links, missing image alt text, Open Graph tags, and heading hierarchy. Returns an empty array in non-browser environments.

logSEOIssues(): void

Browser-only. Runs checkSEOIssues() and logs results to the console in a grouped format with color.

generateSEOReport(): string

Browser-only. Returns a plain-text SEO audit report string.

Structured Data (JSON-LD)

generateArticleStructuredData(article: ArticleData): object

Generate NewsArticle JSON-LD. Includes headline, description, author, publisher, dates, images, tags, word count.

generateReviewStructuredData(review: ReviewData): object

Generate Review + Product JSON-LD with rating, pros/cons, and optional price/availability.

generateFAQStructuredData(faqs): object

Generate FAQPage JSON-LD from an array of { question, answer }.

generateBreadcrumbStructuredData(breadcrumbs): object

Generate BreadcrumbList JSON-LD from an array of { name, url }.

URL Utilities

sanitizeSlug(slug): string

Lowercase, strip special characters, collapse hyphens, trim leading/trailing hyphens.

sanitizeUrl(url, baseUrl?): string

Normalize a URL path: remove trailing slashes, double slashes, resolve relative paths.

generateArticleUrl(slug): string

Generate a canonical /articles/{slug} path from a slug.

needsUrlRedirect(url): boolean

Returns true if a URL has issues requiring a redirect (trailing slash, uppercase, encoded spaces, double slashes).

getRedirectUrl(url, baseUrl?): string

Returns the corrected/canonical URL for a path that needs a redirect.

HTTP

fetchWithRetry(url, options?): Promise<{ text, status, headers }>

Fetch with automatic retries, exponential backoff, configurable timeout, UA rotation, and max response size. Does not retry on 4xx errors.

Types

All types are exported from the package entry point:

import type {
  ScrapeOptions,
  ScrapeResult,
  OGMetadata,
  TwitterMetadata,
  ExtractResult,
  FeedSource,
  FeedArticle,
  FeedPagination,
  FeedResult,
  FeedOptions,
  TocHeading,
  InjectHeadingIdsResult,
  SEOIssue,
  ArticleData,
  ReviewData,
} from '@theloremi/toolkit'

Ecosystem

Part of The Loremi SDK family:

• @theloremi/toolkit -- this package
• @theloremi/cms-client -- headless CMS client SDK
• @theloremi/cleepa -- digital asset management SDK
• @theloremi/mailer -- zero-config transactional email SDK

License

MIT -- The Loremi Ltd, 2026.