@push.rocks/smartsitemap

🗺️ A comprehensive TypeScript sitemap library with a chainable builder API — supporting standard, news, image, video, and hreflang sitemaps with auto-splitting, streaming, validation, and RSS feed integration.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Install

pnpm install @push.rocks/smartsitemap

✨ Features

• 🔗 Chainable Builder API — Fluent, composable API where every method returns this
• 📰 News Sitemaps — Google News-compatible with proper namespace handling
• 🖼️ Image Sitemaps — Full image:image extension support
• 🎬 Video Sitemaps — Full video:video extension with all fields
• 🌍 hreflang / i18n — xhtml:link alternate language annotations
• 📑 Sitemap Index — Automatic splitting at 50K URLs with index generation
• 🌊 Streaming — Node.js Readable stream for million-URL sitemaps
• ✅ Validation — URL validation, size limits, spec compliance checks
• 📊 Statistics — URL counts, image/video/news counts, size estimates
• 📡 RSS/Atom Feed Import — Convert feeds to sitemaps (unique feature!)
• 🎙️ Feed Creation & Parsing — Create RSS 2.0, Atom 1.0, JSON Feed and Podcast RSS; parse RSS 0.9x/1.0/2.0, Atom 1.0 and OPML with a bounded parser that never expands entities; find the feed links of an HTML page, via the cross-runtime @push.rocks/smartsitemap/feed entry point
• 🤖 robots.txt Discovery — Read the sitemap URLs a site announces in its robots.txt
• 🔒 Public-Only Fetches — Feed, sitemap and robots.txt URLs are fetched from public addresses only, with a body cap per document
• 📄 YAML Config — Declarative sitemap definition from YAML
• 🗂️ Multi-Format Output — XML, TXT, JSON, gzipped buffer
• 🎨 XSL Stylesheets — Browser-viewable sitemaps
• 🔍 Bidirectional Parsing — Parse existing sitemaps back into structured data
• 💪 Full TypeScript — Complete type safety with exported interfaces

Quick Start

import { SmartSitemap } from '@push.rocks/smartsitemap';

// 3 lines to a valid sitemap 🚀
const xml = SmartSitemap.create()
  .addUrl('https://example.com/')
  .addUrl('https://example.com/about')
  .addUrl('https://example.com/blog')
  .toXml();

Output:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
  </url>
  <url>
    <loc>https://example.com/about</loc>
  </url>
  <url>
    <loc>https://example.com/blog</loc>
  </url>
</urlset>

Usage

🌐 Standard Sitemap with Full Control

import { SmartSitemap } from '@push.rocks/smartsitemap';

const xml = SmartSitemap.create({ baseUrl: 'https://example.com' })
  .setDefaultChangeFreq('weekly')
  .setDefaultPriority(0.5)
  .setXslUrl('/sitemap.xsl')
  .add({
    loc: 'https://example.com/',
    changefreq: 'daily',
    priority: 1.0,
    lastmod: new Date(),
  })
  .add({
    loc: 'https://example.com/products',
    changefreq: 'daily',
    priority: 0.9,
    images: [
      { loc: 'https://example.com/img/product1.jpg', title: 'Product 1' },
    ],
  })
  .add({
    loc: 'https://example.com/blog/post-1',
    lastmod: '2025-01-15',
    alternates: [
      { hreflang: 'de', href: 'https://example.com/de/blog/post-1' },
      { hreflang: 'fr', href: 'https://example.com/fr/blog/post-1' },
    ],
  })
  .toXml();

🔗 Builder from a URL Array

const builder = SmartSitemap.fromUrls([
  'https://example.com/',
  'https://example.com/about',
  'https://example.com/contact',
]);

const xml = builder
  .setDefaultChangeFreq('monthly')
  .toXml();

📰 News Sitemap

const xml = SmartSitemap.createNews({
  publicationName: 'The Daily Tech',
  publicationLanguage: 'en',
})
  .addNewsUrl(
    'https://example.com/news/breaking-story',
    'Breaking: TypeScript 6.0 Released!',
    new Date(),
    ['typescript', 'programming'],
  )
  .addNewsUrl(
    'https://example.com/news/another-story',
    'Node.js Gets Even Faster',
    new Date(),
  )
  .toXml();

📰 News Sitemap from RSS Feed

This is smartsitemap's killer feature — no other sitemap library does this:

// From a feed URL
const builder = SmartSitemap.createNews({
  publicationName: 'The Daily Tech',
  publicationLanguage: 'en',
});
await builder.importFromFeedUrl('https://thedailytech.com/rss/');
const xml = builder.toXml();

// Or as a one-liner with the static factory
const feedBuilder = await SmartSitemap.fromFeedUrl('https://example.com/rss/');
const feedXml = feedBuilder.toXml();

Feed URLs are fetched public-only (see Remote Fetches) and parsed with parseFeed of the /feed entry point: relative links resolve against the final URL after redirects, and a feed parseFeed refuses rejects with an Error that names the refusal code. mapItem receives the parsed item (IParsedFeedItem):

import { FeedImporter } from '@push.rocks/smartsitemap';

const urls = await FeedImporter.fromUrl('https://example.com/rss/', {
  newerThan: Date.now() - 7 * 24 * 3600 * 1000, // items without a date are kept
  limit: 100,
  mapItem: (item) => (item.link ? { loc: item.link, lastmod: item.publishedAt } : null),
});

📰 News Sitemap from Articles

Works seamlessly with @tsclass/tsclass IArticle objects from your CMS:

import type { content } from '@tsclass/tsclass';

const articles: content.IArticle[] = [/* from your CMS or database */];

const xml = SmartSitemap.fromArticles(articles, {
  publicationName: 'My Publication',
  publicationLanguage: 'en',
}).toXml();

🖼️ Image Sitemap

const xml = SmartSitemap.create()
  .add({
    loc: 'https://example.com/gallery',
    images: [
      { loc: 'https://example.com/img/photo1.jpg', title: 'Sunset' },
      { loc: 'https://example.com/img/photo2.jpg', caption: 'Mountain view' },
    ],
  })
  .toXml();

🎬 Video Sitemap

const xml = SmartSitemap.create()
  .add({
    loc: 'https://example.com/videos/tutorial',
    videos: [
      {
        thumbnailLoc: 'https://example.com/thumb.jpg',
        title: 'Getting Started with TypeScript',
        description: 'A comprehensive guide to TypeScript for beginners.',
        contentLoc: 'https://example.com/video.mp4',
        duration: 600,
        rating: 4.8,
        publicationDate: new Date(),
        tags: ['typescript', 'tutorial', 'programming'],
      },
    ],
  })
  .toXml();

🌍 hreflang / Internationalization

const xml = SmartSitemap.create()
  .add({
    loc: 'https://example.com/page',
    alternates: [
      { hreflang: 'en', href: 'https://example.com/page' },
      { hreflang: 'de', href: 'https://example.com/de/page' },
      { hreflang: 'fr', href: 'https://example.com/fr/page' },
      { hreflang: 'x-default', href: 'https://example.com/page' },
    ],
  })
  .toXml();

📑 Automatic Sitemap Index Splitting

When you exceed 50K URLs, smartsitemap automatically splits into a sitemap index:

const builder = SmartSitemap.create({
  baseUrl: 'https://example.com',
  maxUrlsPerSitemap: 45000, // default is 50000
});

// Add hundreds of thousands of URLs
for (const page of allPages) {
  builder.addUrl(page.url, page.lastModified);
}

const set = builder.toSitemapSet();
// set.needsIndex === true
// set.indexXml → '<?xml ...><sitemapindex>...'
// set.sitemaps → [
//   { filename: 'sitemap-1.xml', xml: '...' },
//   { filename: 'sitemap-2.xml', xml: '...' },
//   { filename: 'sitemap-3.xml', xml: '...' },
// ]

// Or build an index manually
const index = SmartSitemap.createIndex()
  .addSitemap('https://example.com/sitemap-blog.xml')
  .addSitemap('https://example.com/sitemap-products.xml', new Date())
  .toXml();

🌊 Streaming for Large Sitemaps

For sitemaps with millions of URLs that can't fit in memory:

import { createWriteStream } from 'fs';
import { createGzip } from 'zlib';
import { SitemapStream } from '@push.rocks/smartsitemap';

const stream = new SitemapStream();
const output = createWriteStream('/var/www/sitemap.xml.gz');
stream.pipe(createGzip()).pipe(output);

// Stream URLs from a database cursor
for await (const page of databaseCursor()) {
  stream.pushUrl({
    loc: page.url,
    lastmod: page.updatedAt,
    changefreq: 'weekly',
  });
}

stream.finish();

🔀 Merge, Dedupe, Filter & Sort

Combine multiple sitemap sources with powerful collection operations:

const blogSitemap = SmartSitemap.create()
  .setDefaultChangeFreq('weekly')
  .addFromArray(blogUrls);

const productSitemap = SmartSitemap.create()
  .setDefaultChangeFreq('daily')
  .addFromArray(productUrls);

const xml = SmartSitemap.create()
  .merge(blogSitemap)
  .merge(productSitemap)
  .dedupe()
  .filter(url => !url.loc.includes('/deprecated/'))
  .sort((a, b) => a.loc.localeCompare(b.loc))
  .toXml();

📄 YAML Configuration

Define sitemaps declaratively:

const yaml = `
baseUrl: https://example.com
defaults:
  priority: 0.5
urls:
  daily:
    - /
    - /blog
  weekly:
    - /docs
    - /tutorials
  monthly:
    - /about
    - /contact
  yearly:
    - /privacy
    - /terms
`;

const builder = await SmartSitemap.fromYaml(yaml);
const xml = builder.toXml();

A config may also list feeds: (url, type: news | standard, publicationName, publicationLanguage); they are fetched public-only. fromYaml(yaml, fetchOptions) and importFromYaml(yaml, fetchOptions) pass fetch options to those fetches.

✅ Validation

Catch errors before they reach search engines:

const result = SmartSitemap.create()
  .addUrl('not-a-valid-url')
  .add({ loc: 'https://example.com/', priority: 1.5 }) // out of range
  .validate();

console.log(result.valid);   // false
console.log(result.errors);
// [
//   { field: 'loc', message: 'Invalid URL: "not-a-valid-url"', url: 'not-a-valid-url' },
//   { field: 'priority', message: 'Priority must be between 0.0 and 1.0', url: 'https://example.com/' },
// ]

📊 Statistics

Get insight into your sitemap:

const stats = SmartSitemap.create()
  .addUrl('https://example.com/')
  .add({ loc: 'https://example.com/gallery', images: [{ loc: '/img/1.jpg' }] })
  .stats();

console.log(stats);
// {
//   urlCount: 2,
//   imageCount: 1,
//   videoCount: 0,
//   newsCount: 0,
//   alternateCount: 0,
//   estimatedSizeBytes: 750,
//   needsIndex: false,
// }

🗂️ Multi-Format Output

const builder = SmartSitemap.create()
  .addUrl('https://example.com/')
  .addUrl('https://example.com/about');

// XML (default)
const xml = builder.toXml();

// Plain text (one URL per line)
const txt = builder.toTxt();
// "https://example.com/\nhttps://example.com/about"

// JSON
const json = builder.toJson();

// Gzipped XML buffer (for serving compressed)
const gzipped = await builder.toGzipBuffer();

🔍 Parse Existing Sitemaps

Read and parse sitemaps back into structured data:

// From URL
const parsed = await SmartSitemap.parseUrl('https://example.com/sitemap.xml');
console.log(parsed.type); // 'urlset' or 'sitemapindex'
console.log(parsed.urls); // ISitemapUrl[]

// From XML string
const result = await SmartSitemap.parse(sitemapXmlString);

// Parse and get a pre-populated builder for modification
const builder = await SitemapParser.toBuilder(existingSitemapXml);
builder
  .addUrl('https://example.com/new-page')
  .filter(url => !url.loc.includes('/old/'))
  .toXml();

// Detect type without full parsing
SitemapParser.detectType('<urlset ...>');       // 'urlset'
SitemapParser.detectType('<sitemapindex ...>'); // 'sitemapindex'

parseUrl() fetches public-only (see Remote Fetches), with a body cap of 50 MiB, the sitemaps.org maximum of an uncompressed sitemap, and rejects when the server answers with a non-2xx status instead of parsing an error page.

🤖 Discover Sitemaps via robots.txt

RobotsTxtParser reads the Sitemap: lines of a robots.txt (sitemaps.org and RFC 9309): case-insensitive, comments stripped, duplicates removed, file order kept.

import { RobotsTxtParser, SmartSitemap } from '@push.rocks/smartsitemap';

const robots = await RobotsTxtParser.parseUrl('https://example.com/robots.txt');
for (const sitemapUrl of robots.sitemaps) {
  const parsed = await SmartSitemap.parseUrl(sitemapUrl);
  console.log(sitemapUrl, parsed.type);
}

// From a string
RobotsTxtParser.parse('User-agent: *\nSitemap: https://example.com/sitemap.xml');
// { sitemaps: ['https://example.com/sitemap.xml'] }

RobotsTxtParser.parseUrl() fetches public-only, with a body cap of 500 KiB (512,000 bytes, the parsing limit RFC 9309 requires), and rejects when the server answers with a non-2xx status.

Remote Fetches Are Public-Only

Every URL the main entry fetches (SitemapParser.parseUrl, SmartSitemap.parseUrl, RobotsTxtParser.parseUrl, the feed imports FeedImporter.fromUrl/fromUrlAsNews, SmartSitemap.fromFeedUrl, importFromFeedUrl of both builders, and the feeds: of a YAML config) goes through one fetch with @push.rocks/smartrequest's publicOnly() policy, because these URLs often come from someone else (a customer's robots.txt, a feed URL a user enters):

• Public addresses only, for the first URL and for every redirect: no loopback, private, link-local or cloud metadata address, and no special-use name such as localhost. Ports 80 and 443 only by default. The connection goes to exactly the address that was checked.

• Redirects are followed (up to 5 by default); the final URL is the feed's document URL, so its relative links resolve against it.

• The body is capped while it arrives and after decompression:

  | Document | Cap | | --- | --- | | feed | 10 MiB (10,485,760 bytes, the default maxInputLength of parseFeed) | | sitemap | 50 MiB (52,428,800 bytes) | | robots.txt | 500 KiB (512,000 bytes) |

• Timeouts: 10 s to connect (TCP and TLS), 15 s for the response headers and 30 s in total, including the body and redirects (smartrequest's publicOnly() defaults); fetchOptions.publicOnly.totalTimeout raises the total, connectTimeout and responseTimeout the others.

• The Content-Type charset decodes the body (a feed's bytes go to parseFeed with the header).

• Errors: a refused or failed request rejects with smartrequest's PublicOnlyError, whose code says why (address_not_public, host_not_allowed, port_not_allowed, response_too_large, too_many_redirects, dns_failed, connect_timeout, and the others the smartrequest readme lists); a non-2xx answer rejects with an Error naming the status.

• Runtimes: the main entry is Node.js-only; publicOnly() needs Node.js or Bun.

Every fetching method takes fetchOptions (IRemoteFetchOptions; for the feed imports it is IFeedImportOptions.fetchOptions). fetchOptions.publicOnly takes smartrequest's public-only settings (timeouts, maxRedirects, allowedPorts, allowedContentTypes, ca, resolve, allowAddress, allowSpecialUseDomains, allowSingleLabelNames), except maxResponseSize: the caps above are fixed. Widening the address policy is for a local test server or a private network you mean to reach:

import { RobotsTxtParser, type IRemoteFetchOptions } from '@push.rocks/smartsitemap';

// a test server on 127.0.0.1, reached under one name only
const fetchOptions: IRemoteFetchOptions = {
  publicOnly: {
    resolve: async (hostname) => (hostname === 'shop.example.com' ? ['127.0.0.1'] : []),
    allowedPorts: [port],
    allowAddress: (address, target) =>
      address.isPublic || (address.address === '127.0.0.1' && target.hostname === 'shop.example.com'),
  },
};
const robots = await RobotsTxtParser.parseUrl(`http://shop.example.com:${port}/robots.txt`, fetchOptions);

📡 Feeds (@push.rocks/smartsitemap/feed)

The /feed entry point creates RSS 2.0, Atom 1.0, JSON Feed 1.0 and Podcast RSS (iTunes tags and the Podcast 2.0 namespace), parses RSS 0.9x/1.0/2.0, Atom 1.0 and OPML, and finds the feed links of an HTML page. Unlike the Node.js-only main entry it runs in Node.js, Bun and browsers: it depends only on @tsclass/tsclass, htmlparser2 and entities, and it never touches the network: parsing takes a string or bytes, and fetching is the caller's. Deno is not supported. Generated feeds name @push.rocks/smartsitemap as their <generator>.

Creating a Basic Feed

import { Smartfeed } from '@push.rocks/smartsitemap/feed';

const smartfeed = new Smartfeed();

// Create a feed
const feed = smartfeed.createFeed({
  domain: 'example.com',
  title: 'Tech Insights',
  description: 'Latest insights in technology and innovation',
  category: 'Technology',
  company: 'Example Inc',
  companyEmail: '[email protected]',
  companyDomain: 'https://example.com'
});

// Add an item
feed.addItem({
  title: 'TypeScript 5.0 Released',
  timestamp: Date.now(),
  url: 'https://example.com/posts/typescript-5',
  authorName: 'Jane Developer',
  imageUrl: 'https://example.com/images/typescript.jpg',
  imageType: 'image/jpeg', // the RSS enclosure needs the image's MIME type
  imageLength: 48213, // optional: bytes; RSS writes length="0" (unknown) without it
  content: 'TypeScript 5.0 brings exciting new features...'
});

// Export as RSS, Atom, or JSON
const rss = feed.exportRssFeedString();
const atom = feed.exportAtomFeed();
const json = feed.exportJsonFeed();

Items Without an Image, with Full Content, Categories and a Thumbnail

The image is optional: an item without imageUrl gets no enclosure and no image in any format. content is the summary; contentHtml carries the full article HTML, categories the item's categories in order (a term, or a term with the taxonomy it belongs to), and thumbnail a small image written as Media RSS <media:thumbnail>:

feed.addItem({
  title: 'Release 5.0',
  timestamp: Date.now(),
  url: 'https://example.com/news/release-5?lang=en&ref=feed', // escaped on output
  authorName: 'Jane Developer',
  content: 'Version 5 is out.',
  contentHtml: '<p>Version 5 is out. <b>Highlights</b>: …</p>',
  categories: ['Release', { term: 'TypeScript', domain: 'https://example.com/topics' }],
  thumbnail: { url: 'https://example.com/thumbs/release-5.jpg', width: 320, height: 180 },
  id: 'news-release-5', // a guid that is not a URL: isPermaLink="false"
});

| Field | RSS 2.0 | Atom 1.0 | JSON Feed 1.0 | | --- | --- | --- | --- | | content | <description> | <content type="html">, or <summary type="html"> with contentHtml | content_html, or summary (as plain text) with contentHtml | | contentHtml | <content:encoded> (CDATA; a ]]> inside splits the section) | <content type="html"> | content_html | | categories | one <category> each, <category domain="…"> for a domain | one <category term="…"/> each, with scheme="…" for a domain | tags (the terms) | | thumbnail (url, width, height) | <media:thumbnail url width height> | <media:thumbnail url width height> | not written (JSON Feed has no thumbnail field) | | imageUrl, imageType, imageLength | <enclosure url length type>; imageType required | <link rel="enclosure"> with type and length when given | image | | id, idIsPermaLink | <guid isPermaLink> | <id> | id | | authorName | <author>companyEmail (authorName)</author> | <author><name> | author.name |

exportRssFeedString() throws for an item with imageUrl but no imageType: RSS 2.0 requires the enclosure's MIME type, and the writer does not guess it. Atom and JSON Feed need no type.

The Media RSS namespace (xmlns:media="http://search.yahoo.com/mrss/") is declared on the root element only when an item has a thumbnail; a thumbnail's width and height are optional positive integers (pixels). A category domain is any non-empty string, often a URL.

Stable Output

RSS <lastBuildDate> and the Atom feed <updated> default to the time of each export, so two exports of the same content differ. Set buildTimestamp (Unix milliseconds) for the same bytes, and so the same ETag, on every export, for example the newest item's timestamp:

const feed = smartfeed.createFeed({
  // ...the options above
  buildTimestamp: Math.max(...posts.map((post) => post.timestamp)),
});

buildTimestamp also sets the <lastBuildDate> of exportPodcastRss().

The XML writers escape every text and attribute value, write tab, LF and CR in them as character references, and drop the characters XML 1.0 forbids (C0 controls but tab, LF and CR; U+FFFE; U+FFFF; lone surrogates), also from CDATA sections, so the output is well-formed XML.

Custom Feed URL

You can specify a custom URL for your feed's self-reference instead of the default https://${domain}/feed.xml:

const feed = smartfeed.createFeed({
  domain: 'example.com',
  title: 'My Blog',
  description: 'Latest posts',
  category: 'Technology',
  company: 'Example Inc',
  companyEmail: '[email protected]',
  companyDomain: 'https://example.com',
  feedUrl: 'https://cdn.example.com/feeds/main.xml' // Custom feed URL
});

// The feedUrl will be used in:
// - RSS: <atom:link href="..." rel="self">
// - Atom: <link href="..." rel="self">
// - JSON Feed: "feed_url" field

This is particularly useful when your feed is hosted on a CDN or different domain than your main site.

Creating a Podcast Feed

import { Smartfeed } from '@push.rocks/smartsitemap/feed';

const smartfeed = new Smartfeed();

const podcast = smartfeed.createPodcastFeed({
  domain: 'podcast.example.com',
  title: 'The Tech Show',
  description: 'Weekly discussions about technology',
  category: 'Technology',
  company: 'Tech Media Inc',
  companyEmail: '[email protected]',
  companyDomain: 'https://example.com',
  // iTunes tags
  itunesCategory: 'Technology',
  itunesAuthor: 'John Host',
  itunesOwner: {
    name: 'John Host',
    email: '[email protected]'
  },
  itunesImage: 'https://example.com/artwork.jpg',
  itunesExplicit: false,
  itunesType: 'episodic',
  // Podcast 2.0 tags
  podcastGuid: '92f49cf0-db3e-5c17-8f11-9c5bd9e1f7ec', // Permanent GUID
  podcastMedium: 'podcast', // or 'music', 'video', 'film', 'audiobook', 'newsletter', 'blog'
  podcastLocked: true, // Prevent unauthorized imports
  podcastLockOwner: '[email protected]'
});

// Add an episode
podcast.addEpisode({
  title: 'Episode 42: The Future of AI',
  authorName: 'John Host',
  imageUrl: 'https://example.com/episode42.jpg',
  timestamp: Date.now(),
  url: 'https://example.com/episodes/42',
  content: 'In this episode, we explore the future of artificial intelligence...',
  audioUrl: 'https://example.com/audio/episode42.mp3',
  audioType: 'audio/mpeg',
  audioLength: 45678900, // bytes
  itunesDuration: 3600, // seconds
  itunesEpisode: 42,
  itunesSeason: 2,
  itunesEpisodeType: 'full',
  itunesExplicit: false,
  // Modern podcast features
  persons: [
    { name: 'John Host', role: 'host' },
    { name: 'Jane Guest', role: 'guest', href: 'https://example.com/jane' }
  ],
  transcripts: [
    { url: 'https://example.com/transcripts/ep42.txt', type: 'text/plain' }
  ],
  funding: [
    { url: 'https://example.com/support', message: 'Support the show!' }
  ]
});

// Export podcast RSS with iTunes and Podcast namespace
const podcastRss = podcast.exportPodcastRss();

Parsing Feeds and OPML

parseFeed, parseOpml, buildOpml and htmlToText are synchronous and pure: they take a string or bytes, never fetch anything, and return plain JSON data (dates are epoch milliseconds). A document they do not accept is a refusal with a code, never an exception; only a programming error (invalid options or limits, a documentUrl that is not absolute http:/https:, invalid buildOpml input) throws a TypeError.

import { parseFeed } from '@push.rocks/smartsitemap/feed';

const result = parseFeed(feedXml, { documentUrl: 'https://example.com/feed.xml' });
if (!result.ok) {
  console.log(result.refusal.code, result.refusal.message); // e.g. 'internal_subset'
} else {
  const { feed } = result;
  console.log(feed.format, feed.title, feed.isPodcast);
  for (const item of feed.items) {
    console.log(item.guid, item.link, item.title, item.publishedAt, item.enclosures);
  }
}

Fetching is the caller's. Pass the bytes, the final URL after redirects and the Content-Type header, so relative URLs resolve and the charset decodes. A URL that comes from someone else needs a public-only fetch on a server (smartrequest's publicOnly() or createPublicOnlyFetch() in Node.js and Bun, as the main entry's feed import does):

import { SmartRequest } from '@push.rocks/smartrequest';
import { parseFeed } from '@push.rocks/smartsitemap/feed';

const response = await SmartRequest.create()
  .url(untrustedFeedUrl)
  .publicOnly({ maxResponseSize: 5 * 1024 * 1024 })
  .get();
const contentType = response.headers['content-type'];
const result = parseFeed(new Uint8Array(await response.arrayBuffer()), {
  documentUrl: response.url,
  contentType: Array.isArray(contentType) ? contentType[0] : contentType,
  limits: { maxInputLength: 5 * 1024 * 1024, maxNodes: 100_000, maxItems: 200 },
});

Formats: RSS 0.9x and 2.0 (rss with a channel), RSS 1.0 and 0.90 (rdf:RDF, items beside the channel) and Atom 1.0, including xml:base and type="xhtml" content. RSS of every version is format: 'rss', with formatVersion as the document states it. Atom 0.3, JSON Feed and HTML pages are not_a_feed. Element names are matched ASCII case-insensitively and namespaces by their URI (lowercased, without http(s):// and a trailing /), so content:encoded under any prefix is found, and the iTunes https and DTDs/Podcast spellings match.

What an item carries (IParsedFeedItem): guid (RSS guid, Atom id, RSS 1.0 rdf:about), link (RSS link; the Atom alternate link, a text/html one first; else an RSS guid that is an http(s) URL and not isPermaLink="false"), title, authors (RSS author as email (Name), dc:creator, Atom person constructs), summary (RSS description, Atom summary), content (content:encoded, Atom content; contentSourceUrl for Atom content@src), categories (RSS category with domain as scheme, dc:subject, Atom term/scheme/label), publishedAt (pubDate, dc:date, Atom published), updatedAt (Atom updated), enclosures (every RSS enclosure and Atom link rel="enclosure"), itunes (duration, episode, season, episode type, explicit, image, title, subtitle, summary, author, block), media (media:content, media:thumbnail, media:group, media:title, media:description) and imageUrl (the first of: item media:thumbnail, an image media:content, a media:group thumbnail, itunes:image, an image/* enclosure). The feed (IParsedFeed) carries the channel's fields, the iTunes channel fields (owner, nested categories, type, complete, new-feed-url), selfUrl (atom:link rel="self"), isPodcast (any channel-level iTunes element, or an audio/* enclosure on at least half of the first 20 items), items in document order and omittedItems.

Text and HTML. An IParsedText has text, plain text, and, when the source is HTML (an RSS description, content:encoded, Atom type="html" or "xhtml"), html with baseUrl (the xml:base in scope, else documentUrl) for resolving its relative URLs.

• html is the document's HTML as it is, unsanitized, typed TUntrustedHtml. Sanitize it before rendering it anywhere. The brand documents that in every signature; it cannot stop an assignment to innerHTML.
• text comes from htmlToText, not from a sanitizer, so Tom & Jerry is Tom & Jerry: escape it wherever you put it into HTML.
• RSS titles are plain text whose HTML character references are decoded (a double-escaped ’ becomes ’); markup in them stays text and is never parsed, so a title <script>… reads <script>…; a character XML 1.0 forbids that a reference decodes to () is removed. Atom titles follow their type.

Refusals (TParseRefusalCode):

| Code | The document | | --- | --- | | too_large | is longer than maxInputLength (checked before parsing) | | internal_subset | has a DOCTYPE with [: an internal subset, which is where entity declarations live (billion laughs, external entities, parameter entities), or a [ in a quoted literal | | declaration_refused | has any other <!…> declaration anywhere: <!ENTITY>, <!ELEMENT>, <!ATTLIST>, <!NOTATION>, a conditional section, a lowercase <!doctype, a second DOCTYPE, or a DOCTYPE inside the root | | content_before_root | has text other than XML whitespace, or a CDATA section, before the root element (JSON, an HTML error page, a DOCTYPE with a quoted >) | | no_root | has no root element (empty input included) | | unclosed_root | ends before its root element does | | depth_exceeded, node_limit_exceeded, text_limit_exceeded, attribute_limit_exceeded | breaks a bound (below) | | not_a_feed | parseFeed: is not RSS 0.9x/1.0/2.0 or Atom 1.0 | | not_opml | parseOpml: is not opml with a body | | outline_limit_exceeded | parseOpml: has more outlines than maxOutlines |

The XML reader under both parsers is htmlparser2 in XML mode behind these rules:

• No entity is ever expanded. Only the five XML entities and numeric character references are decoded; entities a DTD declares, external entities and unknown named entities (  is one in XML) stay literal text. Nothing is ever fetched.
• One DOCTYPE without [ is allowed before the root (the Netscape RSS 0.91 DOCTYPE parses and its DTD is never loaded); see the refusals for everything else.
• Before the root only XML whitespace, comments, processing instructions (the XML declaration included) and that DOCTYPE may come; a leading byte order mark is dropped. After the root nothing is read.
• Every bound is checked while the document streams, so a document stops being read the moment it breaks one.
• Lenient where real feeds are wrong: an unescaped & in a URL (?a=1&b=2) is kept; &amp without ; stays literal; an element closed by an ancestor's end tag is accepted; the prefixes itunes, media, content, dc, atom and rdf resolve without a declaration.
• Characters XML 1.0 forbids (C0 controls but tab, LF and CR; U+FFFE; U+FFFF; lone surrogates) are removed; � becomes U+FFFD.
• Attributes are never used as object keys (__proto__ is a plain attribute); of duplicate attributes the first wins.

Bytes are decoded by RFC 7303 section 3.2: a byte order mark (UTF-8, UTF-16LE, UTF-16BE), else the charset of contentType (RFC 9110 parameter syntax: a token or a quoted string, the name case-insensitive), else the encoding of an XML declaration at the very start of the document (UTF-16 without a BOM is recognized from <?), else UTF-8. A label the runtime's TextDecoder does not know (utf8mb4, which servers send) is passed over, and a declaration saying UTF-16 over ASCII-compatible bytes is ignored. Malformed sequences become U+FFFD. contentType is ignored for a string.

Bounds (defaultFeedParseLimits, defaultOpmlParseLimits; every one can be set per call with limits, as a positive integer, maxDepth at most 1,000):

| Limit | parseFeed | parseOpml | On breach | | --- | --- | --- | --- | | maxInputLength | 10,485,760 | 4,194,304 | too_large; bytes for a Uint8Array, UTF-16 code units for a string | | maxDepth | 64 | 32 | depth_exceeded; the root element is depth 1 | | maxNodes | 200,000 | 100,000 | node_limit_exceeded; elements, text nodes and attributes together (adjacent text and CDATA are one text node) | | maxTextLength | 1,048,576 | 65,536 | text_limit_exceeded; per text node and per attribute value, after decoding | | maxAttributes | 64 | 64 | attribute_limit_exceeded; per element, namespace declarations included | | maxItems / maxOutlines | 1,000 | 10,000 | feed: later items are left out and counted in omittedItems; OPML: outline_limit_exceeded |

Memory: the document tree takes about 70 to 200 bytes per node (measured on Node.js 25.9: 67 for elements with 64 empty attributes, 143 for empty elements, 169 for elements with a short text, 196 for typical RSS items), so about 13 to 38 MiB at the default 200,000 nodes, on top of the input (a decoded string takes 2 bytes per character). 200,000 nodes of typical items parse in about 150 ms. Lower the limits per call where that is too much.

Field bounds (fixed): a value outside them is absent, never an error.

• URLs resolve against the xml:base in scope, else documentUrl; only absolute http: and https: URLs of at most 4,096 characters are kept (no javascript:, no data:).
• guid and Atom id: at most 2,000 characters, else absent (a cut one would collide).
• Short text (titles, names, labels, iTunes strings): whitespace collapsed, cut to 2,000 characters at a code point boundary.
• Categories: at most 50 per item or feed, deduplicated on term and scheme; a term over 200 characters drops the category, a scheme or label over 200 characters is dropped.
• Per item: at most 20 authors, 20 enclosures, 50 media contents and 50 thumbnails; at most 20 iTunes categories per feed, 20 subcategories each.
• HTML: an Atom xhtml serialization longer than maxTextLength makes the field absent.
• Dates: RFC 822/1123/2822 (optional weekday, 2- or 4-digit year, optional seconds, UT, GMT, Z, the US zone names, ±hhmm) and RFC 3339/W3CDTF (fractions, offsets, date-only, no zone meaning UTC), read by the parser itself rather than Date.parse, whose non-ISO formats differ between engines. Every calendar field is validated (no February 30, no hour 25), and only 1970-01-01T00:00:00Z ≤ t < 2100-01-01T00:00:00Z is kept.
• Numbers: durations (H:M:S, M:S or seconds) 0 to 3,600,000 seconds, rounded; lengths and file sizes 0 to 2^53−1; episode and season 1 to 1,000,000; media sizes 0 to 1,000,000. Media types are lowercased type/subtype without parameters.

OPML

import { parseOpml, buildOpml } from '@push.rocks/smartsitemap/feed';

const result = parseOpml(opmlBytes, { contentType: 'text/x-opml; charset=utf-8' });
if (result.ok) {
  for (const feed of result.opml.feeds) {
    console.log(feed.folders.join(' / '), feed.text, feed.xmlUrl);
  }
}

const opml = buildOpml({
  title: 'My subscriptions',
  dateCreated: Date.now(),
  outlines: [
    { text: 'Tech', children: [{ text: 'Example', type: 'rss', xmlUrl: 'https://example.com/feed.xml' }] },
  ],
});

• parseOpml reads OPML 1.0 and 2.0 with the same reader, refusals and bounds. outlines is the outline tree; feeds lists every outline with an xmlUrl in document order, with the texts (else titles) of its ancestors as folders. Attribute names are matched case-insensitively.
• xmlUrl and htmlUrl are the raw strings, trimmed (absent past 4,096 characters), not validated: check each one before you fetch it, and report a refusal per URL.
• buildOpml writes OPML 2.0 in UTF-8. Text and attributes are escaped (& < > " ', and tab, LF and CR as character references, so values survive attribute normalization elsewhere); characters XML 1.0 forbids are dropped. It throws a TypeError for an outline without a non-empty text, a field longer than 2,000 characters (URLs: 4,096, without surrounding whitespace), nesting deeper than 30, more than 10,000 outlines, a document longer than 4 MiB, or a dateCreated outside 1970 to 2099. So every document it writes parses back with parseOpml under the default limits.

Plain Text from HTML

import { htmlToText } from '@push.rocks/smartsitemap/feed';

htmlToText('<p>Tom &amp; Jerry</p><p>at the <b>caf&eacute;</b></p><script>x()</script>');
// 'Tom & Jerry at the café'

htmlToText reads HTML (the HTML entity table, so   and é decode) and keeps the text from the parser's event stream: inline markup joins its text, block elements and br separate words, whitespace collapses, and the content of script, style, template, noscript, iframe, object, svg, math, title and textarea is skipped. Characters XML 1.0 forbids are removed after decoding, so  leaves nothing. It stops at a nesting depth of 256 and once the result reaches maxLength (default 1,048,576). It is not a sanitizer: the result is text.

Finding Feed Links in HTML

import { discoverFeedLinks } from '@push.rocks/smartsitemap/feed';

const result = discoverFeedLinks(htmlBytes, {
  documentUrl: 'https://example.com/blog/', // the final URL, after redirects
  contentType: 'text/html; charset=utf-8', // the response's Content-Type, for bytes
});
if (result.ok) {
  for (const link of result.links) {
    console.log(link.type, link.url, link.title); // 'rss' | 'atom' | 'json'
  }
}

discoverFeedLinks scans an HTML document (a string or bytes) for <link rel="alternate"> elements whose type is application/rss+xml (rss), application/atom+xml (atom) or application/feed+json (json), and returns them in document order with their title. It never fetches anything.

• Matching: rel is a space-separated list that must contain alternate; rel and the type essence (before any ; parameters) are compared ASCII case-insensitively. application/json is not a JSON Feed type: it is the type of many other alternate links. Links inside template, svg and math, comments and raw-text elements such as script are not part of the document.
• URLs: each href is resolved against the document base: the first <base href> (resolved against documentUrl, wherever it is in the document), else documentUrl. Links that do not resolve to an absolute http: or https: URL, or are longer than 4,096 characters, are left out, and each URL is returned once (its first link). The URLs are not checked further: check each one before you fetch it.
• Titles: entities decoded, whitespace collapsed, characters XML 1.0 forbids removed, cut to 2,000 characters; absent when empty.
• Bytes are decoded as browsers sniff HTML: a byte order mark, then the charset of contentType, then a <meta charset> or <meta http-equiv="Content-Type" content> in the first 1,024 bytes (a UTF-16 label there means UTF-8), then UTF-8. Strings are scanned as they are.
• Bounds: an input longer than maxInputLength (default 10 MiB: bytes, or UTF-16 code units for a string) is refused with { ok: false, refusal: { code: 'too_large' } }; at most maxLinks (default 50) links are returned, counted after resolving, filtering and deduplication, so duplicates and links left out never take a place; the distinct links after them are counted in omittedLinks. Scanning stops at a nesting depth of 256. defaultFeedLinkLimits holds the defaults; invalid options or limits throw a TypeError.

Creating Feeds from Article Arrays

import { Smartfeed } from '@push.rocks/smartsitemap/feed';
import type { IArticle } from '@tsclass/tsclass';

const smartfeed = new Smartfeed();

const articles: IArticle[] = [
  // Your article objects conforming to @tsclass/tsclass IArticle interface;
  // featuredImageUrl is optional and becomes an enclosure link without a type
];

const feedOptions = {
  domain: 'blog.example.com',
  title: 'My Blog',
  description: 'Thoughts on code and design',
  category: 'Programming',
  company: 'Example Inc',
  companyEmail: '[email protected]',
  companyDomain: 'https://example.com'
};

// Creates an Atom feed from articles
const atomFeed = await smartfeed.createFeedFromArticleArray(feedOptions, articles);

Feed API Reference

Parsing Functions

| Function | Returns | Description | | --- | --- | --- | | parseFeed(input: string \| Uint8Array, options?: IParseFeedOptions) | TParseFeedResult | { ok: true, feed: IParsedFeed } or { ok: false, refusal: IParseRefusal }; options documentUrl, contentType, limits | | parseOpml(input: string \| Uint8Array, options?: IParseOpmlOptions) | TParseOpmlResult | { ok: true, opml: IParsedOpml } or { ok: false, refusal }; options contentType, limits | | buildOpml(document: IBuildOpmlDocument) | string | OPML 2.0; a TypeError for invalid input | | htmlToText(html: string, options?: IHtmlToTextOptions) | string | Plain text of an HTML fragment; option maxLength | | discoverFeedLinks(input: string \| Uint8Array, options: IDiscoverFeedLinksOptions) | TDiscoverFeedLinksResult | { ok: true, links: IDiscoveredFeedLink[], omittedLinks } or { ok: false, refusal } (too_large); options documentUrl (required), contentType, limits | | defaultFeedParseLimits, defaultOpmlParseLimits, defaultFeedLinkLimits | Readonly<IFeedParseLimits>, Readonly<IOpmlParseLimits>, Readonly<IFeedLinkLimits> | The default bounds |

Smartfeed Class

The main class for creating feeds. Parsing is parseFeed(), above.

createFeed(options: IFeedOptions): Feed

Creates a standard feed (RSS/Atom/JSON).

Options:

• domain (string) - Feed domain (e.g., 'example.com')
• title (string) - Feed title
• description (string) - Feed description
• category (string) - Feed category
• company (string) - Company/organization name
• companyEmail (string) - Contact email
• companyDomain (string) - Company website URL (absolute)
• feedUrl (string, optional) - Custom URL for the feed's self-reference (defaults to https://${domain}/feed.xml)
• buildTimestamp (number, optional) - Unix timestamp in milliseconds for RSS <lastBuildDate> and the Atom feed <updated>; defaults to the time of each export

createPodcastFeed(options: IPodcastFeedOptions): PodcastFeed

Creates a podcast feed with iTunes and Podcast namespace support.

iTunes Options:

• itunesCategory (string) - iTunes category
• itunesSubcategory (string, optional) - iTunes subcategory
• itunesAuthor (string) - Podcast author
• itunesOwner (object) - Owner info with name and email
• itunesImage (string) - Artwork URL (1400x1400 to 3000x3000, JPG/PNG)
• itunesExplicit (boolean) - Explicit content flag
• itunesType ('episodic' | 'serial', optional) - Podcast type
• itunesSummary (string, optional) - Detailed summary
• copyright (string, optional) - Custom copyright
• language (string, optional) - Language code (default: 'en')

Podcast 2.0 Options:

• podcastGuid (string) - Required. Globally unique identifier (GUID) for the podcast
• podcastMedium ('podcast' | 'music' | 'video' | 'film' | 'audiobook' | 'newsletter' | 'blog', optional) - Content medium type
• podcastLocked (boolean, optional) - Prevents unauthorized podcast imports (e.g., to other platforms)
• podcastLockOwner (string, optional) - Email of who can unlock (required if podcastLocked is true)

createFeedFromArticleArray(options: IFeedOptions, articles: IArticle[]): Promise<string>

Creates an Atom feed from an array of @tsclass/tsclass article objects. Every article needs a url; featuredImageUrl is optional.

Feed Class

Represents a feed that can be exported in multiple formats.

addItem(item: IFeedItem): void

Adds an item to the feed.

Item Properties:

• title (string) - Item title
• timestamp (number) - Unix timestamp in milliseconds
• url (string) - Absolute URL to the item
• authorName (string) - Author name
• content (string) - Item summary, HTML (RSS <description>)
• imageUrl (string, optional) - Absolute URL to the item's image; without it no image is written
• imageType (string, optional) - The image's MIME type (image/…); required for the RSS export of an item with imageUrl
• imageLength (number, optional) - The image's size in bytes; RSS writes length="0" (unknown) without it
• contentHtml (string, optional) - The full HTML of the item (RSS <content:encoded>)
• categories (Array<string | { term: string; domain?: string }>, optional) - The item's categories, in order; domain is the taxonomy (RSS domain, Atom scheme)
• thumbnail ({ url: string; width?: number; height?: number }, optional) - A thumbnail image, RSS and Atom <media:thumbnail>
• id (string, optional) - Unique identifier, the guid (uses URL if not provided)
• idIsPermaLink (boolean, optional) - Whether id is the item's permanent URL; defaults to false for an id, and needs an id

exportRssFeedString(): string

Exports the feed as RSS 2.0 XML.

exportAtomFeed(): string

Exports the feed as Atom 1.0 XML.

exportJsonFeed(): string

Exports the feed as JSON Feed 1.0.

PodcastFeed Class

Extends Feed with podcast-specific functionality.

addEpisode(episode: IPodcastItem): void

Adds a podcast episode to the feed.

Episode Properties (in addition to IFeedItem):

• audioUrl (string) - Absolute URL to audio file
• audioType (string) - MIME type (e.g., 'audio/mpeg')
• audioLength (number) - File size in bytes
• itunesDuration (number) - Duration in seconds
• itunesEpisode (number, optional) - Episode number
• itunesSeason (number, optional) - Season number
• itunesEpisodeType ('full' | 'trailer' | 'bonus', optional)
• itunesExplicit (boolean, optional) - Explicit content flag
• itunesSubtitle (string, optional) - Short description
• itunesSummary (string, optional) - Detailed summary
• persons (array, optional) - People involved (hosts, guests)
• chapters (array, optional) - Chapter markers
• transcripts (array, optional) - Transcript links
• funding (array, optional) - Donation/support links

imageUrl is required for an episode (<itunes:image>); imageType, imageLength, contentHtml, categories, thumbnail and idIsPermaLink are not episode fields.

exportPodcastRss(): string

Exports the podcast feed as RSS 2.0 with iTunes and Podcast namespace extensions.

Validation & Security

The feed module includes comprehensive validation to ensure feed integrity and security:

• URL Validation - All URLs must be absolute and use http/https protocols
• Email Validation - Email addresses are validated against RFC standards
• Domain Validation - Proper domain format checking
• Timestamp Validation - Ensures timestamps are valid and reasonable
• Content Sanitization - Prevents XSS attacks through proper XML escaping
• Duplicate Detection - Prevents duplicate item IDs in feeds
• Required Field Checking - Validates all required fields are present

Parsing has its own security model: no entity expansion, refused DTD subsets and declarations, bounds checked while streaming, and HTML handed out unsanitized as TUntrustedHtml (see Parsing Feeds and OPML).

Best Practices

Feed Item IDs

Feed item IDs should be permanent and never change once published. This allows feed readers to properly track which items have been read:

feed.addItem({
  id: 'post-2024-01-15-typescript-tips', // Permanent ID
  title: 'TypeScript Tips',
  url: 'https://example.com/posts/typescript-tips',
  // ... other fields
});

If you don't provide an id, the url will be used as the guid with isPermaLink="true". Make sure URLs don't change for published items.

An id is written with isPermaLink="false": RSS 2.0 reads a permalink guid as the item's URL. When your id is the permanent URL of the item, set idIsPermaLink: true; the id must then be an absolute URL. addItem() refuses idIsPermaLink without an id.

HTTPS URLs

Always use HTTPS URLs for security and privacy. The library will warn you if HTTP URLs are used:

// ✅ Good
imageUrl: 'https://example.com/image.jpg'

// ⚠️ Will trigger a warning
imageUrl: 'http://example.com/image.jpg'

Podcast Artwork

For podcast feeds, artwork should be:

• Square (1:1 aspect ratio)
• Between 1400x1400 and 3000x3000 pixels
• JPG or PNG format
• Maximum 512 KB file size (Apple Podcasts requirement)

Podcast 2.0 Compatibility

The library fully supports the Podcast 2.0 namespace, making your feeds compatible with modern podcast platforms like:

• Podcast Index - The open podcast directory
• Castopod - Open-source podcast hosting platform
• Podverse - Open-source podcast app
• And other Podcast 2.0-compliant apps

Key Podcast 2.0 Features:

• podcast:guid - Permanent unique identifier for your podcast
• podcast:medium - Declare if your feed is a podcast, music, video, etc.
• podcast:locked - Protect your podcast from unauthorized imports
• podcast:person - List hosts, co-hosts, and guests with rich metadata
• podcast:transcript - Link to transcript files in various formats
• podcast:funding - Add donation/support links for your listeners

These features are included in the RSS export when you use exportPodcastRss().

Feed Types

Full TypeScript definitions are included. Import types as needed:

import type {
  IFeedOptions,
  IFeedItem,
  IFeedThumbnail,
  IFeedCategory,
  TFeedCategory,
  IPodcastFeedOptions,
  IPodcastItem,
  IPodcastOwner,
  IPodcastPerson,
  IPodcastChapter,
  IPodcastTranscript,
  IPodcastFunding,
  // parsing
  TUntrustedHtml,
  IXmlReadLimits,
  IFeedParseLimits,
  IOpmlParseLimits,
  IParseFeedOptions,
  IParseOpmlOptions,
  TParseRefusalCode,
  IParseRefusal,
  TParseFeedResult,
  TParseOpmlResult,
  IParsedText,
  IParsedPerson,
  IParsedCategory,
  IParsedEnclosure,
  IParsedMediaThumbnail,
  IParsedMediaContent,
  IParsedMedia,
  IParsedItunesCategory,
  IParsedItunesFeed,
  IParsedItunesItem,
  IParsedFeedItem,
  IParsedFeed,
  IParsedOpmlOutline,
  IParsedOpmlFeed,
  IParsedOpml,
  IBuildOpmlOutline,
  IBuildOpmlDocument,
  IHtmlToTextOptions,
  // feed links in HTML
  IFeedLinkLimits,
  IDiscoverFeedLinksOptions,
  TFeedLinkType,
  IDiscoveredFeedLink,
  TDiscoverFeedLinksResult,
} from '@push.rocks/smartsitemap/feed';

🏗️ Real-World Integration Examples

Express.js / Hono / Fastify Server

import { SmartSitemap } from '@push.rocks/smartsitemap';

// Serve dynamic sitemap
app.get('/sitemap.xml', async (req, res) => {
  const xml = SmartSitemap.create()
    .setDefaultChangeFreq('weekly')
    .addFromArray(await getUrlsFromDatabase())
    .toXml();

  res.header('Content-Type', 'application/xml');
  res.send(xml);
});

// Serve news sitemap from RSS
app.get('/news-sitemap.xml', async (req, res) => {
  const builder = SmartSitemap.createNews({ publicationName: 'My Site' });
  await builder.importFromFeedUrl('https://mysite.com/rss/');

  res.header('Content-Type', 'application/xml');
  res.send(builder.toXml());
});

// Auto-split with sitemap index
app.get('/sitemap-index.xml', async (req, res) => {
  const builder = SmartSitemap.create({ baseUrl: 'https://mysite.com' });
  builder.addFromArray(await getAllUrls()); // 200K+ URLs

  const set = builder.toSitemapSet();
  res.header('Content-Type', 'application/xml');
  res.send(set.indexXml ?? set.sitemaps[0].xml);
});

Static Site Generator

import { SmartSitemap } from '@push.rocks/smartsitemap';
import { writeFileSync } from 'fs';

const xml = SmartSitemap.create()
  .setDefaultChangeFreq('weekly')
  .add({ loc: 'https://mysite.com/', changefreq: 'daily', priority: 1.0 })
  .add({ loc: 'https://mysite.com/about', changefreq: 'monthly' })
  .addFromArray(blogPostUrls)
  .dedupe()
  .toXml();

writeFileSync('./public/sitemap.xml', xml);

API Reference

SmartSitemap (Static Factories)

| Method | Returns | Description | |--------|---------|-------------| | SmartSitemap.create(options?) | UrlsetBuilder | Create a standard sitemap builder | | SmartSitemap.createNews(options) | NewsSitemapBuilder | Create a news sitemap builder | | SmartSitemap.createIndex(options?) | SitemapIndexBuilder | Create a sitemap index builder | | SmartSitemap.fromUrls(urls, options?) | UrlsetBuilder | Builder from URL string array | | SmartSitemap.fromYaml(yaml, fetchOptions?) | Promise<UrlsetBuilder> | Builder from YAML config; its feeds: are fetched public-only | | SmartSitemap.fromFeedUrl(url, options?) | Promise<UrlsetBuilder> | Builder from RSS/Atom feed URL (fetched public-only, options.fetchOptions) | | SmartSitemap.fromFeedString(xml, options?) | Promise<UrlsetBuilder> | Builder from RSS/Atom feed string | | SmartSitemap.fromArticles(articles, options) | NewsSitemapBuilder | Builder from IArticle array | | SmartSitemap.parse(xml) | Promise<IParsedSitemap> | Parse sitemap XML string | | SmartSitemap.parseUrl(url, fetchOptions?) | Promise<IParsedSitemap> | Fetch (public-only, at most 50 MiB) and parse sitemap | | SmartSitemap.validate(xml) | Promise<IValidationResult> | Validate sitemap XML |

RobotsTxtParser

| Method | Returns | Description | |--------|---------|-------------| | RobotsTxtParser.parse(robotsTxt) | IParsedRobotsTxt | Read the Sitemap: URLs of robots.txt content | | RobotsTxtParser.parseUrl(url, fetchOptions?) | Promise<IParsedRobotsTxt> | Fetch (public-only, at most 500 KiB) and parse a robots.txt |

UrlsetBuilder (Chainable)

| Method | Returns | Description | |--------|---------|-------------| | .add(url) | this | Add a URL with full ISitemapUrl options | | .addUrl(loc, lastmod?) | this | Add by URL string | | .addUrls(urls) | this | Add multiple ISitemapUrl objects | | .addFromArray(locs) | this | Add from plain string array | | .merge(other) | this | Merge in another builder's URLs | | .filter(predicate) | this | Filter URLs in-place | | .map(transform) | this | Transform URLs in-place | | .sort(compareFn?) | this | Sort URLs (default: alphabetical) | | .dedupe() | this | Remove duplicate URLs by loc | | .setDefaultChangeFreq(freq) | this | Set default changefreq | | .setDefaultPriority(priority) | this | Set default priority (0.0–1.0) | | .setXslUrl(url) | this | Set XSL stylesheet URL | | .importFromFeedUrl(url, options?) | Promise<this> | Import from RSS/Atom feed URL (fetched public-only, options.fetchOptions) | | .importFromFeedString(xml, options?) | Promise<this> | Import from RSS/Atom feed string | | .importFromYaml(yaml, fetchOptions?) | Promise<this> | Import from YAML config | | .importFromArticles(articles) | this | Import from IArticle array | | .toXml() | string | Export as sitemap XML | | .toTxt() | string | Export as plain text | | .toJson() | string | Export as JSON | | .toGzipBuffer() | Promise<Buffer> | Export as gzipped XML | | .toSitemapSet() | ISitemapSet | Auto-split with index | | .toStream() | SitemapStream | Export as Node.js Readable stream | | .validate() | IValidationResult | Validate against spec | | .stats() | ISitemapStats | Get statistics | | .getUrls() | ISitemapUrl[] | Get the raw URL array | | .count | number | Get URL count |

NewsSitemapBuilder (extends UrlsetBuilder)

| Method | Returns | Description | |--------|---------|-------------| | .addNewsUrl(loc, title, date, keywords?) | this | Add a news article with publication info |

SitemapIndexBuilder

| Method | Returns | Description | |--------|---------|-------------| | .add(entry) | this | Add a sitemap index entry | | .addSitemap(loc, lastmod?) | this | Add by URL string | | .addSitemaps(entries) | this | Add multiple entries | | SitemapIndexBuilder.fromBuilder(builder, baseUrl) | {index, sitemaps[]} | Auto-split a builder | | .toXml() | string | Export as sitemap index XML | | .count | number | Get entry count |

SitemapStream

| Method | Description | |--------|-------------| | .pushUrl(url) | Push a URL entry to the stream | | .finish() | Signal end of stream, writes closing tag | | .count | Number of URLs written |

Key Types

interface ISitemapUrl {
  loc: string;                       // Required — absolute URL
  lastmod?: Date | string | number;  // Date, ISO string, or timestamp (ms)
  changefreq?: TChangeFreq;          // 'always'|'hourly'|'daily'|'weekly'|'monthly'|'yearly'|'never'
  priority?: number;                 // 0.0 to 1.0
  images?: ISitemapImage[];          // Image extension
  videos?: ISitemapVideo[];          // Video extension
  news?: ISitemapNews;               // News extension
  alternates?: ISitemapAlternate[];  // hreflang alternates
}

interface ISitemapOptions {
  baseUrl?: string;
  xslUrl?: string;
  defaultChangeFreq?: TChangeFreq;
  defaultPriority?: number;
  prettyPrint?: boolean;        // default: true
  maxUrlsPerSitemap?: number;   // default: 50000
  gzip?: boolean;
  validate?: boolean;           // default: true
}

interface INewsSitemapOptions extends ISitemapOptions {
  publicationName: string;      // Required
  publicationLanguage?: string; // default: 'en'
}

interface IFeedImportOptions {
  publicationName?: string;     // news mapping
  publicationLanguage?: string; // news mapping
  newerThan?: Date | number;    // items without a date are kept
  limit?: number;
  mapItem?: (item: IParsedFeedItem) => ISitemapUrl | null; // null skips the item
  fetchOptions?: IRemoteFetchOptions;
}

interface IRemoteFetchOptions {
  // smartrequest public-only settings, except maxResponseSize (the caps are fixed)
  publicOnly?: Omit<IPublicOnlyConfig, 'maxResponseSize'>;
}

Migrating from 4.x

5.0.0 replaces feed parsing, corrects the feed writer's image enclosures and guids, and makes every fetch of the main entry public-only:

• Smartfeed.parseFeedFromString() and Smartfeed.parseFeedFromUrl() are removed. Use parseFeed(input, options): synchronous, a result instead of an exception (check result.ok), and no fetching. Fetch the feed yourself and pass the bytes, the final URL (documentUrl) and the Content-Type (contentType); for a URL from untrusted input, fetch public-only (see Parsing Feeds and OPML).

• IParsedFeed and IParsedItem change shape. The loose [key: string]: any types are replaced by fixed fields; the item type is now IParsedFeedItem:

  | 4.x | 5.x | | --- | --- | | feed.description (string) | feed.description (IParsedText: text, html, baseUrl) | | feed.image.url | feed.imageUrl, feed.iconUrl | | item.id | item.guid | | item.pubDate, item.isoDate (strings) | item.publishedAt, item.updatedAt (epoch milliseconds) | | item.author (string) | item.authors (IParsedPerson[]) | | item.content, item.contentSnippet (strings) | item.summary, item.content (IParsedText) | | item.categories (strings) | item.categories (IParsedCategory[]: term, scheme, label) | | item.enclosure (one) | item.enclosures (all), plus imageUrl, itunes, media |

• The main entry's IFeedItem is removed. IFeedImportOptions.mapItem receives IParsedFeedItem, which both entry points export.

• Documents the reader refuses no longer parse: a DOCTYPE with an internal subset, any other <!…> declaration, content before the root, and documents past a bound. The feed imports of the main entry reject them with an Error naming the refusal code.

• Every fetch of the main entry is public-only (see Remote Fetches): a loopback, private or link-local address, a special-use name such as localhost, and a port other than 80 and 443 are refused with a PublicOnlyError unless fetchOptions.publicOnly admits them; redirects are followed; bodies are capped at 10 MiB (feeds), 50 MiB (sitemaps) and 500 KiB (robots.txt).

• The RSS image enclosure needs imageType. 4.x wrote every item image as type="image/jpeg" length="0"; 5.x writes the imageType (and imageLength) you give, and exportRssFeedString() throws for an item with imageUrl but no imageType. Pass the image's MIME type with every imageUrl you export to RSS. Atom no longer claims image/jpeg either: its enclosure link carries type only when imageType is given.

• A feed item id is a guid with isPermaLink="false" (4.x wrote "true" for any id). When your id is the item's permanent URL, set idIsPermaLink: true.

• PodcastFeed's inherited exportRssFeedString() throws for a feed with episodes. An episode's imageUrl is its artwork and episodes have no imageType, so the plain RSS export refuses the image (4.x wrote it as an image/jpeg enclosure), and it never wrote the audio enclosure or the iTunes and Podcast 2.0 fields. Export podcasts with exportPodcastRss().

• @push.rocks/smartrequest 6 and the new dependencies htmlparser2 and entities; fast-xml-parser is no longer a direct dependency (the main entry reaches it through @push.rocks/smartxml).

Migration from @push.rocks/smartfeed

@push.rocks/smartfeed is deprecated; its creation API moved unchanged to the @push.rocks/smartsitemap/feed entry point, where only the import path changes. Its parsing (parseFeedFromString, parseFeedFromUrl, IParsedFeed, IParsedItem) is replaced by parseFeed as Migrating from 4.x describes.

| @push.rocks/smartfeed | @push.rocks/smartsitemap | | --- | --- | | import { Smartfeed, Feed, PodcastFeed } from '@push.rocks/smartfeed' | import { Smartfeed, Feed, PodcastFeed } from '@push.rocks/smartsitemap/feed' | | types IFeedOptions, IFeedItem, IPodcastFeedOptions, IPodcastItem, IPodcastOwner, IPodcastPerson, IPodcastChapter, IPodcastTranscript, IPodcastFunding | same names from @push.rocks/smartsitemap/feed | | smartfeed.parseFeedFromString(xml) | parseFeed(xml) from @push.rocks/smartsitemap/feed: synchronous, returns { ok: true, feed } or { ok: false, refusal }, so check result.ok before reading result.feed |

Feeds generated now carry <generator>@push.rocks/smartsitemap</generator> (RSS, Atom and Podcast RSS) instead of @push.rocks/smartfeed. The feed module's IFeedItem is an item you add to a feed; a parsed item is IParsedFeedItem.

Migration from @push.rocks/smartrobots

@push.rocks/smartrobots is deprecated; robots.txt sitemap discovery lives in the main entry.

| @push.rocks/smartrobots | @push.rocks/smartsitemap | | --- | --- | | new Smartrobots().parseRobotsTxtFromUrl(url) | RobotsTxtParser.parseUrl(url) | | new Smartrobots().parseRobotsTxt(text) (async) | RobotsTxtParser.parse(text) (synchronous) | | result { sitemaps: string[] } | IParsedRobotsTxt ({ sitemaps: string[] }) |

Sitemap: lines are now matched case-insensitively with optional whitespace, CRLF line endings and comments are handled, duplicates are removed, and parseUrl() fetches public-only and rejects on a non-2xx response.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at [email protected].

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.