Feedsweep

Tidy up the HTML content in web feeds. Fix feed-specific quirks so content displays in its best possible form.

Feedsweep takes raw feed item HTML and runs it through a pipeline that genuinely improves the output: fixing lazy-loaded images so they actually render, resolving relative URLs to absolute, stripping tracking pixels for privacy (plus tracking params and redirect wrappers via the cleanUrlFn option), highlighting code blocks, normalizing broken markup from common feed quirks, auto-linking bare URLs, and converting embeds into framework-agnostic placeholders. It ships with sensible defaults, so the common video, audio and chart providers (YouTube, Vimeo, SoundCloud and more) are recognized without any configuration.

Installation

npm install feedsweep linkedom

linkedom is an optional peer dependency. You only need it if you use the bundled parseHtml helper — see DOM library for jsdom / happy-dom / browser-native alternatives.

Quick Start

import { transformContent } from 'feedsweep'
import { parseHtml } from 'feedsweep/linkedom'

const result = await transformContent('<p>Check <img data-src="photo.jpg"> and visit /about</p>', {
  parseHtmlFn: parseHtml,
  baseUrl: 'https://example.com/post/1',
})

Transforms

Inventory of every transform exported from the package. Most are enabled by default; pass a custom stringTransforms / domTransforms array via transformContent options to override. Transforms marked Heuristic (opt-in) make a best-judgement guess and may drop content, so they are excluded from the standard pipeline — enable them with heuristics: true (see Options).

| Transform | Description | | --- | --- | | decodeDoubleEncodedTags | Decode double-escaped tags (<tag>) back to real HTML | | fixLazyImages | Promote lazy-loaded data-src / data-original to real src | | fixLazyIframes | Promote a lazy or consent-parked iframe src (real URL in a data-* attribute) to real src, skipping placeholder pages | | fixLazyVideos | Promote a lazy <video> src and data-poster to real attributes | | fixLazyAudios | Promote a lazy <audio> src to real src | | fixDropboxMediaUrls | Force raw=1 on a Dropbox share URL used as a media src, so it serves the file and not the preview page | | convertLazyImageContainers | Convert a container parking an image URL in a lazy attribute into a real <img> | | flattenPictureElements | Collapse <picture> to one <img>, keeping the best modern-format source | | hoistFigcaptionFromAnchor | Move a <figcaption> out of the figure's click-through link | | wrapOrphanFigcaptions | Wrap an image and its detached <figcaption> in a <figure> | | mergeWrappedCaptionText | Fold caption text sharing a wrapper with the <figcaption> into it | | canonicalizeAlignment | Normalize media alignment into a single data-align hook | | mergeConsecutiveOneLinerPres | Merge consecutive single-line <pre> blocks into one | | replacePreLineBreaks | Replace <br> with newlines inside <pre> | | stripInterBlockBreaks | Remove stray <br> tags between block elements | | stripBoundaryBreaks | Remove <br> tags at block boundaries | | stripDuplicateTitleHeading | Remove a leading heading that repeats the article title | | demoteHeadings | Shift headings down a level so they sit below the reader's page title | | unwrapHeadingBold | Unwrap redundant bold wrapping a whole heading | | normalizeAnchoredHeadings | Strip static-site-generator permalink anchors from headings (Sphinx/MkDocs, Docusaurus, AnchorJS, Zola) | | cleanAnchorUrls | Clean anchor hrefs (redirects, tracking params) via the cleanUrlFn option, after the shared url repair: an escaped &, a run of slashes, a query glued onto the path and whitespace in values | | stripDeadAnchors | Unwrap links with empty, #, or javascript: hrefs | | stripNonContentElements | Strip non-content chrome — subscribe/share/related widgets, ads, author bios | | convertUcozSpoilerButtons | Replace a titled uCoz spoiler button with its title in bold | | stripHiddenElements | Strip elements hidden from view (hidden attribute, inline display:none / visibility:hidden), and unhide the slides, spoilers, tabs and players a reader can reveal | | removeTrackingPixels | Strip 1×1 tracking pixels, keeping real images | | convertEmojis | Replace emoji and forum smilie markup with the real glyph, marking with data-emoji the images and fallback text that have none | | resolveMediaDimensions | Backfill width/height on media so aspect ratio survives style stripping | | convertBreaksToParagraphs | Convert <br><br> runs into real <p> blocks | | wrapBareInlineInParagraphs | Wrap loose inline content in <p> blocks | | hoistBlocksFromParagraphs | Hoist block elements out of enclosing paragraphs, keeping only halves that still render | | wrapCargoGalleryImages | Wrap Cargo portfolio captions and images in <figure> blocks so they stay apart | | convertBloggerCaptionTables | Convert a Blogger caption table (table.tr-caption-container) into a <figure> with the image and its <figcaption> | | injectEnclosures | Inject feed enclosures as native media or embed placeholders, collapsing a media group to one rendition and merging a player page entry with its media file; an image enclosure injects only when the content has no image of its own, captioned with its description; anything else becomes a data-file-* placeholder, after the content | | surfaceParkedMarkup | Dissolve a container holding an embed encoded in an attribute, a lazy loader's div.load-later or a Borlabs Cookie content blocker, into the markup it holds | | rebuildPublicalbumGalleries | Rebuild a Publicalbum Google Photos album widget into its photos as linked images under a captioned <figure> | | surfaceTemplateEmbeds | Hoist a video embed out of a lazy-load <template> (e.g. Better Core Video Embeds) so it renders in a reader | | unwrapDrupalOembedIframes | Point a Drupal media oEmbed proxy frame (/media/oembed?url=) at the page url it wraps | | unwrapWpcomWidgetFrames | Rebuild a WordPress.com wpcomwidgets.com frame into the Flash <embed> it wraps, so a Flash resolver can claim it | | unwrapRebelmouseEmbedFrames | Point a RebelMouse embed frame (/res/scraper/embed/, /res/community/twitter_embed/) at the player or tweet it wraps | | surfaceNoscriptEmbeds | Hoist a video <iframe> out of a <noscript> lazy-load fallback (e.g. WP Rocket, a3 Lazy Load); ignores non-video noscript iframes like Google Tag Manager | | rebuildEmbedPlusEmbeds | Rebuild a real <iframe> from an "Embed Plus for YouTube" facade (.epyt-facade[data-facadesrc]) | | rebuildIframelyEmbeds | Rebuild an Iframely anchor facade (an empty a[data-iframely-url]) into the destination's frame or a plain link, and point an Iframely frame at the player of the url it proxies | | rebuildWordpressEmbeds | Rebuild a WordPress embed block that holds only its source url into the platform's frame, when a resolver claims the url | | rebuildLiteVideoEmbeds | Rebuild a real <iframe> from a lite-youtube / lite-vimeo web component's videoid, playlistid or player-src, carrying over its start, title and poster | | rebuildEmbettyVideoEmbeds | Rebuild a real <iframe> from an embetty-video web component's type and video-id, carrying over start-at and poster-image | | rebuildLyteEmbeds | Rebuild a real <iframe> from a WP YouTube Lyte facade (WYL_/lyte_ id) or the thumbnail it writes into feeds | | rebuildRocketYoutubePreviews | Rebuild a real <iframe> from a WP Rocket YouTube preview facade (.rll-youtube-player[data-id]), carrying over data-query | | rebuildVideoJsEmbeds | Rebuild a native <video> from a Video.js <video-js> element, reading the source from a <source> child or the data-setup JSON | | rebuildWmakerEmbeds | Rebuild a real <iframe> from a WMaker item's dead Flash player, pointing it at the modern embed page named by the article id in the item's permalink | | rebuildWistiaEmbeds | Rebuild a real <iframe> from a Wistia JS-API inline embed facade (wistia_async_{id} class) or legacy API embed mount (wistia_{id} id) | | rebuildLazyLoadForVideos | Rebuild a real <iframe> from a "Lazy Load for Videos" facade (a.preview-lazyload), recovering the YouTube/Vimeo id from data-video-uri or href and carrying over data-video-title | | rebuildLazyYtEmbeds | Rebuild a real <iframe> from a lazy YouTube facade parking the id in a data attribute (data-youtube-id, data-youtube, .youtube-embed, .youtube-player) | | rebuildElementorVideoEmbeds | Rebuild a real <iframe> from an Elementor video widget's deferred data-settings (YouTube / Vimeo / Dailymotion / VideoPress) | | rebuildEmbedPrivacyLinks | Rebuild the link an Embed Privacy feed carries in place of an embed into that url's frame, when a resolver claims the url | | rebuildEmbedlyEmbeds | Unwrap an Embedly media widget to the inner provider iframe, carrying the poster as data-thumbnail | | rebuildDocumentcloudEmbeds | Rebuild a real <iframe> from DocumentCloud's old viewer snippet onto the document's embed page, reading the document from the inline DV.load call | | rebuildGettyImagesEmbeds | Rebuild a real <iframe> from a Getty Images gie widget facade, composing the player URL from the inline config the loader script never runs | | rebuildExternalVideoEmbeds | Rebuild a real <iframe> from a Seesaa or Sakura blog's external_video.js block onto the YouTube or Nicovideo video it names | | rebuildJsfiddleEmbeds | Rebuild a real <iframe> from a JSFiddle loader script onto the fiddle's own page | | rebuildVokiEmbeds | Rebuild a real <iframe> from Voki's script embed onto the scene's share page, reading the scene from the inline AC_Voki_Embed call | | rebuildDeferredIframes | Rebuild a real <iframe> from a URL parked in a <div> attribute (Pym.js data-pym-src, @newswire/frames data-frame-src) | | rebuildGofundmeEmbeds | Rebuild a real <iframe> from a GoFundMe campaign widget's empty div.gfm-embed[data-url] | | rebuildPiktochartEmbeds | Rebuild a real <iframe> from a Piktochart script snippet's div.piktowrapper-embed, keeping any prose the publisher wrote inside it | | linkifyGistEmbeds | Replace a GitHub Gist script embed, an <amp-gist> or a gist loader's unfilled mount with a link to the gist | | linkifyPodomaticPlaylists | Replace a Podomatic Flash player set to a podcast's whole feed with a link to the podcast's page | | linkifyPaypalDonateForms | Replace a PayPal donate button form with its button image linked to PayPal's donate page | | fixSubstackMentions | Rebuild a Substack @-mention (empty span.mention-wrap) into an inline <a>@name</a> link, so the name survives instead of vanishing mid-sentence | | fixSubstackImageLinks | Remint the <img> inside an emptied Substack lightbox anchor (a.image-link whose image child was stripped) from the anchor's own image href | | convertNoteEmbeds | Convert note.com's empty embed figures (figure[embedded-service][data-src]): media services become plain iframes for the widget pass, own-post embeds become plain links | | convertAmpNativeElements | Convert AMP custom elements with a native equivalent (amp-img, amp-anim, amp-video, amp-audio, amp-iframe) into that element | | convertDatawrapperEmbeds | Convert Datawrapper chart embeds (iframe, script/noscript, and link forms) into a static image linking to the interactive chart | | convertTradingEconomicsEmbeds | Replace a Trading Economics chart iframe with the static chart image its page shows, linked to the indicator page when the iframe names one | | convertAsciinemaEmbeds | Replace an asciinema loader script with the cast's static SVG render, linked to the recording | | convertGiphyEmbeds | Convert a Giphy gif shipped as an iframe (giphy.com/embed/{id}, media.giphy.com, giphy.com/gifs/{id}) into an <img> of the gif, linked to its Giphy page | | convertSmartframeEmbeds | Convert SmartFrame's <smartframe-embed> element into the picture it names as a static image | | convertParkedMediaContainers | Add a real <video>/<audio> to a container parking a media file URL in a lazy media attribute | | convertWidgets | Convert recognized widgets: embeds become data-embed-* placeholders, platform-hosted media becomes a real <video>/<audio> (from an id template or a media-file src) | | assignVideoPosters | Heuristic (opt-in): move a redundant video-poster image (inline or an enclosure) onto the embed as its poster, then drop the standalone image | | stripDuplicateEnclosures | Heuristic (opt-in): remove an injected enclosure that duplicates inline content (image size-variants, exact audio/video/embed) | | stripDuplicateLeadingImages | Heuristic (opt-in): remove a leading image the body repeats as the very next image (featured-image prepends), keeping the larger copy | | stripWordpressEmbedFrames | Remove the frame of a WordPress post embed whose blockquote becomes a cite, so the post shows once | | convertCiteCards | Convert link-preview cards into data-cite-* placeholders | | enrichEmbedPlaceholders | Fill placeholder metadata via the caller's enrichEmbedFn (no-op unless set) | | enrichCitePlaceholders | Fill cite placeholder metadata via the caller's enrichCiteFn (no-op unless set) | | neutralizeUnsafeUrls | Replace dangerous-scheme URLs (and any the isSafeUrlFn option rejects) with an inert sentinel, keeping the element | | proxyAssetUrls | Rewrite media URLs through a caller-supplied proxy, keeping each original in data-proxied-<attr> | | resolveRelativeUrls | Resolve relative URLs to absolute against the base URL | | shortenSamePageLinkFragments | Shorten absolute in-page links back to bare #fragment hrefs | | unwrapWrappers | Remove redundant outer <div> / <article> / <section> wrappers | | unwrapDoublyNestedLists | Unwrap a list that only wraps a single same-type list | | wrapTablesForScroll | Wrap tables in a horizontal-scroll container | | mergeFragmentedLists | Merge consecutive sibling lists of the same type | | paragraphizePlainText | Wrap bare plain text in <p> tags | | stripOversizedBase64Sources | Drop oversized inline base64 media sources before parsing | | stripWordBreaks | Remove <wbr> tags so split URLs rejoin before linkifying | | linkifyUrls | Wrap bare URLs in links | | markTimestamps | Wrap line-leading timestamps for player seeking | | stripLeadingIndentation | Strip fake leading indentation (nbsp / fixed-width spaces) from block text | | trimPreWhitespace | Remove shared leading indentation from <pre> blocks | | unwrapNestedCodeWrappers | Collapse redundant code-in-code / pre-in-pre double-wraps | | highlightCode | Syntax-highlight code blocks that declare a language and expose the language for a badge | | stripDuplicateRules | Collapse a run of thematic breaks to the first one | | stripMarkdownEscapeBackslashes | Strip leaked Markdown escape backslashes at paragraph starts | | stripEmptyTags | Remove empty elements | | stripComments | Remove HTML comments | | unwrapCdataComments | Unwrap malformed <!--[CDATA[ … ]]--> wrappers before parsing | | unwrapCdataMarkers | Unwrap a whole-value <![CDATA[ … ]]> marker so content isn't dropped | | stripControlChars | Strip rendering-hostile control characters before parsing |

An embed placeholder states how big it is in one of two ways, never both. Where something really measured the player, it carries data-embed-width and data-embed-height in pixels, or just one of them where that is all the platform states (a podcast player 200 pixels tall has no width worth naming). Where nothing measured it and only the shape is known, from a responsive wrapper or the platform's own ratio attribute, it carries data-embed-ratio instead: a CSS aspect-ratio value written from the numbers the source stated, such as 16/9, 800/600 or 1.7777777777777777/1, and ready to assign to style.aspectRatio as it stands. Nothing is reduced or rounded, so the value traces back to what the markup said. A player that adds a fixed panel below its video, such as a tracklist or a toolbar, also carries data-embed-ratio-height-offset: the pixels to add to the ratio's height, so a reader can size it as padding-bottom: calc(100% * 9 / 16 + 175px). It appears only beside data-embed-ratio.

A placeholder can lack data-embed-src when only a fetch can find the player, such as a Typeform live embed, whose id Typeform's api maps to the form. It still carries data-embed-provider and data-embed-id, so an enrichEmbedFn can fill the player in, and a reader shows nothing for it until then.

A placeholder may also carry data-embed-params: settings the publisher chose for that one embed that a reader may override, such as the language of a widget's labels, written as a query string like l=german. They are kept off data-embed-src, so a reader appends them when it builds the frame, or sets its own in their place.

data-embed-src keeps the scheme the feed wrote, so an older embed can still point at an http: player. A browser blocks an http: frame on an https: page as mixed content, so a reader served over https: should send Content-Security-Policy: upgrade-insecure-requests. Chromium, Firefox and WebKit then load the frame over https:, which plays wherever the platform still serves the player there.

Options

import { fixLazyImages, resolveRelativeUrls, stripControlChars, transformContent } from 'feedsweep'
import { parseHtml } from 'feedsweep/linkedom'
import { cleanUrl, defaultUnwrappers, downloadMeasurementUnwrappers, imageProxyUnwrappers } from 'urlpurify'

const result = transformContent(html, {
  // Required: function that turns an HTML string into a `Document`. See "DOM library".
  parseHtmlFn: parseHtml,
  // Base URL for resolving relative URLs.
  baseUrl: 'https://example.com/post/1',
  // Other URLs of the same item, such as its guid. An in-page link written against one of them shortens to a fragment.
  sameSiteUrls: ['https://example.com/?p=1'],
  // Resolve a relative URL against the base URL (defaults to standard URL resolution).
  resolveUrlFn: (url, baseUrl) => resolve(url, baseUrl),
  // Rewrite anchor hrefs: unwrap redirects and strip tracking params, also on the url and the src of embed placeholders.
  cleanUrlFn: cleanUrl,
  // Clean a url only to compare it with another, as every duplicate check and the emoji
  // recognition do. The output keeps the url cleanUrlFn gives. Defaults to cleanUrlFn. The image
  // proxy unwrappers let a resized copy match its source and a proxied emoji be recognised.
  matchUrlFn: (url) =>
    cleanUrl(url, {
      unwrappers: [...defaultUnwrappers, ...downloadMeasurementUnwrappers, ...imageProxyUnwrappers],
    }),
  // Feed item enclosures (audio/video/image), injected into the content.
  enclosures: [{ url: 'https://example.com/audio.mp3', type: 'audio/mpeg' }],
  // Images also attached to the feed itself (logo, icon). An item enclosure that repeats one of them is not injected.
  feedImageUrls: ['https://example.com/logo.png'],
  // Route image/video/audio URLs through a proxy. Return `undefined` to leave a URL untouched.
  assetProxyFn: (url, type) => `https://proxy.example.com/?type=${type}&url=${encodeURIComponent(url)}`,
  // Extra URL safety policy (e.g. SSRF/allowlist); return `false` to neutralize. A dangerous-scheme floor always applies.
  isSafeUrlFn: (url, type) => isSafe(url, type),
  // Populate embed placeholder metadata from a remote source (e.g. YouTube oEmbed). Called once
  // per document with every embed, each carrying its provider and id plus the page url and player
  // src when the placeholder has them; answer positionally, one entry per embed in the same order,
  // undefined where nothing was found.
  enrichEmbedFn: (embeds) => Promise.all(embeds.map(({ provider, id, url }) => fetchMetadata(provider, id, url))),
  // Populate cite placeholder fields the card markup does not carry. Positional, like enrichEmbedFn.
  enrichCiteFn: (cites) => Promise.all(cites.map(({ provider, url }) => fetchCite(provider, url))),
  // Normalize a cite card's site-formatted display date (e.g. "2018.10.14"); return
  // undefined to keep the raw string verbatim.
  parseDateFn: (raw) => parseDate(raw),
  // Swap the code highlighter (defaults to highlight.js; may be async).
  highlightFn: (text, language) => myHighlighter.highlight(text, language),
  // The item's title. A body that opens with a heading repeating it loses that heading.
  articleTitle: 'Post title',
  // Opt into the heuristic transforms. Ignored if a custom domTransforms is set.
  heuristics: true,
  // Run a custom string transform pipeline before parsing (omit to use defaults).
  stringTransforms: [stripControlChars],
  // Run a custom DOM transform pipeline (omit to use defaults).
  domTransforms: [fixLazyImages, resolveRelativeUrls],
})

All caller-provided functions (parseHtmlFn, resolveUrlFn, cleanUrlFn, matchUrlFn, assetProxyFn, isSafeUrlFn, enrichEmbedFn, enrichCiteFn, parseDateFn, highlightFn) must not throw — an exception is not caught and rejects the transformContent promise.

Code blocks are highlighted only when they declare a language (language-* class, data-language, Pandoc/Rouge/Expressive Code/etc.); unlabeled blocks are left plain rather than guessed at. The default highlighter is highlight.js (exported as defaultHighlightFn / hljsHighlightFn); replace it with highlightFn.

The stringTransforms and domTransforms options each fully replace the corresponding default phase when provided. The default string phase is exported as defaultStringTransforms. The heuristics flag (default false) selects between two exported DOM pipelines: defaultStandardDomTransforms (the safe defaults) and defaultAllDomTransforms (standard plus heuristicDomTransforms spliced in after injectEnclosures). Setting domTransforms explicitly overrides heuristics. Every transform and pipeline is also exported individually from feedsweep, so you can compose any pipeline — list transforms explicitly, or spread defaultStandardDomTransforms / heuristicDomTransforms to extend or filter the defaults.

The platforms feedsweep recognizes, the hosts it treats as trackers, the selectors it strips as non-content, the lazy-loading attributes it reads and the player labels it strips from a title or description are all built in and not configurable. A platform or attribute that is missing belongs in the library: open an issue or a pull request.

Feedsweep is not a sanitizer. Apart from the dangerous-scheme floor on URLs, markup such as <script> and on* handlers passes through as the feed wrote it. Run an HTML sanitizer on the output, not before feedsweep: some transforms decode markup a feed parks in an attribute, so a sanitizer that ran first never saw it.

DOM library

Feedsweep is parser-agnostic. You provide parseHtmlFn — a function that turns an HTML string into a Document. Use any DOM library that produces a standards-compliant Document. The test suite runs the full pipeline against both linkedom and jsdom.

// linkedom (recommended default)
import { transformContent } from 'feedsweep'
import { parseHtml } from 'feedsweep/linkedom'

await transformContent(html, { parseHtmlFn: parseHtml, baseUrl })

// jsdom
import { transformContent } from 'feedsweep'
import { JSDOM } from 'jsdom'

await transformContent(html, {
  parseHtmlFn: (raw) => new JSDOM(`<!doctype html><body>${raw}</body>`).window.document,
  baseUrl,
})

// happy-dom
import { transformContent } from 'feedsweep'
import { Window } from 'happy-dom'

await transformContent(html, {
  parseHtmlFn: (raw) => {
    const window = new Window()
    window.document.body.innerHTML = raw
    return window.document
  },
  baseUrl,
})

The bundled feedsweep/linkedom parser bakes in two workarounds for linkedom-specific spec violations (attribute case-folding and SVG XML mode). jsdom and happy-dom do not need them.