@posty5/product-scraper

Multi-platform product scraper for Node.js. Give it a product URL from Amazon, noon, AliExpress — or any storefront at all — and get back one normalised Product.

The page is fetched once and passed through a provider pipeline: the platform provider for that site first, then a schema.org/SEO-data provider, then a generic OpenGraph/heuristics provider. Later tiers only fill what earlier ones missed, so a selector change on Amazon degrades the result instead of breaking it.

Nothing is cached. No page, DOM, product, or exchange rate survives the call that produced it.

npm install @posty5/product-scraper

Requires Node 20.18.1 or newer. That floor comes from cheerio, whose undici dependency reads the File global at import time — a global that only exists from Node 20. On Node 18 the package fails at import, not at call time. Node 18 has been end-of-life since April 2025.

Quick start

import { scrapeProduct } from '@posty5/product-scraper';

const result = await scrapeProduct('https://www.amazon.eg/dp/B0CX23V2ZK');

if (result.status === 'ok') {
  console.log(result.product.name);     // 'Anker 737 Power Bank (PowerCore 24K)'
  console.log(result.product.price);    // 1299
  console.log(result.product.currency); // 'EGP'
  console.log(result.platform);         // 'amazon'
}

How the pipeline works

                    ┌──────────────────────────────┐
   product URL ───► │  fetch the page — ONCE       │
                    └──────────────┬───────────────┘
                                   │  PageDocument (HTML + one shared DOM)
                    ┌──────────────▼───────────────┐
              (1)   │  platform provider           │   amazon / noon / aliexpress
                    │  matched on the URL          │   — whichever claims the URL
                    └──────────────┬───────────────┘
                                   │  still missing a required field?
                    ┌──────────────▼───────────────┐
              (2)   │  seo-schema provider         │   schema.org JSON-LD
                    │  works on any storefront     │   (the SEO product data)
                    └──────────────┬───────────────┘
                                   │  still missing a required field?
                    ┌──────────────▼───────────────┐
              (3)   │  generic-fetch provider      │   OpenGraph, Twitter cards,
                    │  last resort                 │   microdata, DOM heuristics
                    └──────────────┬───────────────┘
                                   ▼
                          merged, normalised Product

Three rules govern the whole thing:

1. One fetch. The PageDocument holds the HTML and one lazily-parsed cheerio DOM, shared by every provider. Running three providers costs one HTTP request and one parse.
2. First provider wins per field. A fallback can only fill a field that is still empty — a generic price guess never overwrites Amazon's exact price.
3. Stop when done. The pipeline halts as soon as every required field (name and price by default) is filled. Remaining providers are marked skipped in the trace.

A provider that throws is recorded in the trace and the pipeline moves on, so one broken selector can't fail a scrape a later tier could still satisfy.

The result

interface ScrapeResult {
  url: string;              // what you passed in
  finalUrl?: string;        // after redirects, when different
  status: 'ok' | 'error';
  product?: Product;        // present when status === 'ok'
  platform?: string;        // 'amazon' | 'noon' | 'aliexpress' | 'unknown'
  complete?: boolean;       // were all required fields filled?
  error?: string;
  trace: ProviderTrace[];   // what each provider did
}

interface Product {
  name?: string;
  description?: string;
  images: string[];         // highest quality first
  price?: number;
  originalPrice?: number;   // list / "was" price, when discounted
  currency?: string;        // ISO 4217
  rating?: number;          // normalised to a 0–5 scale
  reviewsCount?: number;
  estimatedSales?: number;  // from "N bought recently" style badges
  availability?: 'in_stock' | 'out_of_stock' | 'preorder' | 'unknown';
  brand?: string;
  sku?: string;             // ASIN on Amazon, SKU on noon, item id on AliExpress
  categories?: string[];    // breadcrumbs, broadest first
  canonicalUrl?: string;
  platform?: string;
}

status: 'error' never throws — check status and read error. Invalid options (a malformed targetCurrency, an empty urls array) do throw, because those are programming mistakes rather than page problems.

Reading the trace

The trace is how you find out why a field came back the way it did:

const result = await scrapeProduct(url);

for (const step of result.trace) {
  console.log(step.provider, step.kind, step.skipped ? 'skipped' : step.fields);
}
// amazon        platform  [ 'name', 'images', 'currency', 'rating', … ]
// seo-schema    schema    [ 'price' ]
// generic-fetch generic   skipped

Supported platforms

| Provider | Sites | Reads | | --- | --- | --- | | amazon | every amazon.* marketplace, amzn.to, a.co | server-rendered DOM + the colorImages script block for full-resolution gallery images | | noon | noon.com (Egypt, Saudi Arabia, UAE) | the __NEXT_DATA__ payload, located by shape; data-qa selectors as backup; currency from the country segment of the URL | | aliexpress | aliexpress.com and the regional editions | the inline JSON global (window.runParams or _dida_config_._init_data_), both dialects | | seo-schema | anything with JSON-LD | schema.org Product — including @graph wrappers, AggregateOffer, ImageObject images, and JSON-LD with comments or trailing commas | | generic-fetch | anything at all | og:* / twitter:* meta tags, product:price:*, schema.org microdata, breadcrumbs, and price/rating DOM heuristics |

The two fallback tiers are why an unlisted store often still works:

// Never seen this shop, but it publishes JSON-LD for Google Shopping.
const result = await scrapeProduct('https://shop.example.de/produkt/kaffeemuehle');
result.platform;          // 'unknown'
result.product?.name;     // 'Kaffeemühle Pro 2000'
result.product?.price;    // 149.9

Currency conversion

Set targetCurrency and pass rates you already hold — no network access, fully deterministic:

const result = await scrapeProduct({
  url: 'https://www.amazon.eg/dp/B0CX23V2ZK',
  targetCurrency: 'USD',
  currencyRates: { USD: 1, EGP: 48.5 },   // all relative to one base
});

result.product?.price;    // 26.78
result.product?.currency; // 'USD'

Omit currencyRates and the package fetches live rates from the keyless open.er-api.com feed — one request per call, since nothing is cached:

const result = await scrapeProduct({ url, targetCurrency: 'USD' });

For a batch, rates are fetched once for the whole batch, not once per product.

If the rate lookup fails or the map doesn't cover the pair, the price stays in its original currency and the result is still ok. A missing rate degrades the result; it never corrupts it.

Using the converter directly

import { CurrencyConverter, convertPrice } from '@posty5/product-scraper';

// Live rates.
const converter = new CurrencyConverter();
await converter.convert(100, 'USD', 'EGP');   // → 4850

// Offline rates — no network at all.
const offline = new CurrencyConverter({ rates: { USD: 1, EGP: 48.5 } });
offline.offline;                              // true
await offline.convert(100, 'USD', 'EGP');     // → 4850

// Converting many prices: fetch the map once, then convert synchronously.
const rates = await converter.rates();
convertPrice(100, 'USD', 'EGP', rates);
convertPrice(250, 'EUR', 'EGP', rates);

Plug in your own feed with a RateProvider:

const converter = new CurrencyConverter({
  provider: {
    name: 'my-api',
    fetchRates: async (base) => {
      const response = await fetch(`https://my-api.test/rates?base=${base}`);
      return (await response.json()).rates;
    },
  },
});

Batch scraping

import { scrapeProducts } from '@posty5/product-scraper';

const batch = await scrapeProducts({
  urls: [amazonUrl, noonUrl, aliexpressUrl],
  concurrency: 2,        // pages in parallel   (default 1)
  delayMs: 2000,         // pause between batches (default 3000)
  targetCurrency: 'EGP',
});

console.log(`${batch.succeeded}/${batch.total} succeeded`);

for (const result of batch.results) {
  if (result.status === 'ok') console.log(result.product?.name);
  else console.warn(result.url, result.error);
}

One URL failing never fails the batch — every entry gets its own ScrapeResult.

Scraping HTML you already have

If the page came from a headless browser, a proxy service, or a fixture, skip the fetch entirely:

import { scrapeProductFromHtml } from '@posty5/product-scraper';

const result = await scrapeProductFromHtml({
  url: 'https://www.aliexpress.com/item/1005006123456789.html',
  html: await playwrightPage.content(),
});

The url must be the real product URL — providers match on it. This entry point performs no network I/O whatsoever (unless you ask for live currency rates).

Reusable scraper

Prefer this when you scrape repeatedly with the same settings:

import { createScraper } from '@posty5/product-scraper';

const scraper = createScraper({
  locale: 'ar_EG',
  targetCurrency: 'EGP',
  timeout: 30_000,
  concurrency: 3,
});

await scraper.scrape(url);
await scraper.scrapeMany([url1, url2]);

Per-call options override the instance defaults.

Adding your own platform provider

A provider is a name, a tier, a URL test, and a function from page to partial product. It never fetches anything — the page is already there.

import { createScraper, defineProvider, matchesDomain, parseAmount } from '@posty5/product-scraper';

const myShop = defineProvider({
  name: 'my-shop',
  kind: 'platform',
  supports: (url) => matchesDomain(url, 'my-shop.com'),

  extract: ({ page }) => ({
    name: page.$('h1.product-title').text().trim(),
    price: parseAmount(page.$('.price-now').text()),
    currency: 'SAR',
    images: page.$('.gallery img').map((_, el) => page.$(el).attr('src')).get(),
  }),
});

const scraper = createScraper().use(myShop);
await scraper.scrape('https://my-shop.com/p/widget');

Your provider runs first for its own domains, and the seo-schema and generic-fetch tiers still back it up for anything you don't extract. Return only the fields you're confident about.

To restrict the chain instead of extending it:

import { createScraper, defaultProviders } from '@posty5/product-scraper';

// Amazon only, but keep both fallback tiers.
const scraper = createScraper({
  providers: defaultProviders().filter(
    (provider) => provider.kind !== 'platform' || provider.name === 'amazon',
  ),
});

Options

Every entry point and the createScraper constructor accept these:

| Option | Default | Notes | | --- | --- | --- | | locale | 'en_US' | Builds the Accept-Language header; also tells providers whether 1.234 means 1234 or 1.234 | | timeout | 20000 | Per-request timeout in ms | | maxRetries | 2 | Retries after the first attempt, for transient errors and bot walls | | retryBackoffMs | 5000 | Multiplied by the attempt number | | userAgent | rotated | Pin one instead of rotating the built-in list | | headers | — | Extra request headers | | signal | — | AbortSignal, honoured by the fetcher and the batch loop | | logger | silent | true for console output, or your own Logger | | requiredFields | ['name', 'price'] | Which fields must be filled before fallbacks stop running | | providers | built-ins | Replace the provider chain for this call | | fetcher | built-in | Replace the page fetcher — proxy pool, headless browser, fixtures | | targetCurrency | — | ISO 4217 code to convert into | | currencyRates | — | Fixed rates, which keep conversion offline | | converter | — | CurrencyConverter used when currencyRates is omitted | | concurrency | 1 | Batch: pages in parallel | | delayMs | 3000 | Batch: pause between batches |

Tuning requiredFields

This is the dial that controls how hard the pipeline works:

// A name is enough — stop as soon as one provider supplies it.
await scrapeProduct({ url, requiredFields: ['name'] });

// Insist on images too, so fallbacks run until a gallery is found.
await scrapeProduct({ url, requiredFields: ['name', 'price', 'images'] });

// Run every provider and merge everything they find.
await scrapeProduct({ url, requiredFields: [] });

Because the page is only fetched once, running more providers costs almost nothing.

Using a proxy or a headless browser

Replace the fetcher. It receives a URL and returns a PageDocument:

import { createPage, createScraper } from '@posty5/product-scraper';
import { chromium } from 'playwright';

const browser = await chromium.launch();

const scraper = createScraper({
  fetcher: async (url, options) => {
    const page = await browser.newPage();
    await page.goto(url, { timeout: options.timeout });
    const html = await page.content();
    await page.close();
    return createPage({ requestedUrl: url, url: page.url(), html });
  },
});

This is the recommended route for AliExpress at scale, and for any site that puts its prices behind client-side rendering and a bot wall.

Blocked pages

Bot walls are detected before parsing and retried with backoff. A page is only treated as blocked when it carries a block signal and no real-product marker — so a product genuinely named "Robot Vacuum" doesn't trip the check.

import { detectBlock } from '@posty5/product-scraper';

detectBlock(html); // { blocked: true, reason: 'captcha' }

Reasons: captcha, rate-limited, bot-wall, empty-response, truncated-response.

When every attempt is blocked, the result is status: 'error' with the reason in error. Slow down (delayMs, concurrency: 1), or route through a proxy or headless browser.

Caching

There is none, by design. Every call fetches the page, parses it, and returns. Exchange rates are fetched per call (once per batch). Nothing is written to disk and nothing is retained between calls.

If you want caching, own it at the boundary — you know your freshness requirements:

const cached = await myCache.get(url);
if (cached) {
  // No network access at all.
  return scrapeProductFromHtml({ url, html: cached });
}

Legal note

Scraping may be restricted by a site's terms of service, by robots.txt, and by local law. You are responsible for how you use this package: check the terms of the sites you target, respect rate limits, and don't republish scraped content you have no right to.

Development

npm install
npm test              # vitest, no network access — everything runs on fixtures
npm run test:coverage
npm run typecheck
npm run lint
npm run build         # tsup → CJS + ESM + .d.ts

Tests run entirely on HTML fixtures in __tests__/fixtures/. Adding a provider means adding a fixture — please don't write tests that hit live sites.

Releasing

Publishing is driven by tags. Pushing a tag that starts with v runs release.yml, which verifies the package and then publishes it to npm. Nothing runs on ordinary pushes or on pull requests, and no version is ever bumped for you — you decide when a release happens by creating the tag.

Cut a release in three commands:

npm run verify

npm version writes package.json, commits, and creates the v<version> tag in one step:

npm version patch

Pushing the tag is what starts the release:

git push --follow-tags

What the workflow does

1. verify — lint, typecheck, tests and build, on Node 20/22/24 in parallel.
2. publish — needs: verify, so a single failing Node version stops the release before anything reaches the registry. It then checks the tag matches package.json (tagging v1.0.1 while the manifest still says 1.0.0 fails the job rather than republishing the wrong number), and publishes with npm provenance.
3. coverage — runs alongside and uploads a coverage artifact. It does not gate the publish.

prepublishOnly runs npm run verify again inside the publish job, so the uploaded tarball is always built from verified source. Only dist/, README.md and LICENSE are packed.

Setup required

Add an NPM_TOKEN repository secret (Settings → Secrets and variables → Actions) holding an npm automation token with publish rights on the @posty5 scope. Without it the publish step fails; verification still runs.

Dry runs

The workflow can also be started manually from the Actions tab, and that path defaults to --dry-run — so the button cannot publish by accident. Untick dry run to publish from a manual run. Locally:

npm publish --dry-run

📖 Resources

• Website: https://posty5.com
• Dashboard: https://studio.posty5.com
• GitHub: https://github.com/Posty5/product-scraper
• Support: https://posty5.com/contact-us

🆘 Support

• Contact Us: https://posty5.com/contact-us
• GitHub Issues: Report bugs or request features

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes
4. Commit your changes: git commit -m 'Add amazing feature'
5. Push to the branch: git push origin feature/amazing-feature
6. Submit a pull request

📄 License

MIT License - see LICENSE file for details.

Made with ❤️ by the Posty5 team