@zjkm666/scraperapi

Generate a customizable Express scraping API from any URL — just name it and point it.

• Input: just a url to scrape + an API name
• GET / POST endpoints are declarative & easy to understand (CSS selectors + buildUrl for POST body params)
• JS-rendered support via Playwright (toggle per-endpoint)
• Optional cache via node-cache

Install & Generate

# scaffold a new API project
npx @zjkm666/scraperapi
# or non-interactive
npx @zjkm666/scraperapi myShopAPI https://quotes.toscrape.com --out ./myShopAPI

# then
cd myShopAPI
npm install
npm run dev   # http://localhost:3000

CLI options:

npx @zjkm666/scraperapi [name] [url] [options]
  --name <name>        API name
  --url <url>          Source URL to scrape
  --out <dir>          Output directory (default: slugified name)
  --template, -t <id>  Template preset: base, anime, product, video, github, profile, facebook, x, movie
  --list-templates     List all templates and exit
  --render-js          Enable Playwright in template
  --no-cache           Disable cache in template
  --yes, -y            Non-interactive

Templates

Pre-built endpoint configurations for common scraping targets:

| Template | Description | JS-rendered? | Endpoints | |---|---|---|---| | base | Generic customizable API (default) | No | /products, /search | | anime | Anime listings, rankings, details | No | /anime, /top, /search, /anime-detail | | product | Product catalogs, search, detail | No | /products, /search, /product | | video | Video listings, trending, detail | Yes | /videos, /trending, /search, /video-detail | | github | Repositories, trending, repo detail | No | /repos, /trending, /search, /repo | | profile | User profiles, user repos, search | No | /profile, /user-repos, /search | | facebook | Public posts, page info, search | Yes | /posts, /search, /page-info ⚠ ToS | | x | Tweets, trends, search, profiles | Yes | /tweets, /trending, /search, /profile ⚠ ToS | | movie | Movie charts, search, detail | No | /movies, /top-rated, /search, /movie-detail |

Use a template:

# Interactive (prompts for name, URL, etc.)
npx @zjkm666/scraperapi --template anime

# Non-interactive
npx @zjkm666/scraperapi --template product --yes --name myShop --url https://example.com --out ./myShop

# List all templates
npx @zjkm666/scraperapi --list-templates

⚠ Legal warning: The facebook and x templates include ToS warnings. Scraping these platforms may violate their Terms of Service. Respect robots.txt, cache aggressively, and consider official APIs for production use.

For JS-rendered sites:

cd myShopAPI
npm install playwright
npx playwright install chromium
# set renderJs: true in api.config.js

Customizing endpoints

All endpoints live in api.config.js — no need to touch server.js for most changes.

// api.config.js
module.exports = {
  name: 'myShopAPI',
  sourceUrl: 'https://example.com',
  cache: { enabled: true, ttl: 300 },
  endpoints: [
    {
      method: 'GET',
      path: '/products',
      selector: '.product-card',
      fields: {
        title: 'h2',
        price: '.price',
        link: 'a@href',
        image: 'img@src',
      },
    },
    {
      method: 'POST',
      path: '/search',
      selector: '.result',
      fields: { title: 'h2', link: 'a@href' },
      buildUrl: (baseUrl, req) => `${baseUrl}/search?q=${encodeURIComponent(req.body.query)}`,
    },
  ],
};

Field shorthand:

| Def | Meaning | |---|---| | "h2" | text of <h2> | | "a@href" | attribute | | ".price | .cost" | fallback | | { selector: "img", attr: "src" } | object form | | (el,$)=> $(el).data('id') | custom function |

Library usage (no scaffolding)

You can also use the package programmatically:

const { createScraperAPI } = require('@zjkm666/scraperapi');

const app = createScraperAPI({
  name: 'demo',
  sourceUrl: 'https://quotes.toscrape.com',
  cache: { enabled: true, ttl: 300 },
  endpoints: [
    { method: 'GET', path: '/quotes', selector: '.quote', fields: { text: '.text', author: '.author' } },
    {
      method: 'POST',
      path: '/search',
      selector: '.quote',
      fields: { text: '.text' },
      buildUrl: (base, req) => `${base}/search?q=${encodeURIComponent(req.body.query || '')}`,
    },
  ],
});

app.listen(3000, () => console.log('http://localhost:3000'));

Low-level helpers are also exported:

const { scrape, fetchHtml } = require('@zjkm666/scraperapi');
const data = await scrape({ url: 'https://example.com', selector: '.item', fields: { title: 'h2' }, renderJs: true });

Template registry (programmatic)

const { templates } = require('@zjkm666/scraperapi');

templates.listTemplates();           // [{ slug, name, description, recommendedSource, renderJs, warning? }]
templates.getTemplate('anime');      // { slug, name, description, recommendedSource, renderJs, warning? }
templates.isValidTemplate('anime');  // true/false
templates.getTemplateSlugs();        // ['base', 'anime', 'product', ...]

Generated project structure

myShopAPI/
├── api.config.js   # ← edit this
├── server.js       # Express server (auto-registers endpoints)
├── scraper.js      # axios+cheerio (+ optional playwright)
├── package.json
├── .env.example
└── README.md

Cache

• Global: cache: { enabled: true, ttl: 300 }
• Per-endpoint: cache: false or cache: { ttl: 60 }
• Clear: curl -X DELETE http://localhost:3000/cache
• Stats: curl http://localhost:3000/cache/stats

JS-rendered

Per-endpoint renderJs: true uses Playwright headless Chromium. Useful for React/Vue/SPAs.

{ method: 'GET', path: '/dynamic', selector: '.card', renderJs: true, fields: {...}, fetchOpts: { waitForSelector: '.card' } }

Legal

Scraping may be subject to the target's robots.txt and Terms of Service. Cache aggressively, respect rate limits, set a proper User-Agent. You are responsible for compliance.

License

ISC