Scraper App

Intelligent document and web content extraction SDK. Parse web pages, Excel spreadsheets, PowerPoint presentations, Python source files, and more into clean, structured, LLM-ready output.

Features

• Web scraping - Extract articles from any blog or news site (RSS, sitemap, HTML)
• Document parsing - Excel (.xlsx/.xls/.csv), PowerPoint (.pptx), Python (.py), PDF
• LLM-ready output - Markdown, plain text, token counts, and RAG-ready chunks
• Smart routing - Auto-detects input type (URL, file, directory) and dispatches to the right parser
• Batch processing - Parallel multi-document parsing with concurrency control
• 92.2% F1 score - Mozilla Readability extraction validated against the Dragnet benchmark

Installation

npm install @tyroneross/scraper-app

Quick Start

Extract a web article

import { extractArticle } from '@tyroneross/scraper-app';

const article = await extractArticle('https://example.com/blog/post');
console.log(article.title);
console.log(article.markdown);
console.log(`${article.wordCount} words, ${article.readingTime} min read`);

Parse any document (auto-detect)

import { parse } from '@tyroneross/scraper-app';

// URL → web scraper
const web = await parse('https://example.com/article');

// Excel file → structured tables
const excel = await parse('./data/report.xlsx');

// PowerPoint → slides with notes
const pptx = await parse('./deck.pptx');

// Python source → functions, classes, docstrings
const py = await parse('./scripts/main.py');

// Directory → batch parse all supported files
const dir = await parse('./documents/');

LLM-ready output

import { scrapeForLLM } from '@tyroneross/scraper-app/llm';

const { markdown, tokens, chunks, frontmatter } = await scrapeForLLM(url);
// tokens: estimated count for context window management
// chunks: pre-split for RAG applications

Batch processing

import { parseMultiple } from '@tyroneross/scraper-app';

const results = await parseMultiple(
  ['./report.xlsx', './deck.pptx', 'https://blog.example.com/post'],
  { concurrency: 4 }
);

API Reference

Core Functions

| Function | Description | |----------|-------------| | parse(input) | Auto-detect input type and parse (URL, file, directory, raw HTML) | | parseMultiple(inputs, opts) | Parse multiple inputs in parallel | | extractArticle(url) | Extract a single article from a URL | | scrapeWebsite(url, opts) | Discover and extract multiple articles from a site | | smartScrape(url) | Auto-detect single article vs. listing page |

Module Exports

| Import | Use For | |--------|---------| | @tyroneross/scraper-app | Core: parse, parseMultiple, extractArticle, scrapeWebsite, smartScrape | | @tyroneross/scraper-app/llm | LLM output: scrapeForLLM, toLLMFormat, estimateTokens | | @tyroneross/scraper-app/batch | Batch: scrapeUrls, extractArticles | | @tyroneross/scraper-app/parsers | Direct access: parseExcelFile, parsePptxFile, parsePythonFile | | @tyroneross/scraper-app/cache | Caching: createCache, MemoryCache, FileCache | | @tyroneross/scraper-app/validation | Validation: validateUrl, canScrape, isValidUrl | | @tyroneross/scraper-app/testing | Testing: createMockScraper, enableMockMode | | @tyroneross/scraper-app/debug | Debug: enableDebugMode, DebugSession | | @tyroneross/scraper-app/react | React hook: useScraper | | @tyroneross/scraper-app/express | Express router: createScraperRouter | | @tyroneross/scraper-app/optimizations | Performance: createConnectionPool, parallelFetch, fastExtract |

Supported Formats

| Format | Extensions | Parser | |--------|------------|--------| | Web pages | URLs (http/https) | Mozilla Readability + Cheerio | | Excel | .xlsx, .xls, .csv, .tsv, .xlsb, .ods | SheetJS (single-pass optimized) | | PowerPoint | .pptx | Custom ZIP + regex parser (up to 14x faster than node-pptx-parser) | | Python | .py | Static analysis (regex-based, no runtime needed) | | PDF | .pdf | pdf-parse | | HTML | Raw HTML strings | Cheerio + Readability |

Performance

The document parsers have been rebuilt from scratch for speed:

| Parser | Speedup vs. v1 | Method | |--------|----------------|--------| | PowerPoint | 5-14x faster | Single-pass ZIP, regex-first XML extraction, chart/diagram support | | Excel | 1-1.6x faster | Single-pass processing (builds rows, markdown, CSV simultaneously) |

Benchmarked across 24 test files with 100% accuracy.

Examples

Discover articles from a blog

import { scrapeWebsite } from '@tyroneross/scraper-app';

const { articles } = await scrapeWebsite('https://techcrunch.com', {
  maxArticles: 10,
  extractFullContent: true
});

for (const article of articles) {
  console.log(`${article.title} (${article.qualityScore})`);
}

Parse Excel for LLM context

import { parse } from '@tyroneross/scraper-app';

const result = await parse('./quarterly-report.xlsx');
console.log(result.markdown); // Markdown tables ready for LLM
console.log(`${result.estimatedTokens} tokens`);

Validate before scraping

import { validateUrl } from '@tyroneross/scraper-app/validation';

const { isReachable, robotsAllowed, suggestedAction } = await validateUrl(url);
if (!robotsAllowed) {
  console.log('Blocked by robots.txt');
}

Express API server

import express from 'express';
import { createScraperRouter } from '@tyroneross/scraper-app/express';

const app = express();
app.use('/api/scraper', createScraperRouter());
// POST /api/scraper/scrape, POST /api/scraper/extract, POST /api/scraper/validate

React hook

import { useScraper } from '@tyroneross/scraper-app/react';

function MyComponent() {
  const { scrape, data, isLoading, error } = useScraper();
  return <button onClick={() => scrape(url)}>Extract</button>;
}

Configuration

import { configure } from '@tyroneross/scraper-app';

// Suppress console output in production
configure({ quiet: true });

Caching

import { createCache } from '@tyroneross/scraper-app/cache';

const cache = createCache({ provider: 'memory', ttlMs: 3600000 });

Rate limiting

import { createRateLimiter } from '@tyroneross/scraper-app';

const limiter = createRateLimiter('moderate');
// Presets: 'conservative' (1 req/s), 'moderate' (2 req/s), 'aggressive' (4 req/s)

Testing

Mock mode (no network)

import { enableMockMode, disableMockMode } from '@tyroneross/scraper-app/testing';

enableMockMode();
const article = await extractArticle('https://any-url.com'); // returns mock data
disableMockMode();

F1 score validation

# Dragnet benchmark (91.4% F1, validates 92.2% claim)
npm run test:f1:dragnet

# Quick custom test
npm run test:f1

Development

# Install dependencies
npm install

# Run dev server (web UI)
npm run dev

# Build SDK
npm run build:sdk

# Type check
npm run typecheck

License

MIT

Links

• npm package
• [GitHub](https://github.com/tyroneross/Scraper App)
• [Issues](https://github.com/tyroneross/Scraper App/issues)