@asno-dev/oley

Zero-dependency TypeScript/JavaScript client for the Oley Web Intelligence API. Scrape, crawl, extract, search, screenshot, PDF, NLP, and prepare web data for AI.

Works in Node 18+, Bun, Deno, and browsers.

Install

npm install @asno-dev/oley

Quick Start

import { Oley } from '@asno-dev/oley';

const oley = new Oley({
  baseUrl: 'http://localhost:3000', // or process.env.OLEY_BASE_URL
  apiKey: 'your-api-key',           // optional, or process.env.OLEY_API_KEY
});

// Scrape a page
const page = await oley.scrape('https://example.com', {
  formats: ['markdown', 'metadata'],
});
console.log(page.markdown);

// Take a screenshot
const screenshot = await oley.screenshot('https://example.com');

// Deep research (Perplexity-style)
const research = await oley.research('What is the best Firecrawl alternative?');
console.log(research.answer);

Configuration

new Oley({
  baseUrl?: string;  // Default: process.env.OLEY_BASE_URL || 'http://localhost:3000'
  apiKey?: string;   // Default: process.env.OLEY_API_KEY
  timeout?: number;  // Default: 60000 (ms)
})

API Methods

Unified Endpoint

| Method | Description | |--------|-------------| | fire(req) | Single endpoint for all operations |

Scraping

| Method | Description | |--------|-------------| | scrape(url, opts?) | Full scrape with configurable formats | | markdown(url, opts?) | Markdown only | | text(url, opts?) | Plain text only | | links(url, opts?) | Links with internal/external counts | | images(url, opts?) | Images | | videos(url, opts?) | Videos | | headings(url, opts?) | Heading structure | | metadata(url, opts?) | Page metadata | | page(url, opts?) | All data (html, markdown, text, metadata, links, images, videos, headings) |

Article / Readability

| Method | Description | |--------|-------------| | article(url) | Readability article extraction | | readability(url) | Raw Readability result |

Schema / Structured Data

| Method | Description | |--------|-------------| | schema(url, opts?) | JSON-LD, Microdata, OpenGraph extraction |

Search

| Method | Description | |--------|-------------| | search(query, opts?) | Web search (DuckDuckGo/Google) |

Crawl

| Method | Description | |--------|-------------| | crawl(url, opts?) | Start async BFS crawl | | crawlStatus(id) | Get crawl status/results |

Map (URL Discovery)

| Method | Description | |--------|-------------| | map(url, opts?) | Discover URLs via sitemap or page crawl |

Screenshot / PDF

| Method | Description | |--------|-------------| | screenshot(url, opts?) | Page screenshot (returns ArrayBuffer) | | pdf(url) | Page PDF (returns ArrayBuffer) |

Render

| Method | Description | |--------|-------------| | render(url) | JS-rendered HTML + console/network logs |

LLM Extract

| Method | Description | |--------|-------------| | llmExtract(url, schema) | Extract with LLM using a JSON schema | | llmExtractWithPrompt(url, prompt) | Extract with a natural language prompt |

Intelligence

| Method | Description | |--------|-------------| | company(url) | Company info extraction | | contacts(url) | Contact extraction (emails, phones) | | pricing(url) | Pricing data | | products(url) | Product data | | reviews(url) | Reviews | | jobs(url) | Job listings |

Batch & Pipeline

| Method | Description | |--------|-------------| | batch(req) | Batch scrape multiple URLs | | batchStatus(id) | Get batch job status | | pipeline(steps) | Multi-step processing pipeline |

Deep Research

| Method | Description | |--------|-------------| | research(query, opts?) | Perplexity-style deep research | | synthesize(query, sources) | Synthesize answer from sources |

AI / NLP

| Method | Description | |--------|-------------| | summarize(url, opts?) | Text summarization | | entities(url) | Named entity extraction | | keywords(url) | Keyword extraction | | topics(url) | Topic classification | | sentiment(url) | Sentiment analysis | | chunk(url, opts?) | RAG text chunking | | qa(url, opts?) | Q&A pair generation | | translate(url, lang) | Translation | | rewrite(url, style?) | Text rewriting |

LLM Chat

| Method | Description | |--------|-------------| | chat(prompt, opts?) | Direct LLM call | | llmReady(url) | LLM-ready formatting |

Search Index (Own Engine)

| Method | Description | |--------|-------------| | searchIndex(url) | Index a page | | searchQuery(query) | Query the index | | searchSpider(url) | Start a spider crawl | | searchSpiderStatus(id) | Spider status | | searchStats() | Index stats | | searchIndexClear() | Clear the index | | searchCacheStats() | Cache stats | | searchCacheClear() | Clear cache |

Webhooks

| Method | Description | |--------|-------------| | webhookRegister(url, events) | Register a webhook | | webhooks() | List webhooks |

Cache

| Method | Description | |--------|-------------| | cacheInvalidate(url) | Invalidate URL cache | | cacheStats() | Cache statistics | | cacheClear() | Clear all cache |

System

| Method | Description | |--------|-------------| | health() | Health check |

Examples

Scrape with stealth mode

const result = await oley.scrape('https://example.com', {
  formats: ['markdown', 'html', 'metadata'],
  stealth: true,
  renderJs: true,
  onlyMainContent: true,
});

Extract structured product data with LLM

const product = await oley.llmExtract('https://example.com/product', {
  name: 'string',
  price: 'number',
  description: 'string',
  inStock: 'boolean',
  reviews: 'number',
});

Crawl a website

const { id } = await oley.crawl('https://example.com', {
  maxPages: 100,
  maxDepth: 3,
  sameDomain: true,
});

// Poll for completion
const status = await oley.crawlStatus(id);
console.log(status.pages, 'pages crawled');

Deep research

const result = await oley.research('Best TypeScript frameworks in 2026', {
  depth: 'deep',
  maxSources: 15,
});

console.log(result.answer);
console.log(result.citations);
console.log(result.followups);

Batch scrape

const job = await oley.batch({
  urls: ['https://example.com', 'https://example.org'],
  formats: ['markdown'],
  concurrency: 5,
});

Pipeline

const result = await oley.pipeline([
  { id: 'scrape', action: 'scrape', input: { url: 'https://example.com' } },
  { id: 'summarize', action: 'summarize', input: { maxLength: 200 }, dependsOn: ['scrape'] },
]);

Error Handling

import { Oley, OleyError } from '@asno-dev/oley';

try {
  await oley.scrape('https://invalid-url');
} catch (err) {
  if (err instanceof OleyError) {
    console.error(err.message, err.status);
  }
}

Environment Variables

| Variable | Description | |----------|-------------| | OLEY_BASE_URL | Default base URL | | OLEY_API_KEY | Default API key |

License

MIT