@olostep/mastra-tools

Olostep is a Web search, scraping and crawling API — an API to search, extract and structure web data. This package adds an Olostep Integration to Mastra so your agents and workflows can search, scrape, crawl and map websites with a few lines of code.

Navigation

• Installation
• What you can build
• APIs (parameters, responses, use cases)
• Quick examples
• Parsers
• Support & Resources

Installation

npm install @olostep/mastra-tools

Register the Integration in your Mastra setup:

import { Mastra } from '@mastra/core';
import { createOlostepIntegration } from '@olostep/mastra-tools';

const olostep = createOlostepIntegration();
olostep.registerApis(); // exposes the 4 Olostep APIs to Mastra

export const mastra = new Mastra({
  config: {
    integrations: [olostep],
  },
});

What you can build

• Research agents that search, extract, and summarize web content
• E‑commerce price/stock monitors across thousands of product pages
• SEO workflows that map sites, scrape content, and analyze topics
• Competitive intelligence dashboards powered by live web data
• Knowledge ingestion pipelines for RAG

Available APIs

All APIs accept your Olostep API key (temporary schema parameter) and return Olostep’s structured responses.

1) scrapeWebsite

Extract content from a single URL. Supports HTML, Markdown, JSON, and text. Optional wait to render JS content.

Parameters:

• apiKey (string, required): Your Olostep API key
• url_to_scrape (string, required): URL including http/https
• formats (array, default ['markdown']): 'html' | 'markdown' | 'json' | 'text'
• country (string, optional): 2‑letter country code (e.g. 'US')
• wait_before_scraping (number, optional): milliseconds (0–10000)
• parser (string, optional): e.g. '@olostep/amazon-product'

Response highlights:

• object: 'scrape'
• id, created
• result.markdown_content / html_content / json_content / text_content
• result.*_hosted_url (hosted assets), screenshot_hosted_url

Use cases:

• Page monitoring, product extraction, article ingestion

2) batchScrape

Scrape many URLs in parallel (up to 100k).

Parameters:

• apiKey (string, required)
• batch_array (array, required): [{ url, custom_id? }, ...]
• formats (array, default ['markdown'])
• country, wait_before_scraping, parser (optional)

Response highlights:

• object: 'batch'
• batch_id, status

Use cases:

• Catalog scraping, bulk enrichment, multi‑source ingestion

3) createCrawl

Autonomously discover and scrape pages by following links.

Parameters:

• apiKey (string, required)
• start_url (string, required)
• max_pages (number, default 10)
• follow_links (boolean, default true)
• formats (array, default ['markdown'])
• country, parser (optional)

Response highlights:

• object: 'crawl'
• id, status, created

Use cases:

• Docs/blog ingestion, knowledge base building, site archiving

4) createMap

Extract all URLs from a site for discovery and planning.

Parameters:

• apiKey (string, required)
• url (string, required)
• search_query (string, optional)
• top_n (number, optional)
• include_urls / exclude_urls (array/string, optional)

Response highlights:

• object: 'map'
• id, url, total_urls, urls[]

Use cases:

• Sitemap building, pre‑batch discovery, SEO site audits

Quick examples

Call an API from Mastra:

const result = await mastra.callApi({
  integrationName: 'olostep',
  api: 'scrapeWebsite',
  payload: {
    data: {
      apiKey: process.env.OLOSTEP_API_KEY!,
      url_to_scrape: 'https://example.com',
      formats: ['markdown'],
      country: 'US',
    }
  }
});

Agent that can use Olostep:

import { Agent } from '@mastra/core';

const researchAgent = new Agent({
  name: 'research-assistant',
  instructions: `
    You can search, extract and structure web data with Olostep.
    Use scrapeWebsite for single pages, createMap to discover pages,
    and batchScrape for bulk extraction. Summarize findings.
  `,
  model: 'openai/gpt-4',
});

Parsers (optional)

Use specialized parsers for popular sites to get structured fields automatically:

• @olostep/amazon-product
• @olostep/linkedin-profile
• @olostep/linkedin-company
• @olostep/google-search
• @olostep/google-maps
• @olostep/instagram-profile

Example:

await mastra.callApi({
  integrationName: 'olostep',
  api: 'scrapeWebsite',
  payload: { data: {
    apiKey: process.env.OLOSTEP_API_KEY!,
    url_to_scrape: 'https://www.amazon.com/dp/PRODUCT_ID',
    formats: ['json'],
    parser: '@olostep/amazon-product',
  }}
});

Security

• API key via environment variables
• No secrets in code

Support & Resources

• Docs: https://docs.olostep.com
• Dashboard (API key): https://olostep.com/dashboard
• Support: https://olostep.com/support