ScrapeGraphAI Tools for Vercel AI SDK

ScrapeGraphAI tools for the Vercel AI SDK - Smart web scraping powered by AI.

Installation

npm install ai-sdk-scrapegraph-tools

Prerequisites

Important: You must set your ScrapeGraph API key as an environment variable before using this package:

export SGAI_APIKEY=your_api_key_here

Or create a .env file:

SGAI_APIKEY=your_api_key_here

You can get your API key from ScrapeGraphAI Dashboard.

Usage

import { smartScraperTool } from "ai-sdk-scrapegraph-tools";
import { generateText } from "ai";
import { gateway } from "ai";

const result = await generateText({
  model: gateway("openai/gpt-4o-mini"),
  prompt: "Scrape information about ScrapeGraph AI from https://scrapegraphai.com/",
  tools: {
    smartScraperTool,
  },
});

console.log(result.text);

Available Tools

This package provides the following ScrapeGraph tools:

• smartScraperTool - Extract structured data from websites using AI
• scrapeTool - Get raw HTML content from webpages
• searchScraperTool - Search and extract information from multiple web sources
• markdownifyTool - Convert webpages to clean markdown format
• sitemapTool - Extract all URLs from a website's sitemap
• localScraperTool - Extract structured data from local HTML content
• crawlTool - Start crawl jobs to extract data from multiple pages
• getCrawlRequestTool - Check the status of crawl jobs
• agenticScraperTool - Perform automated actions on webpages
• getCreditsTool - Check your API credits balance
• healthzTool - Check the health status of the ScrapeGraph API
• sendFeedbackTool - Submit feedback for requests
• generateSchemaTool - Generate JSON schemas from natural language prompts
• toonifyTool - Convert images to cartoon style

See the full documentation for more details.

Development

Setup

1. Clone the repository
2. Install dependencies:

pnpm install

3. Create a .env file:

SGAI_APIKEY=your_scrapegraph_api_key_here

4. Add your ScrapeGraph API key to .env. You can get your API key from ScrapeGraphAI Dashboard.

Note: You also need to configure your Vercel AI Gateway credentials if you're using the gateway model provider.

Testing

Test your tool locally:

pnpm test

Building

Build the package:

pnpm build

Publishing

Before publishing, update the package name in package.json to your desired package name.

The package automatically builds before publishing:

pnpm publish

Project structure

.
├── src/
│   ├── index.ts      # Your tool exports
│   └── test.ts       # Test script
├── dist/             # Build output (generated)
├── package.json
├── tsconfig.json
└── README.md

License

MIT