docs-scraper

To install dependencies:

bun install

To run:

bun run index.ts

This project was created using bun init in bun v1.1.30. Bun is a fast all-in-one JavaScript runtime.

Supported Links

• [ ] Notion
• [ ] Docsend
• [ ] Pitch.com
• [ ] Google slides
• [ ] Google drives
• [ ] PDF links

How it works?

1. User sends link via /api/v1/scrape with url and JSON object data that may contain email, password or other values
2. A spin up Playwright to interact with the page
3. For templated projects we will follow our scripts, if blockers are met, it stores the status and required items to proceed
4. User polls /api/v1/results/:jobId to check scraping result and status. If blocked, users should provide blocking items via ``api/v1/update`
5. Scraped content gets stored into a PDF file.
6. Each scrape returns an array of PDF file. For single page link, it returns an array of single PDF file link. For pages that may contain multiple scraping jobs (e.g. Docsend folders), we will return an array of pdf link.
7. /api/v1/scrape returns sessionId and jobId. To scrape with an old session (e.g. to avoid signing in twice), pass sessionId as parameter
8. Falls back to using LLM to scrape when links are not supported.

API Endpoints

| Method | Endpoint | description | | ------ | ---------------------- | ------------------- | | POST | /api/v1/scrape | Submit job | | POST | /api/v1/update | Provide auth data | | GET | /api/v1/results/:jobId | Poll for results | | GET | /api/v1/jobs | List jobs | | DELETE | /api/v1/jobs/:jobId | Cancel job | | GET | /api/v1/sessions | List saved sessions | | DELETE | /api/v1/sessions | Clear sessions |

Tech Stack

• Typescript + express
• BullMQ to queue tasks
• Playwright to interact with links
• Cloudflare R2 to store scraped content as PDF files

Auth

Use a simple api secret to gate access to this service.

1. Configure API_SECRET in your env variables
2. All requests must include x-api-secret to access the service