@aiceafrica/crawler

TypeScript SDK for the AICE Crawler Service multi-tenant web crawler with a page budget and an approve-then-publish flow into the Knowledge Service.

import { CrawlerClient } from "@aiceafrica/crawler"

const client = new CrawlerClient({
  baseUrl: "https://crawler.example.com",
  apiKey: "sk-…",
})

const job = await client.jobs.create({
  startUrls: ["https://docs.example.com"],
  pageBudget: 500,
})

await client.jobs.start(job.id)

const pages = await client.jobs.listPages(job.id)

await client.jobs.approve(job.id)
await client.jobs.publish(job.id)

Endpoints covered:

• POST /jobs, GET /jobs, GET /jobs/{id}
• POST /jobs/{id}/start, GET /jobs/{id}/pages, GET /pages/{id}
• POST /jobs/{id}/approve, POST /jobs/{id}/publish
• PUT /jobs/{id}/schedule, GET /jobs/{id}/runs, POST /jobs/{id}/reclean
• GET /policy, PUT /policy