ai-search-indexer

Index documentation websites into JSON artifacts that power the web components in this monorepo.

This package supports two index formats:

• context.json for wg-doc-search (classic search + Prompt API answer)
• vector-context.json for wg-rag-search (Orama + Transformers.js retrieval + Prompt API answer)

Install

pnpm add ai-search-indexer
npx playwright install chromium

Quick Start (Monorepo)

From repo root:

pnpm install

1. Create a Classic Index (context.json)

Example (Angular Material overview pages):

node packages/indexer/examples/index-angular-material.js

Output:

• packages/web-components/demo/context.json

2. Create a Vector Index (vector-context.json)

Example (~50 Angular Material pages: overview + api):

node packages/indexer/examples/index-vector.js

Output:

• packages/web-components/demo/vector-context.json

Programmatic API

WebsiteIndexer (classic)

import { WebsiteIndexer } from 'ai-search-indexer';

const indexer = new WebsiteIndexer('./context.json');
await indexer.indexUrls([
  'https://material.angular.dev/components/table/overview',
  'https://material.angular.dev/components/sort/overview'
]);
indexer.displayStats();

VectorDocIndexer (RAG)

import { VectorDocIndexer } from 'ai-search-indexer';

const indexer = new VectorDocIndexer('./vector-context.json', {
  embeddingModel: 'Xenova/all-MiniLM-L6-v2'
});

await indexer.indexUrls([
  'https://material.angular.dev/components/table/overview',
  'https://material.angular.dev/components/table/api'
]);

indexer.displayStats();

Vector CLI

Run from packages/indexer:

pnpm index:vector --output ./vector-context.json --urls-file ./urls.txt

You can also pass URLs directly:

pnpm index:vector \
  https://material.angular.dev/components/table/overview \
  https://material.angular.dev/components/table/api

CLI options

• --urls-file <path>: .txt or .json URL list
• --output <path>: output path (default ./vector-context.json)
• --model <name>: embedding model (default Xenova/all-MiniLM-L6-v2)

Output Formats

context.json

Contains indexed_content pages with fields like:

• url
• title
• description
• content
• html_content

vector-context.json

Contains:

• metadata
• pages[]
• chunks[] with per-chunk embedding

Scripts

In packages/indexer/package.json:

• pnpm example
• pnpm example:vector
• pnpm index:vector
• pnpm test

Tutorial

See full step-by-step guide:

• packages/indexer/TUTORIAL.md

License

MIT