zayra-search-engine

A personal, AI-powered search engine module for ZAYRA AI. zayra-search-engine lets ZAYRA search, crawl, index, and retrieve information from the internet — a lightweight personal search system, similar to a mini browser/search engine — without depending on external AI search APIs.

This is version 0.0.1. It contains only reusable search engine logic — no user-specific backend, deployment code, database credentials, API keys, or application-specific business logic. Application-specific UI and backend stay outside this package.

Installation

npm install zayra-search-engine

Usage

import { SearchEngine } from "zayra-search-engine";

const search = new SearchEngine();

await search.crawlAndIndex("https://example.com");
const results = await search.search("latest AI news");

Search Engine Core

const search = new SearchEngine({
  fetch: myFetch,                // optional — defaults to the global fetch() (Node 18+)
  timeoutMs: 15000,
  allowPrivateHosts: false,        // keep false in production; see Security notes below
  indexFilePath: "./search-index.json", // optional — enables SearchIndex.save()/load() with no argument
  rankerOptions: { sourceQuality: { "trusted-news.com": 3 } },
});

await search.crawlAndIndex(seedUrl, crawlOptions?); // crawl + index a page (and, optionally, its links)
await search.indexPage(url);                          // index exactly one page, no link-following
const results = await search.search(query, { limit: 10, previewLength: 200 });

Each result: { url, title, description, preview, score, timestamp } — preview is a query-centered content snippet, ready for a browser-style UI (see Browser Preview Support). A query that matches nothing returns [] rather than throwing.

Web Crawler

import { Crawler } from "zayra-search-engine";

const crawler = new Crawler();
const page = await crawler.fetchPage(url); // { url, title, description, links, content, rawHtml, crawledAt }

const pages = await crawler.crawl(seedUrl, {
  maxDepth: 2,           // 1 = only the seed page; 2 = also pages it links to; etc.
  maxPages: 20,
  sameDomainOnly: true,   // only follow links on the seed's own hostname...
  allowedDomains: [],      // ...unless an explicit allow-list is given here
});

fetchPage() throws CrawlerError if it can't load a page. crawl()'s multi-page traversal instead skips a page it can't fetch and keeps going — one broken or blocked link shouldn't abort an entire crawl job.

Security: crawling is SSRF-safe by default

Every fetch — the seed URL, every followed link, every redirect hop — is checked before it happens:

• Only http/https protocols are allowed — javascript:, data:, file:, etc. are rejected outright.
• Local/private-network hosts are refused by default: localhost, loopback, RFC1918 private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), and link-local addresses (169.254.0.0/16, which includes the 169.254.169.254 cloud metadata endpoint). Override with allowPrivateHosts: true for local development only.
• Redirects are followed manually (not via fetch's automatic redirect-following) so a safe URL that 302s to an internal address is caught, not silently followed. Capped at 5 hops.

Page Reader

import { PageReader } from "zayra-search-engine";

const reader = new PageReader();
reader.clean(html);                  // strips nav/header/footer/aside/scripts/styles/forms, prefers a detected <main>/<article>
reader.read(html);                    // clean() + convert to plain, readable text
reader.hasDetectedMainContent(html);   // whether a <main>/<article> region was found

SearchEngine.crawlAndIndex() calls reader.read() automatically on every crawled page before indexing it.

Index System

import { SearchIndex } from "zayra-search-engine";

const index = new SearchIndex({ filePath: "./search-index.json" }); // optional default path for save()/load()

index.add({ url, title, description, content, keywords?, timestamp? }); // keywords auto-derived if omitted
index.get(url);
index.query("AI news"); // naive full-text/keyword candidate scan — Ranker does the actual scoring

await index.save();  // "Support: local storage first" — persists to a local JSON file
await index.load();

Future: vector database support (prepared, not implemented)

import { VectorIndexProvider } from "zayra-search-engine";

const vectorIndex = new VectorIndexProvider({ backend: myVectorDbClient }); // bring your own client
vectorIndex.isAvailable; // true once a real backend is injected
await vectorIndex.upsert(entry);
await vectorIndex.similaritySearch(queryEmbedding);

A real embedding-backed similarity search needs a vector database client, which this lightweight, zero-dependency package deliberately doesn't bundle. VectorIndexProvider just defines the shape a future backend plugs into.

Ranking System

import { Ranker } from "zayra-search-engine";

const ranker = new Ranker({
  weights: { keyword: 3, freshness: 1, sourceQuality: 1 },
  sourceQuality: { "trusted-news.com": 3 }, // hostname -> quality multiplier, default 1
  freshnessHalfLifeDays: 30,
});

ranker.rank(query, entries); // sorted best-first, each entry gets a `.score` breakdown

Four ranking factors, per spec: keyword match (title/keyword-list/content term matches, weighted by field), freshness (exponential decay by age), source quality (configurable per-hostname multiplier), and relevance — the combined, weighted total of the other three.

Search History

search.history.list();               // previous searches: [{ query, resultCount, timestamp }, ...]
search.history.list({ limit: 20 });    // most recent 20 only

search.cache.get(url);                   // a cached page, or null if missing/expired
search.cache.has(url);

PageCache (TTL-bounded, default 10 minutes) holds recently-fetched pages; SearchHistory (size-capped, default 200 entries) holds past queries and their result counts. Both are populated automatically by crawlAndIndex()/search().

Browser Preview Support

Every search result already carries what a browser-style UI needs — url, title, description, and preview (a query-centered content snippet from buildSnippet()). The frontend components below (ResultCard, BrowserView) render exactly this shape.

Frontend components (optional)

Three small, reusable, unopinionated React building blocks — not a complete application page, user dashboard, or authentication UI:

import { SearchBar } from "zayra-search-engine/frontend/components/SearchBar";
import { ResultCard } from "zayra-search-engine/frontend/components/ResultCard";
import { BrowserView } from "zayra-search-engine/frontend/components/BrowserView";

<SearchBar onSearch={(query) => search.search(query).then(setResults)} />

{results.map((result) => (
  <ResultCard key={result.url} result={result} onOpen={openInBrowserView} />
))}

<BrowserView page={openedPage} loading={isLoading} error={loadError} />

react is an optional peer dependency — importing from src/ (the engine) never pulls React into your bundle; only importing from frontend/components/ does. Each component accepts className/style for restyling and has no CSS framework dependency. BrowserView renders extracted text, not the original page's HTML/CSS/JS.

Testing the components

frontend/components/*.jsx aren't covered by npm test — that suite (test/search-engine.test.js) runs with Node's built-in test runner, which doesn't transform JSX, and this package ships with zero dependencies (no bundler/Babel). The three components were verified separately with esbuild + react-dom/server (syntax-checked and actually server-rendered across every prop state) before release; a consuming app's own build tooling (already set up for JSX) is the natural place to add component tests going forward.

Errors

• SearchError — base class; also thrown directly for an empty search() query
• CrawlerError — an invalid/unsafe URL (code: "UNSAFE_URL"), an unreachable page (code: "PAGE_UNAVAILABLE"), or too many redirects
• IndexError — an invalid entry, or a failed save()/load()

import { CrawlerError } from "zayra-search-engine";

try {
  await search.crawlAndIndex(url);
} catch (err) {
  if (err instanceof CrawlerError) {
    console.error(`${err.code}: ${err.message}`);
  }
}

Note on "empty results": a search matching nothing is a normal outcome, not an error — search() returns [].

Coding rules followed

Per spec: Node.js, async/await, modular architecture, clean reusable exports. No hardcoded websites, no API dependencies, no unnecessary frameworks (the HTML parsing and text utilities are hand-rolled, zero npm dependencies; react is an optional peer dependency for frontend/components/ only).

Project structure

zayra-search-engine/
  src/
    index.js         # public entry point
    engine.js           # SearchEngine — the main class
    crawler.js             # Crawler — fetch, HTML parsing, link-following, SSRF safety
    reader.js                 # PageReader — clean/readable text conversion
    indexer.js                   # SearchIndex + VectorIndexProvider — storage
    ranker.js                       # Ranker — keyword/freshness/source-quality/relevance scoring
    cache.js                           # PageCache + SearchHistory — Search History
    utils.js                              # tokenize, extractKeywords, buildSnippet, stripTagsToText
    errors.js                                # SearchError, CrawlerError, IndexError
  frontend/
    components/
      SearchBar.jsx
      ResultCard.jsx
      BrowserView.jsx
  package.json
  README.md

API reference

class SearchEngine

| Method | Description | | --- | --- | | new SearchEngine(options?) | options.fetch, options.timeoutMs, options.allowPrivateHosts, options.indexFilePath, options.rankerOptions, options.cacheOptions, options.historyOptions, plus lower-level crawler/reader/index/ranker/cache/history overrides. | | crawlAndIndex(seedUrl, crawlOptions?) | Crawl + index a page (and, per crawlOptions, its links). | | indexPage(url) | Index exactly one page. | | search(query, options?) | Returns ranked SearchResult[], [] if nothing matches. | | .crawler / .reader / .index / .ranker / .cache / .history | The component instances this engine is built from. |

Other exports

Crawler, assertSafeUrl(), PageReader, SearchIndex, VectorIndexProvider, Ranker, PageCache, SearchHistory, tokenize(), extractKeywords(), buildSnippet(), normalizeQuery(), decodeEntities(), stripTagsToText(), SearchError, CrawlerError, IndexError

Future compatibility

Designed to connect with zayra-core, zayra-memory, zayra-ai-router, zayra-tools, and zayra-sdk.

License

MIT