searchxng-lib

A lightweight, production-ready Node.js library for aggregating web search results from multiple engines — with built-in deduplication, ranking, caching, and URL content fetching.

Features

• Multi-engine search — Bing, DuckDuckGo, Brave Search API
• Works from any IP — no server/datacenter blocks on default engines
• Deduplication & ranking — results found by multiple engines score higher
• Parallel fetching — all engines run simultaneously with per-engine timeouts
• In-memory caching — configurable TTL to avoid redundant requests
• URL content fetching — parse text, title, and metadata from any page
• Zero-config — works out of the box with no API keys
• TypeScript — full type definitions included
• Dual exports — ESM (import) and CommonJS (require)
• No heavy dependencies — only cheerio for HTML parsing; uses native fetch

Requirements

Node.js >= 18.0.0 (uses native fetch and AbortSignal)

Installation

npm install searchxng-lib

Quick Start

import { search } from 'searchxng-lib';

const results = await search('nodejs best practices');
console.log(results.results);   // array of AggregatedResult
console.log(results.fromCache); // false (first call)

Engines

| Engine | Key needed | Works from server IP | Notes | |---|---|---|---| | bing | No | ✅ Yes | Default. Reliable HTML scraper. | | duckduckgo | No | ✅ Yes | Default. Uses vqd session token. | | brave-api | Yes | ✅ Yes | Most reliable. 2,000 free req/month. | | brave | No | ⚠️ No | Blocked by Cloudflare on cloud IPs. Local only. | | swisscows | No | ⚠️ No | Blocked on cloud IPs. Local only. |

API

search(query, options?)

const results = await search('typescript tutorial', {
  engines:    ['bing', 'duckduckgo'],  // default
  timeout:    8000,                     // ms per engine, default 8000
  maxResults: 20,                       // default 20
  cacheTTL:   600,                      // seconds, default 600
  apiKeys: {
    brave: 'YOUR_BRAVE_API_KEY',        // only needed for 'brave-api' engine
  },
});

Returns: Promise<SearchResponse>

interface SearchResponse {
  results:   AggregatedResult[];
  fromCache: boolean;
  query:     string;
  timestamp: number;
}

interface AggregatedResult {
  title:   string;
  url:     string;
  snippet: string;
  engines: string[];  // which engines returned this result
  score:   number;    // how many engines returned it (higher = more relevant)
}

fetchUrl(url, timeout?)

Fetch and parse a single URL into structured text.

const page = await fetchUrl('https://example.com');
console.log(page.title);       // page title
console.log(page.description); // meta description
console.log(page.text);        // plain text, first 5000 chars
console.log(page.content);     // raw HTML
console.log(page.statusCode);  // 200

fetchUrls(urls, timeout?)

Batch fetch multiple URLs in parallel. Failed URLs are silently skipped.

const pages = await fetchUrls([
  'https://example.com',
  'https://example.org',
]);
pages.forEach(p => console.log(p.title));

initialize(options?)

Set global defaults applied to every search() call.

import { initialize } from 'searchxng-lib';

initialize({
  engines:  ['bing', 'duckduckgo'],
  cacheTTL: 1800, // 30 minutes
});

clearCache() / getCacheSize()

import { clearCache, getCacheSize } from 'searchxng-lib';

console.log(getCacheSize()); // 3
clearCache();
console.log(getCacheSize()); // 0

Examples

Zero-config

import { search } from 'searchxng-lib';

const { results } = await search('what is typescript');
results.forEach(r => {
  console.log(`[score:${r.score}] ${r.title}`);
  console.log(`  ${r.url}`);
});

With Brave Search API (most reliable)

import { search } from 'searchxng-lib';

const results = await search('nodejs event loop', {
  engines: ['bing', 'duckduckgo', 'brave-api'],
  apiKeys: { brave: 'BSA_YOUR_KEY_HERE' },
  maxResults: 10,
});

Get a free Brave API key at https://api.search.brave.com

AI assistant integration

import { search, fetchUrls } from 'searchxng-lib';

async function searchAndFetch(query) {
  const { results } = await search(query, { maxResults: 5 });
  const pages = await fetchUrls(results.slice(0, 3).map(r => r.url));

  // Ready to pass to an LLM
  return results.map((result, i) => ({
    title:    result.title,
    url:      result.url,
    snippet:  result.snippet,
    score:    result.score,
    engines:  result.engines,
    fullText: pages[i]?.text ?? null,
  }));
}

const context = await searchAndFetch('nodejs best practices');
console.log(JSON.stringify(context, null, 2));

Caching

import { search, clearCache, getCacheSize } from 'searchxng-lib';

const r1 = await search('javascript');
console.log(r1.fromCache); // false

const r2 = await search('javascript');
console.log(r2.fromCache); // true — served from cache

clearCache();

CommonJS

const { search, fetchUrl } = require('searchxng-lib');

const results = await search('javascript');
console.log(results.results);

Result Scoring

Results are scored by how many engines return them:

| Score | Meaning | |---|---| | 1 | Found by one engine | | 2 | Found by two engines — higher relevance | | 3 | Found by all three engines — highest relevance |

Results are sorted by score descending automatically.

Error Handling

All engine errors are isolated — if one engine fails or times out, the rest continue.

try {
  const results = await search('query', { timeout: 5000 });
  console.log(`Found ${results.results.length} results`);
} catch (err) {
  console.error('Search failed:', err.message);
}

License

MIT