@isdk/searxng-searcher

SearXNG is a powerful meta-search engine. This module provides not only standard SearXNG scraping but also robust Failover and Discovery mechanisms designed for the instability of public instances.

🚀 Resilient Configuration

Automatic Failover

SearXngSearcher supports multiple baseUrls. If the primary node times out or returns empty results, it automatically switches to the next healthy node in the list.

import { SearXngSearcher } from '@isdk/web-searcher';

const searcher = new SearXngSearcher({
  baseUrls: [
    'https://searx.be',
    'https://searxng.site',
    'https://search.mdosch.de'
  ]
});

// Failover happens automatically
const results = await searcher.search('AI tools');

🔍 Discovery Utilities

For unreliable public nodes, we provide built-in utilities to discover high-quality instances.

1. Discover from searx.space

Fetches data from searx.space and filters by uptime and response time.

import { fetchHqInstances } from '@isdk/web-searcher/utils';

const instances = await fetchHqInstances({
  minUptime: 0.9,      // Monthly uptime > 90%
  maxMedianTime: 1.5,  // Median response time < 1.5s
  engine: 'google'     // Node must support google engine
});

2. Discover from Official GitHub List

Fetches nodes from the SearXNG official instances repository.

import { fetchFromGithub } from '@isdk/web-searcher/utils';

const officialInstances = await fetchFromGithub();

3. Combining with Latency Testing

Combine discovery with the general latency testing tool to find the best nodes for your network.

import { fetchHqInstances, testInstancesByLatency } from '@isdk/web-searcher/utils';

async function getBestNodes() {
  const hq = await fetchHqInstances({ minUptime: 0.95 });
  // Test and sort by real latency
  const fast = await testInstancesByLatency(hq, { timeout: 3000, limit: 5 });
  return fast.map(i => i.url);
}

const searcher = new SearXngSearcher({ baseUrls: await getBestNodes() });

⚙️ Search Options (SearXNG Specific)

Beyond standard options, SearXNG supports:

• categories: Search category (e.g., general, images, news).
• timeRange: Time filter (day, week, month, year).

await searcher.search('Open Source', {
  categories: 'images',
  timeRange: 'week'
});

📡 JSON API Support

SearXNG exposes a native JSON API (format=json). When enabled by the instance operator, this endpoint returns structured JSON instead of HTML, avoiding the need for CSS-selector scraping.

Important: Many public instances disable the JSON API to prevent automated scraping. For reliable JSON access, consider self-hosting your own SearXNG instance.

Format Modes

| Mode | Description | |------|-------------| | 'auto' (default) | Tries JSON first. If it fails, falls back to HTML. Caches result per instance. | | 'json' | Forces JSON mode. Throws an error if the instance doesn't support it. | | 'html' | Skips JSON entirely. Uses traditional CSS-selector scraping. |

// Auto mode: best of both worlds
const results = await searcher.search('query', { format: 'auto' });

// Force JSON (requires a compatible instance)
const jsonResults = await searcher.search('query', { format: 'json' });

// Force HTML (skip JSON attempt)
const htmlResults = await searcher.search('query', { format: 'html' });

Extra Fields from JSON & HTML Responses

When the JSON API is used, each result item includes additional fields not previously available from HTML scraping:

const results = await searcher.search('query', { format: 'json' });

console.log(results[0].engine);     // 'google' — source search engine
console.log(results[0].score);      // 0.95 — relevance score
console.log(results[0].category);   // 'general' — result category

// Query-level metadata attached to every result:
console.log(results[0].suggestions); // ['related search terms', ...]
console.log(results[0].answers);     // ['direct answers', ...]
console.log(results[0].infoboxes);   // [{ title, content, ... }, ...]

HTML Mode Extraction (format='html')

The HTML scraping mode (format='html') also extracts suggestions, answers, and infoboxes from the page, matching the JSON API output shape. This means these fields are available regardless of format:

// HTML mode now returns the same metadata fields
const results = await searcher.search('France', { format: 'html' });

console.log(results[0].suggestions); // ['related search terms', ...] (if available)
console.log(results[0].answers);     // ['direct answers', ...] (if available)
console.log(results[0].infoboxes);   // [{ title: 'France', content: '...' }, ...] (if available)

The extraction is based on the actual SearXNG simple theme templates (source):

| Field | CSS Selector | Source Template | |-------|-------------|-----------------| | suggestions | #suggestions input.suggestion (extracts value attribute) | elements/suggestions.html | | answers | .answer (extracts innerText) | elements/answers.html + answer/legacy.html | | infoboxes | aside.infobox, .infobox (extracts title from h2.title, content from p, image from img) | elements/infobox.html |

Limitations on Public Instances

Not all SearXNG public instances serve these elements. Availability depends on the instance operator's configuration:

| Field | Availability on Public Instances | Reason | |-------|--------------------------------|--------| | suggestions | 🟡 Rare | Requires an autocomplete backend (google, duckduckgo, etc.) in settings.yml. Most privacy-focused instances disable this to avoid leaking queries to third-party APIs. See searx/autocomplete.py. | | answers | 🔴 Very rare | Requires built-in answerers (random.py, statistics.py) or external API keys to be enabled. Most instances disable these. See searx/answerers/. | | infoboxes | 🟢 Common | Provided by Wikipedia/Wikidata engines, which are enabled on most instances. Use entity queries (e.g. "France", "Python") to trigger. |

Test Coverage

• Unit tests (searxng.spec.ts): Full coverage for all three fields using fixtures based on actual SearXNG template source. These tests are reliable and do not depend on external instances.
• Real network tests (searxng.real.spec.ts): Only infobox extraction is tested against real instances (using query "France"). Suggestions and answers are intentionally excluded from real network tests because they are virtually never available on public instances.

Pre-populating the JSON Cache

You can probe instances ahead of time to avoid the JSON-then-HTML fallback cost. This is especially useful when using fetchHqInstances:

import { fetchHqInstances } from '@isdk/searxng-searcher';

const urls = await fetchHqInstances({
  data: searxspaceData,
  // Pre-populate the JSON-support cache automatically
  onInstances: async (urls) => SearXngSearcher.prefillJsonSupport(urls),
});

// Or do it manually:
import { SearXngSearcher } from '@isdk/searxng-searcher';
await SearXngSearcher.prefillJsonSupport(['https://my-instance.com']);

// Check cache results
console.log(SearXngSearcher.jsonSupportedInstances);
// Map { 'https://instance-a.com' => true, 'https://instance-b.com' => false }

How It Works

1. format='auto': The searcher first checks SearXngSearcher.jsonSupportedInstances.
   • true: Uses JSON API directly.
   • false: Skips to HTML scraping.
   • absent: Appends &format=json to the request URL and tries JSON.
2. If the JSON request succeeds, the instance is cached as true.
3. If it fails (403, 429, or invalid response), the instance is cached as false and the request automatically falls back to HTML scraping.

This means the first request to an untested instance incurs a small latency penalty (failed JSON attempt + HTML fallback). Subsequent requests use the cached result and skip straight to the correct mode.

JSON-Support API Reference

The JSON detection functions are available as static methods on SearXngSearcher, accessible from the main package import:

import { SearXngSearcher } from '@isdk/searxng-searcher';

// --- Probe a single instance ---
const supported = await SearXngSearcher.probeInstanceJsonSupport(
  'https://my-instance.com'
);
// → true | false | undefined

// --- Bulk probe (concurrent, allSettled) ---
await SearXngSearcher.prefillJsonSupport([
  'https://instance-a.com',
  'https://instance-b.com',
], 'http://proxy:8088');

// --- Read cached results ---
console.log(SearXngSearcher.jsonSupportedInstances.get('https://instance-a.com'));
// Map { 'https://instance-a.com' => true, 'https://instance-b.com' => false }

// --- Reload from pre-computed file ---
SearXngSearcher.loadJsonSupportedFromFile();

Two-stage probe logic:

1. Stage 1: Fetch /config and check search.formats for 'json'. Works for older SearXNG versions that expose format configuration.
2. Stage 2 (fallback): Fetch /search?q=ping&format=json. Status 200 with JSON body, or 429 (rate-limited), means JSON is supported. This handles newer SearXNG versions (≥2026.7) where /config no longer includes format info. If fetchWeb throws (e.g. 429 from crawler retries), the status code is extracted from the error message automatically.

On module load, the cache is auto-populated from searx-json-supported.json (if the file exists, generated by pnpm update-instances).

Stage 3 — HTML form sniffing (enhanced fallback): When Stage 2 returns 200 OK with HTML content (instead of JSON), the probe scans the page for indicators of JSON availability — such as <input name="format" value="json"> in a search form. This catches instances that only serve JSON via POST form submission (common in SearXNG ≥2026.7), ensuring they are still recognized as JSON-compatible.

Probe result caching

The probe results are cached in SearXngSearcher.jsonSupportedInstances (a static Map<string, boolean>). On module load, the cache is auto-populated from searx-json-supported.json (if the file exists). This file is generated and updated by pnpm update-instances, and is committed to the repository as a fallback data file — if the most recent run fails to reach searx.space or probe instances, the last known good list remains available.

Another committed fallback is searx-space-cache.json which stores the raw searx.space instance metadata. When searx.space is unreachable, the update script falls back to this cache and re-applies the filtering criteria (uptime, latency, engine support).

🔄 Instance Discovery — pnpm update-instances

The update-instances script (src/utils/update-searx-json.ts) aggregates SearXNG instance URLs from multiple sources, tests their connectivity, probes JSON API support, and generates the instance list files.

Multi-source merge

Sources are fetched independently and merged with deduplication:

| Source | Type | Description | |--------|------|-------------| | searx.space | JSON API | Primary source with uptime/latency metadata. Auto-cached to searx-space-cache.json. | | Local cache | File fallback | Used when searx.space is unreachable. Filtering still applies. | | Official GitHub | YAML | searxng/searx-instances — the authoritative instance registry. | | Farside | JSON | benbusby/farside — smart redirect gateway, health-checks every 5 min. | | TheSearXNG | Static list | Curated instances from thedoggybrad/thesearxng. |

Configuration

Extension sources are configured in src/utils/instance-sources.json:

{
  "yamlUrls": ["https://raw.githubusercontent.com/.../instances.yml"],
  "directUrls": ["https://instance-a.com", "..."],
  "farsideUrl": "https://raw.githubusercontent.com/benbusby/farside/master/services.json"
}

Usage

# Direct (uses env HTTP_PROXY if set)
pnpm update-instances

# Via proxy (required for most SearXNG instances from certain networks)
pnpm update-instances:proxy
# or: tsx src/utils/update-searx-json.ts --proxy http://127.0.0.1:8088

Output files

All generated files in src/utils/ are committed to the repository as fallback data:

| File | Contents | |------|----------| | searx.json | All available instances (connectivity-verified, latency-sorted) | | searx-json-supported.json | Subset with confirmed JSON API support | | searx-space-cache.json | Raw searx.space data (for offline reuse) |

📄 License

MIT