trusted-network-providers

A lightweight Node.js library for identifying IP addresses that belong to trusted network providers. Prevent your firewalls from blocking legitimate services like payment processors, search engine crawlers, and monitoring tools.

⚠️ v3.0.0: Requires Node.js >= 22.0.0. No API changes.

⚠️ v2.0.0 Breaking Changes: This version migrates from CommonJS to ES modules. Use import instead of require(). See the migration guide for details.

Quick Start

npm install @headwall/trusted-network-providers

import trustedProviders from '@headwall/trusted-network-providers';

// Load built-in providers (Googlebot, Stripe, PayPal, Cloudflare, etc.)
trustedProviders.loadDefaultProviders();

// Initialize dynamic providers
await trustedProviders.reloadAll();

// Check an IP address
const provider = trustedProviders.getTrustedProvider('66.249.66.87');
console.log(provider); // "Googlebot"

const unknown = trustedProviders.getTrustedProvider('123.123.123.123');
console.log(unknown); // null

Use Cases

• Firewall Management: Whitelist IPs for WordPress hosting networks
• Rate Limiting: Bypass rate limits for trusted crawlers
• Access Control: Allow trusted services through security layers
• Traffic Analysis: Identify and categorize legitimate traffic sources

Built-in Providers

Includes 27 trusted providers out of the box:

• Search Engines: Googlebot, Bingbot, Applebot, AhrefsBot, SemrushBot
• AI Crawlers: GPTBot, OAI-SearchBot (OpenAI) — with Applebot and ChatGPT-User, these carry the ai-crawler category and can be excluded as a set
• Advertising: Google AdsBot, AdSense (Google Special Crawlers)
• User-Triggered Fetchers: Google (Gmail image proxy, Chrome prefetch proxy, Feedfetcher, etc.), ChatGPT-User
• Payment Processors: Stripe, PayPal, Opayo
• Email Services: Outlook, Brevo, Mailgun, Google Workspace
• CDN/Infrastructure: Cloudflare, BunnyNet, Ezoic
• Development Tools: GetTerms, Labrika
• Social Media: FacebookBot
• E-commerce: ShipHero
• Networks: Private/Internal (RFC 1918), Google Services (public DNS)

Key Features

• ✅ Fast synchronous IP lookups (< 1ms typical with caching)
• ✅ Support for IPv4 and IPv6
• ✅ CIDR range matching
• ✅ Dynamic provider updates from external sources
• ✅ Provider lifecycle events (reload, error, stale detection)
• ✅ Configurable result caching with TTL
• ✅ Easy custom provider integration
• ✅ Zero configuration with sensible defaults

API

Core Functions

// Load default providers
loadDefaultProviders();

// Load them, leaving a whole category unregistered
loadDefaultProviders({ excludeCategories: ['ai-crawler'] });

// Update all providers with dynamic data
await reloadAll();

// Check if IP is trusted (returns provider name or null)
getTrustedProvider(ipAddress);

// Check if IP is trusted (returns boolean)
isTrusted(ipAddress);

// Provider management
addProvider(provider);
deleteProvider(providerName);
hasProvider(providerName);
getAllProviders();

// Category management
getProvidersByCategory(category); // the matching providers
deleteProvidersByCategory(category); // removes them, returns the names removed

// Testing
await runTests();

Lifecycle & Monitoring

// Get provider status and metadata
const status = getProviderStatus('Googlebot');
// Returns: { state, lastUpdated, lastError } — or null if no such provider
// State: 'ready' | 'loading' | 'error' | 'stale'

// Listen to provider lifecycle events
trustedProviders.on('reload:success', ({ provider, timestamp }) => {
  console.log(`${provider} reloaded at ${new Date(timestamp)}`);
});

trustedProviders.on('reload:error', ({ provider, error }) => {
  console.error(`${provider} failed to reload:`, error);
});

trustedProviders.on('stale', ({ provider, lastUpdated, staleDuration }) => {
  console.warn(`${provider} is stale (last updated ${staleDuration}ms ago)`);
});

// Configure staleness detection (default: 24 hours)
trustedProviders.setStalenessThreshold(12 * 60 * 60 * 1000); // 12 hours
const threshold = trustedProviders.getStalenessThreshold();

// Emit 'stale' for any provider past the threshold, on demand
trustedProviders.checkStaleness();

// once() fires a listener a single time; off() removes one
trustedProviders.once('reload:success', ({ provider }) => console.log(`first reload: ${provider}`));
trustedProviders.off('stale', myStaleHandler);

Caching & Performance

// Configure IP lookup result cache TTL (default: 1 hour)
trustedProviders.setResultCacheTTL(30 * 60 * 1000); // 30 minutes
const currentTtl = trustedProviders.getResultCacheTTL();

Logging

// Configure log level: 'silent' | 'error' | 'warn' | 'info' | 'debug'
trustedProviders.setLogLevel('info');
const level = trustedProviders.getLogLevel();

Provider Categories

A category groups providers a consumer may want to act on as a set. The one category defined today is ai-crawler, covering Applebot, GPTBot, OAI-SearchBot and ChatGPT-User.

It exists because trusted status is a strong privilege — consumers use it to bypass rate limits and blocklists — and whether an AI crawler has earned that is a judgement each consumer makes for itself. Filtering by category rather than by name means a crawler added to the category in a later release is picked up automatically, where a hand-maintained list of names would silently keep trusting it.

import trustedProviders, { PROVIDER_CATEGORY_AI_CRAWLER } from '@headwall/trusted-network-providers';

// Preferred: never register them in the first place, so no lookup can resolve
// against one in between.
trustedProviders.loadDefaultProviders({ excludeCategories: [PROVIDER_CATEGORY_AI_CRAWLER] });

// Or withdraw them after the fact — reports what it removed, so a change to
// which networks are trusted need never be silent.
const removed = trustedProviders.deleteProvidersByCategory(PROVIDER_CATEGORY_AI_CRAWLER);
console.error(`No longer trusting: ${removed.join(', ')}`);

// Inspect membership without changing anything.
trustedProviders.getProvidersByCategory(PROVIDER_CATEGORY_AI_CRAWLER);

Excluding a category does not mark those networks as bad — it only withdraws the privilege, so they are judged on their own behaviour like any other network.

Providers you register yourself may carry a category of your own; a provider with no category never matches a category filter.

Provider State Constants

The library exports constants for checking provider states programmatically:

import trustedProviders, {
  PROVIDER_STATE_READY,
  PROVIDER_STATE_LOADING,
  PROVIDER_STATE_ERROR,
  PROVIDER_STATE_STALE,
} from '@headwall/trusted-network-providers';

const status = trustedProviders.getProviderStatus('Stripe API');
if (status.state === PROVIDER_STATE_ERROR) {
  // Handle provider failure
}

CLI

A trusted-lookup command is included for ad-hoc IP checks from the terminal.

Run it without installing, via npx:

# Check a single IP
npx -p @headwall/trusted-network-providers trusted-lookup 66.249.66.87

# Check several at once
npx -p @headwall/trusted-network-providers trusted-lookup 66.249.66.87 1.2.3.4

Or install globally to get the bare command:

npm install -g @headwall/trusted-network-providers
trusted-lookup 66.249.90.77

It prints the matching provider for each IP (or "not trusted"), and exits 0 when every IP is trusted or 1 if any are untrusted — handy for scripts and CI:

if npx -p @headwall/trusted-network-providers trusted-lookup "$ip"; then
  echo "$ip is trusted"
fi

Documentation

• Providers - Built-in provider reference and custom provider guide
• Security - Security features and production recommendations
• DNS Security Guide - DNS/SPF provider security considerations
• Migration Guide - Upgrading from v1.x to v2.x
• Regular Maintenance - Refreshing the bundled IP assets

Example: Custom Provider

import trustedProviders from '@headwall/trusted-network-providers';

trustedProviders.addProvider({
  name: 'My Custom Network',
  testAddresses: ['10.0.0.1'],
  ipv4: {
    addresses: ['10.0.0.1'],
    ranges: ['10.0.0.0/24'],
  },
  ipv6: {
    addresses: [],
    ranges: [],
  },
});

Example: Express.js Middleware

import express from 'express';
import trustedProviders from '@headwall/trusted-network-providers';

const app = express();

app.use((req, res, next) => {
  const provider = trustedProviders.getTrustedProvider(req.ip);
  if (provider) {
    req.trustedProvider = provider;
    // Skip rate limiting, logging, etc.
  }
  next();
});

Example: Long-Running Service with Monitoring

import trustedProviders from '@headwall/trusted-network-providers';

// Configure logging for production
trustedProviders.setLogLevel('warn');

// Load providers
trustedProviders.loadDefaultProviders();
await trustedProviders.reloadAll();

// Monitor provider health
trustedProviders.on('stale', ({ provider, lastUpdated }) => {
  console.warn(`Provider ${provider} is stale, triggering reload...`);
  trustedProviders.reloadAll();
});

trustedProviders.on('reload:error', ({ provider, error }) => {
  // Alert your monitoring system
  console.error(`Critical: ${provider} reload failed:`, error.message);
});

// Check provider health periodically
setInterval(
  () => {
    const googlebot = trustedProviders.getProviderStatus('Googlebot');
    if (googlebot.state === 'error' || googlebot.state === 'stale') {
      console.warn(`Googlebot health check failed: ${googlebot.state}`);
    }
  },
  60 * 60 * 1000
); // Every hour

Maintenance

Update bundled IP assets before releases:

./scripts/update-assets.sh

License

MIT © Paul Faulkner

Contributing

Issues and pull requests welcome on GitHub.