@avalw/search-worker
v2.3.6
Published
AVALW Distributed Cache Worker - Intelligent caching with size limits and auto-cleanup
Maintainers
avalw_officialReadme
@avalw/search-worker
Distributed Cache Worker for AVALW Search Network
Contribute your computer's cache storage to the AVALW search network and earn free API access in return.
How It Works
┌─────────────────────────────────────────────────────────────────────────┐
│ AVALW DISTRIBUTED CACHE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌──────────────┐ ┌─────────────┐ │
│ │ User │────────▶│ avalw.org │────────▶│ Coordinator │ │
│ │ Search │ │ Server │ │ Server │ │
│ └─────────┘ └──────────────┘ └──────┬──────┘ │
│ │ │
│ ┌─────────────────────────┼───────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────┐ │
│ │ Worker 1 │ │ Worker 2 │ │ ... │ │
│ │ Cache │ │ Cache │ │ │ │
│ └──────────┘ └──────────┘ └──────┘ │
│ │
│ YOUR WORKER stores search results and serves them when requested │
│ │
└─────────────────────────────────────────────────────────────────────────┘The Flow
1. USER SEARCHES 2. CHECK CACHE 3. SERVE RESULT
"spotify"
┌─────────────┐
┌──────┐ │ Coordinator │ ┌──────────┐
│ User │──search──▶ │ checks │──hit──▶ │ Worker │──result──▶ User
└──────┘ │ workers │ │ serves │
└─────────────┘ └──────────┘
│
miss │
▼
┌──────────┐
│ Server │──result──▶ User + Distribute to Workers
│ process │
└──────────┘What Your Worker Does vs. Doesn't Do
✅ DOES ❌ DOESN'T
───────────────────────────────── ─────────────────────────────────
• Store search RESULTS • Have the search INDEX
• Serve cached results • Process raw documents
• Use 50-200MB RAM • Use significant CPU
• Use 500MB-2GB disk • Access your files
• Connect via WebSocket • Run arbitrary code
• Earn you API access • Collect your dataQuick Start
# 1. Get your token at https://worker.avalw.org
# 2. Run the worker:
npx @avalw/search-worker --token YOUR_TOKENThat's it! Your worker will connect and start contributing.
Installation
# Global install
npm install -g @avalw/search-worker
# Or use npx (no install needed)
npx @avalw/search-worker --token YOUR_TOKENWhat's New in v2.1.0
┌────────────────────────────────────────────────────────────────┐
│ INTELLIGENT CACHE SYSTEM │
├────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Size Limits │ │ Time-Based │ │ Popularity │ │
│ │ 100MB - 2GB │ │ Eviction │ │ Scoring │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ • Cache up to 2GB of search results │
│ • Auto-remove queries unused for 7+ days │
│ • Popular queries stay cached longer │
│ • Background cleanup every hour │
│ │
└────────────────────────────────────────────────────────────────┘Command Line Options
# Basic usage (500MB cache, 7 day retention)
npx @avalw/search-worker --token abc123
# Custom cache size (100-2048 MB)
npx @avalw/search-worker --token abc123 --cache-size 1024
# Custom retention (1-30 days)
npx @avalw/search-worker --token abc123 --max-age 14
# Full customization
npx @avalw/search-worker \
--token abc123 \
--cache-size 1024 \
--max-age 14All Options
┌──────────────────┬─────────────────┬───────────────────────────────────┐
│ Option │ Default │ Description │
├──────────────────┼─────────────────┼───────────────────────────────────┤
│ -t, --token │ $AVALW_TOKEN │ Your worker authentication token │
│ -s, --cache-size │ 500 │ Max cache size in MB (100-2048) │
│ -a, --max-age │ 7 │ Days to keep unused queries (1-30)│
│ -h, --help │ │ Show help │
│ -v, --version │ │ Show version │
└──────────────────┴─────────────────┴───────────────────────────────────┘Environment Variables
export AVALW_WORKER_TOKEN=your-token-here
export AVALW_CACHE_SIZE_MB=1024
export AVALW_MAX_AGE_DAYS=14
npx @avalw/search-workerProgrammatic Usage
const { createWorker } = require('@avalw/search-worker');
const worker = createWorker({
workerToken: 'your-token-here',
// Cache configuration
cacheSizeMB: 1024, // 1GB cache
maxAgeDays: 14, // 14 day retention
// Event handlers
onConnected: (w) => console.log('Connected to coordinator'),
onDisconnected: (w) => console.log('Disconnected'),
onCacheHit: (key) => console.log('Served:', key),
onCacheStore: (key) => console.log('Cached:', key),
onError: (err) => console.error('Error:', err)
});
// Connect
worker.connect();
// Get stats
const stats = worker.getStats();
console.log(`Cache: ${stats.cache.sizeFormatted} / ${stats.cache.maxSizeFormatted}`);
console.log(`Hit rate: ${stats.hitRate}%`);
// Runtime configuration
worker.setCacheSize(2048); // Change to 2GB
worker.setCacheMaxAge(30); // Change to 30 days
// Cleanup
worker.disconnect();API Tiers & Rewards
Your contribution is measured in Core-Hours:
┌─────────────────────────────────────────────────────────────────────┐
│ CORE-HOURS FORMULA │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Core-Hours = Cores × (CPU Speed ÷ 3.0 GHz) × Usage% × Hours │
│ │
│ Example: 8 cores × (3.5 GHz ÷ 3.0 GHz) × 80% × 1 hour │
│ = 8 × 1.17 × 0.8 × 1 = 7.5 Core-Hours │
│ │
└─────────────────────────────────────────────────────────────────────┘Tier Benefits
┌─────────────┬────────────┬───────────┬─────────────────┐
│ Tier │ Core-Hours │ Min Cores │ API Requests │
├─────────────┼────────────┼───────────┼─────────────────┤
│ Starter │ 3 │ 3 │ 1,000/day │
│ Basic │ 12 │ 4 │ 5,000/day │
│ Pro │ 36 │ 6 │ 20,000/day │
│ Business │ 72 │ 8 │ 50,000/day │
│ Enterprise │ 240 │ 16 │ 500,000/day │
└─────────────┴────────────┴───────────┴─────────────────┘Intelligent Cache Details
How Eviction Works
┌─────────────────────────────────────────────────────────────┐
│ EVICTION PRIORITY │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. EXPIRED TTL │
│ └── Items past their time-to-live (5 min default) │
│ │
│ 2. MAX AGE EXCEEDED │
│ └── Items not accessed for N days (7 days default) │
│ │
│ 3. LOWEST POPULARITY │
│ └── Score = log2(accessCount) × recency × stability │
│ │
│ Priority: 1 > 2 > 3 (expired items removed first) │
│ │
└─────────────────────────────────────────────────────────────┘Popularity Scoring
Score = log2(accessCount + 1) × recencyFactor × stabilityFactor
┌─────────────────┬─────────────────────────────────────────┐
│ Factor │ Description │
├─────────────────┼─────────────────────────────────────────┤
│ accessCount │ Number of times the query was accessed │
│ recencyFactor │ 1.0 → 0.0 over 7 days since last access │
│ stabilityFactor │ Higher for consistent access patterns │
└─────────────────┴─────────────────────────────────────────┘
Example: Query accessed 100 times, last accessed 1 day ago
Score = log2(101) × 0.86 × 1.2 = 6.65 × 0.86 × 1.2 = 6.86Resource Usage
┌──────────────────────────────────────────────────────────┐
│ TYPICAL RESOURCE USAGE │
├──────────────────────────────────────────────────────────┤
│ │
│ CPU: 1-5% [▓░░░░░░░░░░░░░░░░░░░] minimal │
│ RAM: 50-200MB [▓▓▓▓░░░░░░░░░░░░░░░░] light │
│ Disk: 500MB-2GB [▓▓▓▓▓▓▓▓░░░░░░░░░░░░] configurable │
│ Network: Low [▓▓░░░░░░░░░░░░░░░░░░] WebSocket │
│ │
└──────────────────────────────────────────────────────────┘Privacy & Security
┌────────────────────────────────────────────────────────────────┐
│ PRIVACY GUARANTEES │
├────────────────────────────────────────────────────────────────┤
│ │
│ ✓ Workers cache RESULTS only, not the search index │
│ ✓ No IP addresses are logged │
│ ✓ Workers don't see who requested a search │
│ ✓ All communication encrypted (WSS/HTTPS) │
│ ✓ No access to your local files │
│ ✓ Open source - audit the code yourself │
│ │
└────────────────────────────────────────────────────────────────┘Running as a Service
Linux (systemd)
# Create service file
sudo nano /etc/systemd/system/avalw-worker.service[Unit]
Description=AVALW Search Worker
After=network.target
[Service]
Type=simple
User=nobody
Environment=AVALW_WORKER_TOKEN=your-token-here
Environment=AVALW_CACHE_SIZE_MB=1024
Environment=AVALW_MAX_AGE_DAYS=14
ExecStart=/usr/bin/npx @avalw/search-worker
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable avalw-worker
sudo systemctl start avalw-worker
# Check status
sudo systemctl status avalw-workerDocker
FROM node:18-alpine
RUN npm install -g @avalw/search-worker
ENV AVALW_WORKER_TOKEN=your-token-here
ENV AVALW_CACHE_SIZE_MB=1024
ENV AVALW_MAX_AGE_DAYS=14
CMD ["avalw-worker"]docker build -t avalw-worker .
docker run -d --name avalw avalw-workerPM2
pm2 start avalw-worker --name "avalw" -- --token YOUR_TOKEN --cache-size 1024
pm2 save
pm2 startupWindows (Task Scheduler)
@echo off
set AVALW_WORKER_TOKEN=your-token-here
set AVALW_CACHE_SIZE_MB=1024
npx @avalw/search-workerTroubleshooting
┌─────────────────────────────────────────────────────────────────┐
│ COMMON ISSUES │
├─────────────────────────────────────────────────────────────────┤
│ │
│ "Connection refused" │
│ └── Check internet connection, firewall settings │
│ │
│ "Invalid token" │
│ └── Get a new token at https://worker.avalw.org │
│ │
│ "Cache full, high eviction rate" │
│ └── Increase --cache-size or decrease --max-age │
│ │
│ "Memory usage too high" │
│ └── Decrease --cache-size (minimum 100MB) │
│ │
│ "Worker keeps disconnecting" │
│ └── Check network stability, worker will auto-reconnect │
│ │
└─────────────────────────────────────────────────────────────────┘System Requirements
┌────────────────┬────────────────────────────────────┐
│ Component │ Requirement │
├────────────────┼────────────────────────────────────┤
│ Node.js │ 14.0.0 or higher │
│ CPU │ 3+ cores for Starter tier │
│ RAM │ 512MB+ free │
│ Disk │ 500MB - 2GB for cache │
│ Network │ Stable internet, WebSocket support │
│ OS │ Windows, macOS, Linux │
└────────────────┴────────────────────────────────────┘Support
- Dashboard: worker.avalw.org
- Documentation: worker.avalw.org/protocol
- Email: [email protected]
- Search Engine: avalw.org
License
MIT License - Free to use, modify, and distribute.
Made with transparency in mind by AVALW
https://avalw.com