PODx - Ultimate X (Twitter) Scraper CLI

🚀 PODx is the ultimate X (Twitter) scraper and crypto analysis tool. Built with modern technologies and optimized for performance, PODx provides comprehensive Twitter data collection, advanced sentiment analysis, cryptocurrency signal generation, and powerful API capabilities.

✨ Features

🐦 Advanced Twitter Scraping

• Real-time scraping from any public Twitter account
• Batch processing with rate limiting and concurrency control
• Search functionality with advanced filters (date ranges, engagement metrics)
• Hashtag and ticker tracking for crypto-related content
• Reply and conversation analysis for deeper insights

💎 Cryptocurrency Analysis

• Token extraction from tweets with blockchain validation
• Sentiment analysis with AI-powered emotion detection
• Bot detection to filter out spam and manipulation
• Trading signal generation based on social sentiment
• Solana contract analysis for DeFi protocols

🔧 Powerful CLI Interface

• Interactive mode with guided workflows
• Batch processing for multiple accounts
• Progress tracking with real-time updates
• Flexible output formats (JSON, CSV, database)
• Configuration management for credentials and preferences

🌐 REST API Server

• JWT authentication with tiered access control
• Real-time endpoints for live data streaming
• Signal API for trading applications
• Token analytics with market correlation
• Rate limiting and request management

📊 Data Analysis & Reporting

• Comprehensive analytics with statistical insights
• Trend detection and pattern recognition
• Export capabilities to multiple formats
• Database integration with Convex
• Performance monitoring and metrics

📦 Installation

System Requirements

• Node.js: 20.0.0 or higher
• Bun: 1.0.0 or higher (recommended)
• Operating System: Linux, macOS, or Windows

Quick Install

# Install globally via npm
npm install -g podx

# Or install globally via bun (recommended)
bun add -g podx

# Or run directly with npx
npx podx --help

Local Development Setup

# Clone the repository
git clone https://github.com/podx/podx.git
cd podx

# Install dependencies
bun install

# Build all packages
bun run build

# Link for development
bun link

# Verify installation
podx --version

Docker Installation

# Pull the official image
docker pull podx/podx:latest

# Run with Docker
docker run -it podx/podx --help

# Run with mounted volume for data persistence
docker run -it -v $(pwd)/data:/app/data podx/podx scrape -u elonmusk

🚀 Quick Start

1. Initial Setup

# First run will guide you through setup
podx

# Or configure manually
podx config

2. Basic Scraping

# Scrape tweets from an account
podx scrape -u elonmusk -c 100

# Scrape multiple accounts
podx batch -u "vitalik,cz_binance,solanatracker"

# Search for crypto content
podx crypto-search -t "BTC,ETH,SOL" -c 200

3. Analysis

# Analyze scraped tweets
podx analyze -f scraped_data/elonmusk/tweets.json

# Generate trading signals
podx analyze -f crypto_tweets.json -o signals.json

# Account reputation analysis
podx accounts -f batch_results.json

4. Start API Server

# Start the REST API server
podx serve -p 3000

# Generate API token
podx token -u trader1 -t premium -d 7d

# Use the API
curl -H "Authorization: Bearer YOUR_TOKEN" \
     http://localhost:3000/api/v1/signals/top

📚 Usage Guide

Interactive Mode

PODx features an intuitive interactive mode for guided usage:

podx

This launches an interactive menu where you can:

• Configure credentials and preferences
• Choose scraping targets and parameters
• Select analysis options
• View results and export data
• Access help and documentation

Command Line Usage

Scraping Commands

# Single account scraping
podx scrape -u username -c 300

# Batch scraping
podx batch -u "user1,user2,user3" -c 200 --concurrent 3

# Search-based scraping
podx search -q "blockchain OR crypto" -c 500 --min-likes 10

# Hashtag tracking
podx hashtags -h "bitcoin,ethereum,defi" -c 300

# Ticker monitoring
podx tickers -t "BTC,ETH,SOL" -c 400

Analysis Commands

# Sentiment analysis
podx analyze -f tweets.json -o sentiment_analysis.json

# Bot detection
podx accounts -f tweets.json -o bot_analysis.json

# Reply analysis
podx replies -f tweets.json -t BTC -c 50

# Solana contract extraction
podx solana-contracts -f tweets.json

System Commands

# View system status
podx status

# Clean data and cache
podx clean --all

# Configuration management
podx config

# Generate API tokens
podx token -u user123 -t premium -d 7d

Advanced Usage

Custom Rate Limiting

# Respectful scraping with delays
podx scrape -u influencer --delay 5000

# High-volume batch processing
podx batch -f accounts.txt --concurrent 2 --delay 10000

Output Customization

# JSON output (default)
podx scrape -u user -c 100 -o results.json

# CSV export for analysis
podx scrape -u user -c 100 -o results.csv --format csv

# Database export
podx convex -f results.json -t tweets

Progress Monitoring

# Real-time progress tracking
podx scrape -u large_account -c 1000 --progress

# Batch progress with ETA
podx batch -f users.txt --progress --eta

🔐 Authentication & Security

Twitter/X Authentication

PODx requires Twitter credentials for scraping:

# Set environment variables
export XSERVE_USERNAME="your_twitter_username"
export XSERVE_PASSWORD="your_twitter_password"
export XSERVE_EMAIL="[email protected]"

# Or use the config command
podx config

API Security

# Generate JWT tokens
podx token -u trader -t premium -d 30d

# Use tokens for API access
curl -H "Authorization: Bearer YOUR_TOKEN" \
     http://localhost:3000/api/v1/signals/top

Data Privacy

• No data collection: PODx only scrapes public Twitter data
• Local storage: All data stored locally by default
• Optional cloud sync: Convex integration for data persistence
• Secure credentials: Encrypted credential storage

🌐 API Reference

REST API Endpoints

Health & Status

GET /api/v1/health
GET /api/v1/ready
GET /api/v1/status

Signals & Analysis

GET /api/v1/signals/top
GET /api/v1/signals?ticker=BTC
POST /api/v1/signals/analyze

Token Information

GET /api/v1/tokens/trending
GET /api/v1/tokens/{symbol}
GET /api/v1/tokens/{symbol}/signals

Authentication

POST /api/v1/auth/login
POST /api/v1/auth/token
GET /api/v1/auth/verify

API Examples

# Get top signals
curl -H "Authorization: Bearer TOKEN" \
     http://localhost:3000/api/v1/signals/top

# Analyze specific token
curl -H "Authorization: Bearer TOKEN" \
     -X POST \
     -d '{"token": "BTC", "timeframe": "24h"}' \
     http://localhost:3000/api/v1/signals/analyze

# Get trending tokens
curl http://localhost:3000/api/v1/tokens/trending?limit=10

📊 Data Analysis Features

Sentiment Analysis

PODx uses advanced AI to analyze tweet sentiment:

• Emotion Detection: Identifies joy, anger, fear, surprise
• Confidence Scoring: Provides confidence levels for analysis
• Trend Analysis: Tracks sentiment changes over time
• Context Awareness: Considers tweet context and engagement

Bot Detection

Advanced algorithms identify bot accounts:

• Behavioral Analysis: Examines posting patterns and engagement
• Content Analysis: Detects repetitive or spammy content
• Network Analysis: Identifies coordinated bot networks
• Machine Learning: Uses ML models for classification

Signal Generation

Generates trading signals based on social data:

• Sentiment Signals: Bullish/bearish sentiment indicators
• Volume Signals: High engagement and mention volume
• Momentum Signals: Rapid changes in social sentiment
• Risk Assessment: Provides confidence and risk levels

Token Extraction

Automatically identifies cryptocurrency mentions:

• Symbol Recognition: Detects ticker symbols ($BTC, $ETH)
• Address Validation: Validates blockchain addresses
• Contract Analysis: Analyzes smart contract interactions
• Multi-chain Support: Bitcoin, Ethereum, Solana, and more

💾 Data Storage & Export

Local Storage

# Default local storage
scraped_data/
├── elonmusk/
│   ├── tweets_2024-01-01T12-00-00.json
│   └── analysis_results.json
└── batch_results/
    ├── tweets_combined.json
    └── bot_analysis.json

Database Integration

# Save to Convex database
podx convex -f analysis.json -t signals

# Query stored data
podx convex -q "signals.where(confidence > 0.8)"

Export Formats

• JSON: Structured data with metadata
• CSV: Spreadsheet-compatible format
• Database: Direct database insertion
• API: Real-time data streaming

⚡ Performance & Optimization

Rate Limiting

PODx includes intelligent rate limiting:

// Automatic rate limiting based on tier
const rateLimits = {
  free: { requests: 100, window: 3600000 },      // 100/hour
  premium: { requests: 1000, window: 3600000 },  // 1000/hour
  enterprise: { requests: 10000, window: 3600000 } // 10000/hour
};

Concurrent Processing

# Optimize for your system
podx batch -f accounts.txt --concurrent 4 --delay 2000

# Auto-scale based on system resources
podx scrape -u large_account --auto-scale

Memory Management

• Streaming Processing: Handles large datasets without memory issues
• Chunked Analysis: Processes data in manageable chunks
• Garbage Collection: Automatic memory cleanup
• Resource Monitoring: Tracks memory and CPU usage

🔧 Configuration

Environment Variables

# Twitter Credentials
XSERVE_USERNAME=your_username
XSERVE_PASSWORD=your_password
[email protected]

# API Configuration
JWT_SECRET=your-secret-key
API_PORT=3000

# Database
CONVEX_URL=your_convex_url

# Scraping
DEFAULT_MAX_TWEETS=300
CONCURRENT_SCRAPES=3
RATE_LIMIT_DELAY=2000

# Storage
DATA_DIR=./scraped_data
ANALYSIS_DIR=./analysis

Configuration File

Create .podxrc in your project root:

{
  "scraping": {
    "maxTweets": 300,
    "concurrent": 3,
    "delay": 2000,
    "rateLimit": 30
  },
  "analysis": {
    "sentimentThreshold": 0.7,
    "botDetectionThreshold": 0.8,
    "tokenConfidenceThreshold": 0.6
  },
  "api": {
    "port": 3000,
    "cors": true,
    "rateLimit": {
      "windowMs": 900000,
      "max": 100
    }
  },
  "logging": {
    "level": "info",
    "format": "json",
    "outputs": ["console", "file"]
  }
}

🧪 Testing & Quality Assurance

Built-in Testing

# Run all tests
bun test

# Run specific test suites
bun test --scraping
bun test --analysis
bun test --api

# Run integration tests
bun test --integration

# Generate coverage report
bun test --coverage

Quality Gates

• Type Safety: 100% TypeScript coverage
• Linting: ESLint with strict rules
• Security: Automated security scanning
• Performance: Benchmark testing
• Compatibility: Cross-platform testing

🚀 Deployment

Production Server

# Build for production
bun run build

# Start production server
NODE_ENV=production podx serve -p 80

# Use process manager
pm2 start "podx serve -p 3000" --name podx-api

Docker Deployment

FROM oven/bun:latest

WORKDIR /app
COPY package.json bun.lockb ./
RUN bun install --production

COPY . .
RUN bun run build

EXPOSE 3000
CMD ["bun", "run", "start:prod"]

# Build and run
docker build -t podx .
docker run -p 3000:3000 podx

Cloud Deployment

Vercel

# Deploy API to Vercel
vercel --prod

# Deploy CLI as serverless functions
vercel functions deploy

Railway

# Deploy to Railway
railway up

Render

# Deploy to Render
render deploy

📈 Monitoring & Analytics

System Monitoring

# View system status
podx status

# Monitor performance
podx status --performance

# View logs
podx status --logs

# Check API health
curl http://localhost:3000/api/v1/health

Analytics Dashboard

PODx provides built-in analytics:

• Scraping Metrics: Success rates, speed, errors
• Analysis Results: Sentiment trends, signal accuracy
• API Usage: Request patterns, performance
• Data Quality: Bot detection rates, token validation

🤝 Contributing

Development Setup

# Fork and clone
git clone https://github.com/yourusername/podx.git
cd podx

# Install dependencies
bun install

# Set up development environment
cp .env.example .env
# Edit .env with your credentials

# Run in development mode
bun run dev

Code Standards

• TypeScript: Strict type checking enabled
• ESLint: Airbnb config with TypeScript support
• Prettier: Consistent code formatting
• Husky: Pre-commit hooks for quality gates

Testing

# Run tests
bun test

# Run tests in watch mode
bun test --watch

# Run tests with coverage
bun test --coverage

# Run integration tests
bun test --integration

Pull Request Process

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request
6. Wait for review and approval

📝 License

This project is licensed under the ISC License. See the LICENSE file for details.

🙏 Acknowledgments

• Bun: For the blazingly fast JavaScript runtime
• Commander.js: For the excellent CLI framework
• Hono: For the lightning-fast web framework
• TypeScript: For type safety and developer experience
• Convex: For the real-time database platform

📞 Support & Community

Getting Help

• 📧 Email: [email protected]
• 💬 Discord: PODx Community
• 📖 Documentation: docs.podx.dev
• 🐛 Issues: GitHub Issues
• 💡 Discussions: GitHub Discussions

Community

• 🌟 Star the repo if you find it useful
• 🔄 Contribute by submitting issues and pull requests
• 📢 Share PODx with your network
• 💝 Sponsor the project to support ongoing development

🚀 Roadmap

Version 2.1 (Coming Soon)

• [ ] Real-time WebSocket streaming
• [ ] Advanced ML models for sentiment analysis
• [ ] Multi-language support
• [ ] Plugin system for custom analyzers

Version 2.2 (Future)

• [ ] Mobile app companion
• [ ] Browser extension
• [ ] Advanced trading integrations
• [ ] Decentralized data storage

Built with ❤️ for the crypto and social media analysis community