@podx/cli

The CLI package provides a comprehensive command-line interface for PODx, offering both interactive and programmatic access to all scraping, analysis, and management features. Built with Commander.js and optimized for Bun runtime.

📦 Installation

# Install from workspace
bun add @podx/cli@workspace:*

# Or install from npm (when published)
bun add @podx/cli

🏗️ Architecture

The CLI package is organized into several key components:

packages/cli/src/
├── cli.ts              # Main CLI entry point and command registration
├── commands/           # Command definitions and registration
│   └── index.ts        # Command registration logic
├── handlers/           # Command handlers and business logic
├── ui/                 # User interface components
│   ├── banner.ts       # CLI banners and branding
│   └── interactive.ts  # Interactive mode interface
├── utils/              # CLI utilities and helpers
└── index.ts            # Main exports

🚀 Quick Start

Interactive Mode

# Launch interactive mode (default when no arguments provided)
podx

# Or explicitly
podx --interactive

Command Mode

# Scrape tweets from a user
podx scrape -u elonmusk -c 100

# Analyze crypto signals
podx analyze -f scraped_data/elonmusk/tweets.json

# Start API server
podx serve -p 3000

# Generate API token
podx token -u alice -t premium -d 7d

📊 Core Commands

Scraping Commands

podx scrape

Scrape tweets from a specific Twitter/X account.

podx scrape -u elonmusk -c 100

Options:

• -u, --username <username> - Target username (without @)
• -c, --count <number> - Number of tweets to scrape (default: 300)

Examples:

# Scrape 100 tweets from @elonmusk
podx scrape -u elonmusk -c 100

# Scrape default amount (300 tweets)
podx scrape -u cryptowhale

# Save to specific directory
podx scrape -u defipulse -c 500

podx batch

Scrape multiple accounts concurrently with rate limiting.

podx batch -u "elonmusk,pmarca,naval" -c 100

Options:

• -u, --usernames <usernames> - Comma-separated usernames
• -f, --file <file> - File containing usernames (one per line)
• -c, --count <number> - Tweets per account (default: 200)
• --concurrent <number> - Concurrent accounts (default: 3)
• --delay <seconds> - Delay between scrapes (default: 5)

Examples:

# Scrape multiple accounts
podx batch -u "elonmusk,vitalik,cz_binance" -c 150

# Use file with usernames
podx batch -f accounts.txt --concurrent 2 --delay 10

# High-volume batch scraping
podx batch -f influencers.txt -c 1000 --concurrent 5 --delay 15

Analysis Commands

podx analyze

Analyze scraped tweets for crypto signals and token mentions.

podx analyze -f scraped_data/elonmusk/tweets.json

Options:

• -f, --file <file> - Input tweets file (default: scraped_tweets.json)
• -o, --output <file> - Output analysis file (default: crypto_analysis.json)

Examples:

# Analyze specific file
podx analyze -f scraped_data/elonmusk/tweets.json -o elon_analysis.json

# Analyze batch results
podx analyze -f batch_results.json -o comprehensive_analysis.json

podx accounts

Analyze account reputations and detect bots/shillers.

podx accounts -f scraped_data/batch_results.json

Options:

• -f, --file <file> - Input tweets file (default: scraped_tweets.json)
• -o, --output <file> - Output analysis file (default: account_analysis.json)

podx replies

Scrape and analyze replies/comments for crypto signals.

podx replies -f crypto_tweets.json -t BTC -c 50

Options:

• -f, --file <file> - Input tweets file
• -t, --token <token> - Token to analyze (e.g., BTC, SOL)
• -o, --output <file> - Output file (default: replies_analysis.json)
• -c, --count <number> - Max replies per tweet (default: 20)

Search Commands

podx search

Search Twitter for specific queries with advanced filters.

podx search -q "AI artificial intelligence" -c 50 --min-likes 5

Options:

• -q, --query <query> - Search query (required)
• -c, --count <number> - Max tweets to fetch (default: 100)
• -m, --mode <mode> - Search mode (Latest|Top|People|Photos|Videos)
• -o, --output <file> - Output file (default: search_results.json)
• --min-likes <number> - Minimum likes filter
• --min-retweets <number> - Minimum retweets filter
• --date-from <date> - Start date (YYYY-MM-DD)
• --date-to <date> - End date (YYYY-MM-DD)
• --no-retweets - Exclude retweets

Examples:

# Search with engagement filters
podx search -q "blockchain OR crypto" -c 200 --min-likes 10 --min-retweets 5

# Search within date range
podx search -q "DeFi" --date-from 2024-01-01 --date-to 2024-01-31

# Search excluding retweets
podx search -q "NFT" --no-retweets -c 300

podx crypto-search

Search Twitter for crypto-related content with enhanced token detection.

podx crypto-search -t "BTC,ETH,SOL" -c 300

Options:

• -t, --tokens <tokens> - Comma-separated token symbols
• -k, --keywords <keywords> - Comma-separated crypto keywords
• -c, --count <number> - Max tweets to fetch (default: 200)
• -m, --mode <mode> - Search mode (Latest|Top)
• -o, --output <file> - Output file (default: crypto_search_results.json)
• --min-engagement <number> - Minimum likes for filtering (default: 0)

Examples:

# Search multiple tokens
podx crypto-search -t "BTC,ETH,SOL,ADA" -c 500

# Search with keywords
podx crypto-search -k "DeFi,NFT,memecoin" --min-engagement 10

# Combined token and keyword search
podx crypto-search -t "BTC,ETH" -k "bullish,bearish" -c 300

podx hashtags

Search Twitter for specific hashtags.

podx hashtags -h "bitcoin,ethereum,solana" -c 150

Options:

• -h, --hashtags <hashtags> - Comma-separated hashtags (required)
• -c, --count <number> - Max tweets to fetch (default: 100)
• -o, --output <file> - Output file (default: hashtag_search.json)

podx tickers

Search Twitter for specific ticker symbols ($BTC, $ETH, etc.).

podx tickers -t "BTC,ETH,DOGE" -c 200

Options:

• -t, --tickers <tickers> - Comma-separated tickers (required)
• -c, --count <number> - Max tweets to fetch (default: 100)
• -o, --output <file> - Output file (default: ticker_search.json)

Specialized Commands

podx solana-contracts

Analyze tweets for Solana contract addresses and smart contracts.

podx solana-contracts -f crypto_search.json

Options:

• -f, --file <file> - Input tweets file (default: scraped_tweets.json)
• -o, --output <file> - Output file (default: solana_contracts.json)

podx reply-search

Search replies for specific tweet IDs (experimental).

podx reply-search -i "1234567890,9876543210" -c 25

Options:

• -i, --ids <ids> - Comma-separated tweet IDs (required)
• -c, --count <number> - Max replies per tweet (default: 50)
• -o, --output <file> - Output file (default: replies_search.json)

podx list

Scrape tweets from a Twitter list (experimental).

podx list -i "12345678" -c 50

Options:

• -i, --id <listId> - Twitter list ID (required)
• -c, --count <number> - Max tweets to fetch (default: 100)
• -o, --output <file> - Output file (default: list_results.json)
• --no-retweets - Exclude retweets

System Commands

podx status

Show current system status, data, and configuration.

podx status

Displays:

• System information and configuration
• Available data files and sizes
• Database status
• API server status
• Recent scraping activity

podx clean

Clean build artifacts and optionally remove data.

podx clean --all

Options:

• --all - Remove all data including scraped tweets

podx config

Setup or modify Twitter credentials and configuration.

podx config

Interactive setup for:

• Twitter API credentials
• Database configuration
• Scraping preferences
• Analysis settings

API & Services

podx serve

Start the PODx REST API server.

podx serve -p 8080

Options:

• -p, --port <port> - Port to run the server (default: 3000)

podx token

Generate JWT tokens for PODX API access.

podx token -u alice -t premium -d 7d

Options:

• -u, --user <userId> - User ID (default: podx-user)
• -t, --tier <tier> - Access tier (free|premium|admin)
• -d, --duration <duration> - Token duration (1h|1d|7d|30d|unlimited)
• -n, --name <name> - Optional name/description

Examples:

# Generate free tier token
podx token -u user123 -t free -d 1h

# Generate premium token
podx token -u alice -t premium -d 7d -n "Alice's Premium Access"

# Generate admin token
podx token -u admin -t admin -d unlimited

podx convex

Save analysis data to Convex database.

podx convex -f crypto_analysis.json -t analysis

Options:

• -f, --file <file> - Input analysis file
• -t, --type <type> - Data type (analysis|tokens|accounts)

podx db

Start the PODx database service (Convex development).

podx db

podx all

Start all PODx services (API + Database + CLI).

podx all

🔧 Advanced Usage

Configuration Management

import { config } from '@podx/core';

// Load configuration
const appConfig = config.load();

// Access configuration values
console.log('Scraping target:', appConfig.scraper.targetUsername);
console.log('Max tweets:', appConfig.scraper.maxTweets);
console.log('Database URL:', appConfig.database.url);

// Modify configuration programmatically
const customConfig = {
  ...appConfig,
  scraper: {
    ...appConfig.scraper,
    maxTweets: 1000,
    rateLimit: 30
  }
};

Custom Command Creation

import { Command } from 'commander';
import { createCLI } from '@podx/cli';

// Create custom CLI instance
const program = createCLI();

// Add custom command
program
  .command('custom-scrape')
  .description('Custom scraping with advanced options')
  .option('-u, --username <username>', 'Target username')
  .option('-d, --depth <depth>', 'Scraping depth', '3')
  .action(async (options) => {
    console.log(`Custom scraping ${options.username} with depth ${options.depth}`);
    // Custom scraping logic here
  });

// Parse arguments
await program.parseAsync();

Interactive Mode Customization

import { runInteractiveMode } from '@podx/cli/handlers/interactive';

// Custom interactive mode
async function customInteractiveMode() {
  const inquirer = (await import('inquirer')).default;

  const answers = await inquirer.prompt([
    {
      type: 'list',
      name: 'action',
      message: 'What would you like to do?',
      choices: [
        'Scrape tweets',
        'Analyze data',
        'Generate reports',
        'Exit'
      ]
    }
  ]);

  switch (answers.action) {
    case 'Scrape tweets':
      // Handle scraping
      break;
    case 'Analyze data':
      // Handle analysis
      break;
    case 'Generate reports':
      // Handle reports
      break;
    case 'Exit':
      process.exit(0);
  }
}

Progress Callbacks

import { ScraperService } from '@podx/scraper';

// Scrape with custom progress reporting
const scraper = new ScraperService();

await scraper.scrapeAccount({
  targetUsername: 'cryptowhale',
  maxTweets: 1000,
  progressCallback: (progress) => {
    const percent = Math.round((progress.count / progress.max) * 100);
    const eta = estimateTimeRemaining(progress);

    console.log(`[${new Date().toISOString()}] Scraped ${progress.count}/${progress.max} tweets (${percent}%) - ETA: ${eta}`);
  }
});

function estimateTimeRemaining(progress: { count: number; max: number }): string {
  const elapsed = Date.now() - startTime;
  const rate = progress.count / elapsed;
  const remaining = (progress.max - progress.count) / rate;
  return `${Math.round(remaining / 1000)}s`;
}

Batch Processing

import { ScraperService } from '@podx/scraper';

const scraper = new ScraperService();
const accounts = ['elonmusk', 'vitalik', 'cz_binance', 'solanatracker'];

console.log('🚀 Starting batch scraping...');

// Process accounts with concurrency control
const results = [];
const concurrencyLimit = 3;
const delayBetweenBatches = 5000;

for (let i = 0; i < accounts.length; i += concurrencyLimit) {
  const batch = accounts.slice(i, i + concurrencyLimit);

  console.log(`📦 Processing batch ${Math.floor(i / concurrencyLimit) + 1}: ${batch.join(', ')}`);

  const batchPromises = batch.map(async (username) => {
    try {
      const tweets = await scraper.scrapeAccount({
        targetUsername: username,
        maxTweets: 200
      });

      console.log(`✅ Completed ${username}: ${tweets.length} tweets`);
      return { username, tweets, success: true };
    } catch (error) {
      console.error(`❌ Failed ${username}:`, error.message);
      return { username, error: error.message, success: false };
    }
  });

  const batchResults = await Promise.all(batchPromises);
  results.push(...batchResults);

  // Delay between batches to respect rate limits
  if (i + concurrencyLimit < accounts.length) {
    console.log(`⏳ Waiting ${delayBetweenBatches / 1000}s before next batch...`);
    await new Promise(resolve => setTimeout(resolve, delayBetweenBatches));
  }
}

console.log('🎉 Batch scraping completed!');
console.log(`📊 Results: ${results.filter(r => r.success).length}/${results.length} successful`);

Custom Analysis Pipelines

import { SentimentAnalyzer, TokenExtractor, BotDetector } from '@podx/scraper/analyzers';

class CustomAnalysisPipeline {
  constructor(
    private sentiment = new SentimentAnalyzer(),
    private tokens = new TokenExtractor(),
    private bots = new BotDetector()
  ) {}

  async analyze(tweets: Tweet[]): Promise<AnalysisResult> {
    console.log('🔍 Starting custom analysis pipeline...');

    // Step 1: Bot detection
    console.log('🤖 Detecting bots...');
    const botAnalysis = await this.bots.analyze(tweets);
    const humanTweets = botAnalysis.results
      .filter(r => r.botProbability < 50)
      .map(r => r.tweet);

    // Step 2: Sentiment analysis
    console.log('😊 Analyzing sentiment...');
    const sentimentResults = await this.sentiment.analyze(humanTweets);

    // Step 3: Token extraction
    console.log('💎 Extracting tokens...');
    const tokenResults = await this.tokens.extract(humanTweets);

    // Step 4: Generate insights
    console.log('📊 Generating insights...');
    const insights = this.generateInsights(sentimentResults, tokenResults);

    return {
      totalTweets: tweets.length,
      humanTweets: humanTweets.length,
      sentiment: sentimentResults,
      tokens: tokenResults,
      insights
    };
  }

  private generateInsights(sentiment: SentimentResult[], tokens: TokenResult[]): Insight[] {
    // Custom insight generation logic
    return [];
  }
}

// Use the pipeline
const pipeline = new CustomAnalysisPipeline();
const result = await pipeline.analyze(tweets);

🔧 Configuration

Environment Variables

# Twitter/X Credentials (Required for scraping)
XSERVE_USERNAME=your_twitter_username
XSERVE_PASSWORD=your_twitter_password
[email protected]

# Database Configuration
DATABASE_URL=postgresql://user:pass@localhost:5432/podx

# API Configuration
JWT_SECRET=your-jwt-secret-key-here
API_PORT=3000

# Scraping Configuration
DEFAULT_MAX_TWEETS=300
SCRAPE_RATE_LIMIT=30
CONCURRENT_SCRAPES=3

# File Storage
DATA_DIR=./scraped_data
ANALYSIS_DIR=./analysis_results

CLI Configuration File

Create a .podxrc file in your project root:

{
  "defaultMaxTweets": 300,
  "concurrentScrapes": 3,
  "rateLimitDelay": 2000,
  "outputFormat": "json",
  "enableProgressBars": true,
  "saveIntermediateResults": true,
  "analysis": {
    "sentimentThreshold": 0.7,
    "botDetectionThreshold": 0.8,
    "tokenConfidenceThreshold": 0.6
  }
}

📊 Output Formats

JSON Output Structure

{
  "metadata": {
    "version": "2.0.0",
    "timestamp": "2024-01-01T12:00:00.000Z",
    "command": "scrape",
    "parameters": {
      "username": "elonmusk",
      "maxTweets": 100
    }
  },
  "data": {
    "tweets": [
      {
        "id": "1234567890123456789",
        "username": "elonmusk",
        "text": "Mars colony update 🚀",
        "createdAt": "2024-01-01T12:00:00.000Z",
        "likes": 125000,
        "retweets": 25000,
        "replies": 5000,
        "hashtags": ["Mars", "SpaceX"],
        "mentions": ["SpaceX"],
        "urls": ["https://spacex.com/mars"],
        "media": [
          {
            "type": "photo",
            "url": "https://pbs.twimg.com/media/...",
            "alt": "Mars habitat rendering"
          }
        ]
      }
    ]
  },
  "analysis": {
    "sentiment": {
      "overall": "positive",
      "confidence": 0.85,
      "distribution": {
        "positive": 0.7,
        "neutral": 0.2,
        "negative": 0.1
      }
    },
    "tokens": [
      {
        "symbol": "BTC",
        "address": "1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2",
        "blockchain": "bitcoin",
        "mentions": 5,
        "sentiment": "bullish",
        "confidence": 0.9
      }
    ]
  }
}

Analysis Output Structure

{
  "summary": {
    "totalTweets": 1000,
    "analyzedTweets": 950,
    "humanTweets": 850,
    "botTweets": 100,
    "timeRange": {
      "from": "2024-01-01T00:00:00.000Z",
      "to": "2024-01-01T23:59:59.000Z"
    }
  },
  "sentimentAnalysis": {
    "overallSentiment": "bullish",
    "averageConfidence": 0.78,
    "sentimentDistribution": {
      "bullish": 0.65,
      "bearish": 0.25,
      "neutral": 0.10
    },
    "topPositiveTopics": ["Bitcoin", "Ethereum", "DeFi"],
    "topNegativeTopics": ["regulation", "volatility"]
  },
  "tokenAnalysis": {
    "mentionedTokens": 25,
    "topTokens": [
      {
        "symbol": "BTC",
        "mentions": 150,
        "averageSentiment": 0.75,
        "priceCorrelation": 0.85
      }
    ],
    "emergingTokens": [
      {
        "symbol": "NEW",
        "mentions": 50,
        "growth": 300,
        "sentiment": "very_positive"
      }
    ]
  },
  "signalAnalysis": {
    "generatedSignals": 15,
    "signalTypes": {
      "BUY": 8,
      "SELL": 4,
      "HOLD": 3
    },
    "averageConfidence": 0.82,
    "topSignals": [
      {
        "token": "BTC",
        "type": "BUY",
        "strength": 9,
        "confidence": 0.95,
        "reason": "Strong bullish sentiment with institutional accumulation"
      }
    ]
  }
}

🧪 Testing

Unit Tests

import { describe, test, expect, mock } from 'bun:test';
import { ScraperService } from '@podx/scraper';

describe('CLI Commands', () => {
  test('should handle scrape command', async () => {
    // Mock the scraper service
    mock.module('@podx/scraper', () => ({
      ScraperService: class {
        async scrapeAccount() {
          return [
            { id: '1', username: 'test', text: 'test tweet' }
          ];
        }
      }
    }));

    // Test CLI command execution
    const { runScraping } = await import('./handlers/scraping.js');
    const result = await runScraping({
      targetUsername: 'testuser',
      maxTweets: 10
    });

    expect(result).toBeDefined();
  });

  test('should validate command arguments', () => {
    // Test argument validation
    const invalidArgs = { username: '', maxTweets: -1 };

    expect(() => {
      validateScrapeArgs(invalidArgs);
    }).toThrow('Invalid arguments');
  });
});

Integration Tests

import { describe, test, expect } from 'bun:test';
import { createCLI } from '@podx/cli';

describe('CLI Integration', () => {
  test('should execute scrape command end-to-end', async () => {
    const cli = createCLI();

    // Mock process.argv for testing
    const originalArgv = process.argv;
    process.argv = ['node', 'podx', 'scrape', '-u', 'testuser', '-c', '5'];

    let output = '';
    const originalLog = console.log;
    console.log = (msg) => { output += msg + '\n'; };

    try {
      await cli.parseAsync();
      expect(output).toContain('Successfully scraped');
    } finally {
      console.log = originalLog;
      process.argv = originalArgv;
    }
  });

  test('should handle errors gracefully', async () => {
    const cli = createCLI();

    const originalArgv = process.argv;
    process.argv = ['node', 'podx', 'scrape', '-u', 'nonexistent'];

    let errorOutput = '';
    const originalError = console.error;
    console.error = (msg) => { errorOutput += msg + '\n'; };

    try {
      await cli.parseAsync();
      expect(errorOutput).toContain('Failed to scrape');
    } finally {
      console.error = originalError;
      process.argv = originalArgv;
    }
  });
});

E2E Tests

import { describe, test, expect } from 'bun:test';
import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

describe('CLI E2E', () => {
  test('should display help information', async () => {
    const { stdout } = await execAsync('podx --help');

    expect(stdout).toContain('PODx');
    expect(stdout).toContain('scrape');
    expect(stdout).toContain('analyze');
    expect(stdout).toContain('serve');
  });

  test('should show version', async () => {
    const { stdout } = await execAsync('podx --version');

    expect(stdout).toMatch(/\d+\.\d+\.\d+/);
  });

  test('should handle invalid commands', async () => {
    try {
      await execAsync('podx invalid-command');
      expect(true).toBe(false); // Should not reach here
    } catch (error) {
      expect(error.code).toBe(1);
      expect(error.stderr).toContain('error');
    }
  });
});

📊 Performance Optimization

Rate Limiting

// Configure rate limiting for different tiers
const rateLimits = {
  free: { requests: 100, window: 3600000 },      // 100/hour
  premium: { requests: 1000, window: 3600000 },  // 1000/hour
  enterprise: { requests: 10000, window: 3600000 } // 10000/hour
};

// Implement rate limiting in commands
async function executeWithRateLimit<T>(
  operation: () => Promise<T>,
  tier: keyof typeof rateLimits
): Promise<T> {
  const limit = rateLimits[tier];

  // Check rate limit
  const canProceed = await checkRateLimit(tier, limit);
  if (!canProceed) {
    throw new Error(`Rate limit exceeded for ${tier} tier`);
  }

  return await operation();
}

Memory Management

// Process large datasets in chunks
async function processLargeDataset(filePath: string, chunkSize = 1000) {
  const fileStream = Bun.file(filePath).stream();
  const reader = fileStream.getReader();

  let buffer: Tweet[] = [];
  let processed = 0;

  try {
    while (true) {
      const { done, value } = await reader.read();

      if (done) break;

      // Parse chunk
      const chunk = JSON.parse(new TextDecoder().decode(value));
      buffer.push(...chunk);

      // Process when buffer is full
      if (buffer.length >= chunkSize) {
        await processChunk(buffer.splice(0));
        processed += chunkSize;

        // Force garbage collection hint
        if (global.gc) global.gc();
      }
    }

    // Process remaining items
    if (buffer.length > 0) {
      await processChunk(buffer);
      processed += buffer.length;
    }

    console.log(`Processed ${processed} items`);
  } finally {
    reader.releaseLock();
  }
}

Concurrent Processing

// Optimize concurrent operations
async function scrapeMultipleAccounts(
  usernames: string[],
  options: { concurrency?: number; delay?: number } = {}
) {
  const { concurrency = 3, delay = 1000 } = options;
  const results: ScrapingResult[] = [];

  // Process in batches
  for (let i = 0; i < usernames.length; i += concurrency) {
    const batch = usernames.slice(i, i + concurrency);

    console.log(`Processing batch ${Math.floor(i / concurrency) + 1}/${Math.ceil(usernames.length / concurrency)}`);

    const batchPromises = batch.map(username =>
      scrapeWithRetry(username, options)
    );

    const batchResults = await Promise.allSettled(batchPromises);

    // Handle results
    batchResults.forEach((result, index) => {
      if (result.status === 'fulfilled') {
        results.push(result.value);
      } else {
        console.error(`Failed to scrape ${batch[index]}:`, result.reason);
      }
    });

    // Rate limiting delay
    if (i + concurrency < usernames.length) {
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }

  return results;
}

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

📝 License

This package is licensed under the ISC License. See the LICENSE file for details.

🔗 Related Packages

• @podx/core - Core utilities and types
• @podx/scraper - Twitter scraping functionality
• @podx/api - REST API server
• podx - Main CLI application

📞 Support

For support and questions:

• 📧 Email: [email protected]
• 💬 Discord: PODx Community
• 📖 Documentation: docs.podx.dev/cli
• 🐛 Issues: GitHub Issues