@the-situation/indexer

ETL indexer and REST/WebSocket API for The Situation prediction markets. Polls the Voyager explorer API for on-chain events, decodes them, stores everything in SQLite, and serves it over HTTP and WebSocket.

Quick Start

# From the repo root
cp packages/indexer/.env.example packages/indexer/.env
# Fill in VOYAGER_API_KEY and ADMIN_API_KEY

bun run --filter '@the-situation/indexer' dev

The server starts at http://localhost:3000. Register a market to begin indexing:

curl -X POST http://localhost:3000/admin/markets \
  -H "Authorization: Bearer <ADMIN_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"address": "0x02de...", "title": "My Market", "category": "crypto"}'

Events will begin appearing within ~15 seconds.

Environment Variables

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | VOYAGER_API_KEY | Yes | — | Voyager explorer API key | | ADMIN_API_KEY | Yes | — | Bearer token for /admin/* endpoints | | DB_PATH | No | ./data/indexer.db | SQLite database file path | | PORT | No | 3000 | HTTP server port | | STARKNET_RPC_URL | No | https://api.cartridge.gg/x/starknet/sepolia | Starknet RPC endpoint | | EVENT_POLL_INTERVAL_MS | No | 15000 | Event polling interval (ms) | | STATE_POLL_INTERVAL_MS | No | 30000 | Market state refresh interval (ms) | | POSITION_POLL_INTERVAL_MS | No | 60000 | Position refresh interval (ms) |

API Reference

Public Endpoints

GET /health

Returns service health and basic stats.

{
  "status": "ok",
  "uptime": 3600,
  "dbStatus": "connected",
  "marketsCount": 1,
  "eventsCount": 42,
  "lastPollAt": null
}

GET /api/markets

List all registered markets with their latest on-chain state.

GET /api/markets/:address

Get a single market by contract address. Returns 404 if not found.

GET /api/markets/:address/events

Paginated event time series for a market.

| Query Param | Default | Description | |-------------|---------|-------------| | page | 1 | Page number | | pageSize | 100 | Items per page (max 500) | | from | — | Unix timestamp lower bound | | to | — | Unix timestamp upper bound |

{
  "data": [
    {
      "id": 1,
      "txHash": "0x...",
      "blockNumber": 6290091,
      "timestamp": 1770487426,
      "eventType": "trade_executed",
      "trader": "0x...",
      "mean": 13,
      "stdDev": 3,
      "lowerBound": 10,
      "upperBound": 16,
      "oldMean": 12,
      "oldStdDev": 3,
      "collateralPosted": "91208000000000000"
    }
  ],
  "total": 42,
  "page": 1,
  "pageSize": 100
}

GET /api/markets/:address/traders

List all discovered traders and their positions for a market.

GET /api/positions/:trader

All positions held by a trader across markets.

GET /api/rankings

Global leaderboard sorted by PnL.

| Query Param | Default | Description | |-------------|---------|-------------| | limit | 50 | Max entries (max 200) |

WS /ws

WebSocket endpoint for real-time event streaming. Receives JSON messages when new events are indexed:

{
  "type": "new_events",
  "marketAddress": "0x...",
  "data": { "count": 3, "latestBlock": 6395140 }
}

Admin Endpoints

All admin endpoints require Authorization: Bearer <ADMIN_API_KEY>.

POST /admin/markets

Register a new market for indexing.

curl -X POST http://localhost:3000/admin/markets \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "address": "0x02de...",
    "title": "BTC Price Market",
    "description": "Predict the price of BTC",
    "category": "crypto",
    "topics": ["btc", "price"]
  }'

PUT /admin/markets/:address

Update market metadata (title, description, category, topics, is_active).

DELETE /admin/markets/:address

Soft-delete a market (stops indexing, keeps historical data).

ETL Pipeline

The indexer runs three polling loops:

| Loop | Interval | Description | |------|----------|-------------| | Event indexer | 15s | Fetches events from Voyager, decodes, stores in SQLite, broadcasts to WebSocket | | State refresher | 30s | Reads current market state from Starknet RPC | | Position refresher | 60s | Reads positions for discovered traders from Starknet RPC |

Supported event types:

| Event | Encoding | Description | |-------|----------|-------------| | TradeExecuted | SQ128x128 (4-limb) | Current-generation trade events | | MarketInitialized | SQ128x128 | Market bootstrap event | | TradeMoved | Q96 | Legacy trade events | | Bootstrapped | Q96 | Legacy bootstrap events |

Events are deduplicated by tx_hash (UNIQUE constraint). The indexer maintains a per-market cursor tracking the last indexed block number.

Building

# Type-check only
bun run --filter '@the-situation/indexer' typecheck

# Full build (compiles to dist/)
bun run --filter '@the-situation/indexer' build

# Run tests
bun run --filter '@the-situation/indexer' test

Deploying to Fly.io

The indexer is configured for Fly.io with a persistent volume for SQLite.

cd packages/indexer

# Create the app (first time only)
fly apps create situation-indexer

# Create a persistent volume for the database
fly volumes create indexer_data --region iad --size 1

# Set secrets
fly secrets set VOYAGER_API_KEY=your-key ADMIN_API_KEY=your-key

# Deploy (uses Dockerfile.indexer at repo root)
fly deploy

# Verify
curl https://situation-indexer.fly.dev/health

The fly.toml is pre-configured with:

• shared-cpu-1x / 512MB RAM
• Persistent volume mounted at /data
• auto_stop_machines = false (always-on for polling)
• HTTPS enforced

Useful Fly commands

# View logs
fly logs

# SSH into the machine
fly ssh console

# Restart the app
fly machines restart

# Check volume
fly volumes list