@the-situation/indexer

ETL indexer and REST/WebSocket API for The Situation prediction markets. Polls the Voyager explorer API for on-chain events, decodes them, stores everything in SQLite, and serves it over HTTP and WebSocket.

Quick Start

# From the repo root
cp packages/indexer/.env.example packages/indexer/.env
# Fill in VOYAGER_API_KEY and ADMIN_API_KEY

bun run --filter '@the-situation/indexer' dev

The server starts at http://localhost:3000. Register a market to begin indexing:

curl -X POST http://localhost:3000/admin/markets \
  -H "Authorization: Bearer <ADMIN_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"address": "0x02de...", "title": "My Market", "category": "crypto"}'

Events will begin appearing within ~15 seconds.

Environment Variables

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | VOYAGER_API_KEY | Yes* | — | Single Voyager explorer API key | | VOYAGER_API_KEYS | Yes* | — | Comma/whitespace-separated Voyager API key pool for redundancy | | ADMIN_API_KEY | Yes | — | Bearer token for /admin/* endpoints | | DB_PATH | No | ./data/indexer.db | SQLite database file path | | PORT | No | 3000 | HTTP server port | | STARKNET_NETWORK | No | sepolia | Network default set (sepolia or mainnet) | | STARKNET_RPC_URL | No | network default | Starknet RPC endpoint | | VOYAGER_API_BASE_URL | No | network default | Voyager API base URL | | EVENT_POLL_INTERVAL_MS | No | 15000 | Event polling interval (ms) | | STATE_POLL_INTERVAL_MS | No | 30000 | Market state refresh interval (ms) | | POSITION_POLL_INTERVAL_MS | No | 60000 | Position refresh interval (ms) |

VOYAGER_API_KEY or VOYAGER_API_KEYS must be set. When multiple keys are configured, the indexer rotates across them. A key that receives a Voyager 429/503 response is cooled down independently, and requests continue through the remaining keys until every key is cooling down.

API Reference

Public Endpoints

GET /health

Returns service health and basic stats.

{
  "status": "ok",
  "uptime": 3600,
  "dbStatus": "connected",
  "marketsCount": 1,
  "eventsCount": 42,
  "lastPollAt": null
}

GET /api/markets

List all registered markets with their latest on-chain state.

GET /api/markets/:address

Get a single market by contract address. Returns 404 if not found.

GET /api/markets/:address/events

Paginated event time series for a market.

| Query Param | Default | Description | |-------------|---------|-------------| | page | 1 | Page number | | pageSize | 100 | Items per page (max 500) | | from | — | Unix timestamp lower bound | | to | — | Unix timestamp upper bound |

{
  "data": [
    {
      "id": 1,
      "txHash": "0x...",
      "blockNumber": 6290091,
      "timestamp": 1770487426,
      "eventType": "trade_executed",
      "trader": "0x...",
      "mean": 13,
      "stdDev": 3,
      "lowerBound": 10,
      "upperBound": 16,
      "oldMean": 12,
      "oldStdDev": 3,
      "collateralPosted": "91208000000000000"
    }
  ],
  "total": 42,
  "page": 1,
  "pageSize": 100
}

GET /api/markets/:address/traders

List all discovered traders and their positions for a market.

GET /api/positions/:trader

All positions held by a trader across markets.

GET /api/rankings

Global leaderboard sorted by PnL.

| Query Param | Default | Description | |-------------|---------|-------------| | limit | 50 | Max entries (max 200) |

WS /ws

WebSocket endpoint for real-time event streaming. Receives JSON messages when new events are indexed:

{
  "type": "new_events",
  "marketAddress": "0x...",
  "data": { "count": 3, "latestBlock": 6395140 }
}

Admin Endpoints

All admin endpoints require Authorization: Bearer <ADMIN_API_KEY>.

POST /admin/markets

Register a new market for indexing.

curl -X POST http://localhost:3000/admin/markets \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "address": "0x02de...",
    "title": "BTC Price Market",
    "description": "Predict the price of BTC",
    "category": "crypto",
    "topics": ["btc", "price"]
  }'

PUT /admin/markets/:address

Update market metadata (title, description, category, topics, is_active).

DELETE /admin/markets/:address

Soft-delete a market (stops indexing, keeps historical data).

ETL Pipeline

The indexer runs three polling loops:

| Loop | Interval | Description | |------|----------|-------------| | Event indexer | 15s | Fetches events from Voyager, decodes, stores in SQLite, broadcasts to WebSocket | | State refresher | 30s | Reads current market state from Starknet RPC | | Position refresher | 60s | Reads positions for discovered traders from Starknet RPC |

Supported event types:

| Event | Encoding | Description | |-------|----------|-------------| | TradeExecuted | SQ128x128 (4-limb) | Current-generation trade events | | MarketInitialized | SQ128x128 | Market bootstrap event | | MarketSettled | market-type specific | Settlement event (continuous + multinoulli) | | MarketSettledMulti | multinoulli only | Multi-outcome settlement event | | TradeMoved | Q96 | Legacy trade events | | Bootstrapped | Q96 | Legacy bootstrap events |

Supported market types for indexing: normal, lognormal, bivariate, multinoulli.

Events are deduplicated by a stable per-event identity (event_uid) rather than transaction hash, so multiple events from the same transaction are indexed correctly. The indexer maintains a per-market cursor tracking the last indexed block number.

Building

# Type-check only
bun run --filter '@the-situation/indexer' typecheck

# Full build (compiles to dist/)
bun run --filter '@the-situation/indexer' build

# Run tests
bun run --filter '@the-situation/indexer' test

# Run opt-in performance suite (prints throughput/latency/memory metrics)
bun run --filter '@the-situation/indexer' test:perf

# Run strict perf checks (enables hard memory-trend assertions)
bun run --filter '@the-situation/indexer' test:perf:strict

Performance Suite Notes

The perf suite is intentionally opt-in (RUN_PERF=1) and uses relative regression guards to reduce machine-specific flakes:

• Ingestion throughput: compares second-half median throughput vs first-half.
• Query latency: checks tail amplification (p95 vs p50) under mixed API load.
• Memory trend: compares retained heapUsed/rss medians across sustained rounds.

In strict mode (PERF_STRICT=1), the suite requires GC availability and enforces hard memory trend thresholds suitable for dedicated performance runners.

Tests print the measured summaries (min, p50, p95, p99, max, mean) to help interpret changes across commits.

Deploying to Fly.io

The indexer is configured for Fly.io with a persistent volume for SQLite.

cd packages/indexer

# Create the Sepolia app (first time only)
fly apps create situation-indexer

# Create a persistent volume for the database
fly volumes create indexer_data --region iad --size 1

# Set secrets
fly secrets set VOYAGER_API_KEYS=key-1,key-2,key-3 ADMIN_API_KEY=your-key

# Deploy (uses Dockerfile.indexer at repo root)
fly deploy

# Verify
curl https://situation-indexer.fly.dev/health

Mainnet Deployment

Mainnet uses a separate Fly app and volume so the production database starts fresh and cannot mix Sepolia cursor state with mainnet events.

cd packages/indexer

# Create the mainnet app (first time only)
fly apps create situation-indexer-mainnet

# Create the mainnet database volume
fly volumes create indexer_mainnet_data --app situation-indexer-mainnet --region iad --size 1

# Set secrets on the mainnet app
fly secrets set --app situation-indexer-mainnet VOYAGER_API_KEYS=key-1,key-2,key-3 ADMIN_API_KEY=your-key

# Deploy with mainnet RPC and Voyager API endpoints
fly deploy --config fly.mainnet.toml

# Verify
curl https://situation-indexer-mainnet.fly.dev/health

fly.mainnet.toml is configured with:

• STARKNET_NETWORK=mainnet
• STARKNET_RPC_URL=https://api.cartridge.gg/x/starknet/mainnet
• VOYAGER_API_BASE_URL=https://api.voyager.online/beta
• Persistent volume indexer_mainnet_data mounted at /data

The fly.toml is pre-configured with:

• shared-cpu-1x / 512MB RAM
• Persistent volume mounted at /data
• auto_stop_machines = false (always-on for polling)
• HTTPS enforced

Useful Fly commands

# View logs
fly logs

# SSH into the machine
fly ssh console

# Restart the app
fly machines restart

# Check volume
fly volumes list

Deploying to Hetzner

Hetzner packaging lives under packages/indexer/deploy/hetzner. It uses the same root Dockerfile.indexer, persists SQLite under /var/lib/situation-indexer, binds the container to localhost by default, and includes a systemd unit plus a generic rsync deploy script.

cd packages/indexer/deploy/hetzner
cp .env.example .env
# Fill in VOYAGER_API_KEYS and ADMIN_API_KEY
docker compose -f compose.yml up -d --build
curl http://127.0.0.1:3000/health

See packages/indexer/deploy/hetzner/README.md for the Fly parity notes, mainnet/Sepolia env templates, Fly runtime env export helper, systemd install, and SQLite migration commands.