@the-situation/indexer

ETL indexer and REST/WebSocket API for The Situation prediction markets. Polls the Voyager explorer API for on-chain events, decodes them, stores everything in SQLite, and serves it over HTTP and WebSocket.

Quick Start

# From the repo root
cp packages/indexer/.env.example packages/indexer/.env
# Fill in VOYAGER_API_KEY and ADMIN_API_KEY

bun run --filter '@the-situation/indexer' dev

The server starts at http://localhost:3000. Register a market to begin indexing:

curl -X POST http://localhost:3000/admin/markets \
  -H "Authorization: Bearer <ADMIN_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"address": "0x02de...", "title": "My Market", "category": "crypto"}'

Events will begin appearing within ~15 seconds.

Environment Variables

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | VOYAGER_API_KEY | Yes* | — | Single Voyager explorer API key | | VOYAGER_API_KEYS | Yes* | — | Comma/whitespace-separated Voyager API key pool for redundancy | | ADMIN_API_KEY | Yes | — | Bearer token for /admin/* endpoints | | DB_PATH | No | ./data/indexer.db | SQLite database file path | | PORT | No | 3000 | HTTP server port | | STARKNET_NETWORK | No | sepolia | Network default set (sepolia or mainnet) | | STARKNET_RPC_URL | No | network default | Starknet RPC endpoint | | VOYAGER_API_BASE_URL | No | network default | Voyager API base URL | | EVENT_POLL_INTERVAL_MS | No | 15000 | Event polling interval (ms) | | STATE_POLL_INTERVAL_MS | No | 30000 | Market state refresh interval (ms) | | POSITION_POLL_INTERVAL_MS | No | 60000 | Position refresh interval (ms) |

VOYAGER_API_KEY or VOYAGER_API_KEYS must be set. When multiple keys are configured, the indexer rotates across them. A key that receives a Voyager 429/503 response is cooled down independently, and requests continue through the remaining keys until every key is cooling down.

API Reference

Public Endpoints

GET /health

Returns service health and basic stats.

{
  "status": "ok",
  "uptime": 3600,
  "dbStatus": "connected",
  "marketsCount": 1,
  "eventsCount": 42,
  "lastPollAt": null
}

GET /api/markets

List all registered markets with their latest on-chain state.

GET /api/markets/:address

Get a single market by contract address. Returns 404 if not found.

GET /api/markets/:address/events

Paginated event time series for a market.

| Query Param | Default | Description | |-------------|---------|-------------| | page | 1 | Page number | | pageSize | 100 | Items per page (max 500) | | from | — | Unix timestamp lower bound | | to | — | Unix timestamp upper bound |

{
  "data": [
    {
      "id": 1,
      "txHash": "0x...",
      "blockNumber": 6290091,
      "timestamp": 1770487426,
      "eventType": "trade_executed",
      "trader": "0x...",
      "mean": 13,
      "stdDev": 3,
      "lowerBound": 10,
      "upperBound": 16,
      "oldMean": 12,
      "oldStdDev": 3,
      "collateralPosted": "91208000000000000"
    }
  ],
  "total": 42,
  "page": 1,
  "pageSize": 100
}

GET /api/markets/:address/traders

List all discovered traders and their positions for a market.

GET /api/positions/:trader

All positions held by a trader across markets.

GET /api/rankings

Global leaderboard sorted by PnL.

| Query Param | Default | Description | |-------------|---------|-------------| | limit | 50 | Max entries (max 200) |

WS /ws

WebSocket endpoint for real-time event streaming. Receives JSON messages when new events are indexed:

{
  "type": "new_events",
  "marketAddress": "0x...",
  "data": { "count": 3, "latestBlock": 6395140 }
}

Admin Endpoints

All admin endpoints require Authorization: Bearer <ADMIN_API_KEY>.

POST /admin/markets

Register a new market for indexing.

curl -X POST http://localhost:3000/admin/markets \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "address": "0x02de...",
    "title": "BTC Price Market",
    "description": "Predict the price of BTC",
    "category": "crypto",
    "topics": ["btc", "price"]
  }'

PUT /admin/markets/:address

Update market metadata (title, description, category, topics, is_active).

DELETE /admin/markets/:address

Soft-delete a market (stops indexing, keeps historical data).

ETL Pipeline

The indexer runs three polling loops:

| Loop | Interval | Description | |------|----------|-------------| | Event indexer | 15s | Fetches events from Voyager, decodes, stores in SQLite, broadcasts to WebSocket | | State refresher | 30s | Reads current market state from Starknet RPC | | Position refresher | 60s | Reads positions for discovered traders from Starknet RPC |

Supported event types:

| Event | Encoding | Description | |-------|----------|-------------| | TradeExecuted | SQ128x128 (4-limb) | Current-generation trade events | | MarketInitialized | SQ128x128 | Market bootstrap event | | MarketSettled | market-type specific | Settlement event (continuous + multinoulli) | | MarketSettledMulti | multinoulli only | Multi-outcome settlement event | | TradeMoved | Q96 | Legacy trade events | | Bootstrapped | Q96 | Legacy bootstrap events |

Supported market types for indexing: normal, lognormal, bivariate, multinoulli.

Events are deduplicated by a stable per-event identity (event_uid) rather than transaction hash, so multiple events from the same transaction are indexed correctly. The indexer maintains a per-market cursor tracking the last indexed block number.

Building

# Type-check only
bun run --filter '@the-situation/indexer' typecheck

# Full build (compiles to dist/)
bun run --filter '@the-situation/indexer' build

# Run tests
bun run --filter '@the-situation/indexer' test

# Run opt-in performance suite (prints throughput/latency/memory metrics)
bun run --filter '@the-situation/indexer' test:perf

# Run strict perf checks (enables hard memory-trend assertions)
bun run --filter '@the-situation/indexer' test:perf:strict

Performance Suite Notes

The perf suite is intentionally opt-in (RUN_PERF=1) and uses relative regression guards to reduce machine-specific flakes:

• Ingestion throughput: compares second-half median throughput vs first-half.
• Query latency: checks tail amplification (p95 vs p50) under mixed API load.
• Memory trend: compares retained heapUsed/rss medians across sustained rounds.

In strict mode (PERF_STRICT=1), the suite requires GC availability and enforces hard memory trend thresholds suitable for dedicated performance runners.

Tests print the measured summaries (min, p50, p95, p99, max, mean) to help interpret changes across commits.

Deploying to Hetzner

Hetzner packaging lives under packages/indexer/deploy/hetzner. It uses the same root Dockerfile.indexer, persists SQLite under /var/lib/situation-indexer, binds the container to localhost by default, and includes a systemd unit plus a generic rsync deploy script.

cd packages/indexer/deploy/hetzner
cp .env.example .env
# Fill in VOYAGER_API_KEYS and ADMIN_API_KEY
docker compose -f compose.yml up -d --build
curl http://127.0.0.1:3000/health

See packages/indexer/deploy/hetzner/README.md for the mainnet env template, systemd install, public HTTPS via Caddy/sslip.io, and SQLite migration commands.