quant-daemon

Market data daemon for the Quant Workbench editor. Deliberately not part of the editor fork: it is a separate process with its own lifecycle, and keeping it out keeps the fork's diff against upstream VS Code small.

Two planes

provider (coinbase / binance / synthetic)
        │
        ▼
   ┌─────────┐  JSON   ┌────────────────┐
   │ control │────────▶│ extension host │   subscribe, history, quotes
   │  :8787  │         └────────────────┘
   ├─────────┤
   │  data   │ binary  ┌────────────────┐
   │  :8788  │────────▶│ chart webviews │   packed 32-byte tick records
   └─────────┘         └────────────────┘

Ticks never traverse the extension host. Webview.postMessage has no transfer list, so every tick routed that way costs a structured clone plus two IPC hops, on a thread shared with every other extension. Webviews open their own socket and read frames straight out of an ArrayBuffer.

Batching is the whole trick

Ticks are accumulated and flushed on a fixed interval (--flush-ms, default 16ms ≈ 60Hz). Frame count is therefore a function of the clock, not the feed:

| Target rate | Ticks/s delivered | Frames/s | Ticks/frame | Bytes/s | |---|---|---|---|---| | 10,000 | 9,862 | 62 | 159 | 309 KiB | | 50,000 | 49,437 | 62 | 797 | 1.5 MiB | | 200,000 | 197,250 | 62 | 3,181 | 6.2 MiB |

Measured on this machine with --provider synthetic, 8 symbols, decoded by a real client (npm run build && node dist/probe.js). Zero malformed frames at every rate.

The flat 62 frames/s column is the point. A webview handles 62 messages/second whether the feed is doing 10k or 200k ticks/s. The same load over postMessage would be 200,000 structured clones per second.

Backpressure

A client whose send buffer exceeds maxBufferedBytes (1 MiB) gets frames dropped, not queued. Stale ticks have no value, and an unbounded queue turns one slow consumer into a daemon-wide memory leak. Drops are counted and reported under --bench.

Providers

| Provider | Asset class | Auth | Notes | |---|---|---|---| | coinbase | crypto | none | Default. matches channel + candles REST. Trades 24/7. | | alpaca | US equities | API key | Trades channel + bars REST. Market hours only. | | binance | crypto | none | aggTrade + klines. Returns HTTP 451 from some networks, including this one. | | synthetic | — | — | Offline. --rate sets trades/sec. For benchmarking. | | bars | US equities | none | History only, no ticks. Reads a compacted HIST series over HTTP. Defaults to bars.sophostechne.com. |

Coinbase is the default because it needs no credentials, isn't geo-blocked here, and trades continuously — so it works at any hour, which matters for development.

bars is the odd one, and deliberately so: equity ticks are the part of this system that cannot be given away, since every real-time source is licensed per subscriber, while bars derived from IEX HIST captures may be redistributed outright. So it serves the half that can be published and leaves live prices to a provider the user holds credentials for. Put it last, and it claims only the symbols the live feed did not:

node dist/index.js --provider coinbase,alpaca,bars

There is nothing to connect to: start() reads the catalogue, history() reads one file, and subscribe() does nothing. A symbol or timeframe that is not published returns no bars rather than an error, so BarSeries builds those buckets from live trades — the same contract the other providers keep.

Running several venues at once

node dist/index.js --provider coinbase,alpaca --symbols BTC-USD,ETH-USD,AAPL,MSFT

Each symbol is routed to the provider that lists it, so crypto and equities coexist in one daemon and one watchlist. Clients see a single control plane and a single tick stream; they never learn there is more than one venue behind them.

Routing is by each provider's claims(), first match wins, so order matters — synthetic claims everything and must come last. A symbol nobody claims is reported rather than guessed at:

No configured provider trades 'MSFT'. Running: coinbase, binance.

This is also what keeps colliding tickers apart:

| Symbol | Routes to | Is | |---|---|---| | META | alpaca | Meta Platforms, the equity | | META-USD | coinbase | a token trading near $4 |

Passed blindly to whichever provider happened to be running, those were the same request — and the wrong answer looked entirely plausible.

Neither crypto provider's finest candle is below 1 minute (nor Alpaca's), so 1s and 5s history returns empty and those buckets build from live trades rather than being fabricated from coarser data.

Alpaca (US equities)

Credentials come from the environment, never argv — argv lands in shell history and ps:

export APCA_API_KEY_ID=...
export APCA_API_SECRET_KEY=...
node dist/index.js --provider alpaca --symbols AAPL,MSFT,SPY

Paper-trading keys work; generate them at https://app.alpaca.markets. Use the Trading API, not the Broker API — Broker API is for running a brokerage on behalf of other people and carries the corresponding regulatory obligations.

Two things will make a working setup look broken:

• --feed iex (the default) is one venue, carrying a small share of consolidated volume. Prices disagree with the real tape and thin names barely print. --feed sip is the consolidated tape and needs a paid plan; without one it returns 403.
• Equities keep market hours. Outside roughly 09:30–16:00 ET the trade stream is silent, which is indistinguishable from a dead connection unless you read the status messages.

Usage

Published as @sophostechne/quant-daemon, so it runs without a checkout:

npx @sophostechne/quant-daemon               # coinbase, BTC-USD/ETH-USD/SOL-USD
npx @sophostechne/quant-daemon --help

From a checkout:

npm install
npm run build

npm start                                    # coinbase, BTC-USD/ETH-USD/SOL-USD
node dist/index.js --provider coinbase --symbols BTC-USD,ETH-USD --bench
node dist/index.js --provider synthetic --rate 200000 --bench
node dist/index.js --help

Verify the data plane end to end:

node dist/probe.js ws://127.0.0.1:8788 10

The probe decodes frames using the same fixed-offset reads as the chart webview and reports tick counts, byte rates, malformed frames and the observed price range. It exits non-zero if nothing arrived or any frame was malformed.

Protocol

src/protocol.ts is mirrored in two other places that must change with it:

• src/protocol.ts in quant-vsce
• media/chart/chart.js there (hand-decodes the same layout in the webview)

Symbols are interned to a uint16 by the daemon and announced on the control plane via symbolMap. Frames carry the id, not the string — that is what keeps a tick at 32 bytes. Ids are stable for the daemon's lifetime and are not recycled on unsubscribe, because in-flight frames would otherwise resolve to the wrong instrument.

Historical bars from IEX HIST

src/hist/ builds bars from IEX's published daily captures, which are free and, unusually, carry an explicit right to redistribute what you derive from them.

npm run hist -- --date 20260812 --symbols AAPL,MSFT --timeframe 1d --out bars.json
npm run hist -- --file capture.pcapng.gz --timeframe 1m          # a local capture

npm run ingest -- --from 20260801 --to 20260812 --timeframes 1d,1h --root data/bars

ingest accumulates days on disk: one file per timeframe per day under --root, plus a manifest recording what happened to each date. It is built for a backfill that cannot finish in one sitting — two years is ~500 sessions at ~10 GB each — so days are independent, the manifest is written after every one, and a re-run skips what is already recorded. The unit of lost work is one day, never the run. Every requested timeframe is built from a single pass, because the download is the entire cost.

compact then transposes those day files into the per-symbol series a chart actually asks for:

npm run compact -- --root data/bars          # -> data/bars/series/{timeframe}/{SYMBOL}.json

Ingest is organised the way the source is — one capture, one day, every symbol — and serving that shape would mean reading every day file to answer one request for one symbol. Compacting transposes it once, offline, so a request becomes a single file read that stays cacheable until the next session closes. index.json beside each timeframe lists every symbol with its bar count and range, which is what a catalogue endpoint serves.

The transpose is the awkward direction, since every symbol needs a piece of every day. Days are accumulated until a bar budget is reached and then merged a batch at a time, so memory is bounded by the budget rather than by the length of the backfill. Bars are keyed by time on merge rather than appended: re-compacting a day must replace its bars, and a series holding two bars at one timestamp would be quietly wrong everywhere downstream. A re-run compacts only what arrived since; --force rebuilds from every day file and produces byte-identical output.

Weekends are skipped without asking. Holidays are not: rather than carry a calendar that goes stale, a day with no capture is recorded as no-session. The archive answers 403 rather than 404 for a date that never traded — it denies listing, so it will not confirm an object's absence — which makes a withdrawn archive indistinguishable from a quiet market one day at a time. So the job stops after six consecutive absent weekdays: no US market closure is that long, and recording a year of that as "no session" would leave nothing downstream able to tell the difference.

The chain is pcapng → Ethernet/IPv4/UDP → IEX-TP → TOPS trade reports → Bar, streamed throughout. A day of TOPS is around 10 GB gzipped and yields a few hundred kilobytes of daily bars: trade reports are under 2% of the messages and quote updates are the rest. So this belongs on whatever host serves bars, run once per trading day — not on a user's machine, and not in the request path. The archive is not listable, so URLs are constructed from the date; a 404 means a non-trading day or a feed version this tool predates.

Two caveats that belong in front of anyone reading a chart built this way. IEX is a few percent of consolidated US equity volume, so these bars are real prints but not the consolidated tape — thinner and gappier than a SIP-derived series, and IEX says plainly that the data is "intended to provide you with a reference point only, rather than as a basis for making trading decisions." And buckets with no trades are absent rather than carried forward, because a backtest cannot tell an invented bar from a real one.

Redistribution requires attribution. IEX_ATTRIBUTION in src/hist/tops.ts carries the exact wording, and it is emitted in the output document so it travels with the data:

Data provided for free by IEX. By accessing or using IEX Historical Data, you agree to the IEX Historical Data Terms of Use.

Not done yet

• No authentication on either port. Both bind to 127.0.0.1 only, but any local process can connect. Fine for a single-user desktop, not for a shared host.
• No persistence: bar history is in-memory and rebuilt from the provider on restart.
• Order routing and strategy execution are out of scope by design — they belong in their own process for the same reason ticks avoid the extension host.