@securities/archiver

An international market-data collector and publisher. It records issuers (publicly listed companies), their securities and listings, and per-symbol prices/splits/dividends/fundamentals, it derive some analysis, and stores everything as directories of JSON files. Feeds append the files in place; the read surface is static JSON.

The field contract lives in src/contract.js, and fully-worked, machine-checked artifact examples live in examples/.

What it does:

• collects market data (prices, splits, dividends) from:
  • SEC EDGAR, Yahoo, FMP, Tiingo, Stooq, AlphaVantage, etc.
  • data is stored as-published (immutable), only current year chunks are updated
  • Prices are raw, as-traded. Split-adjusted prices are derived.
• archives the data as chunked JSON files
• serves the JSON via local HTTP, nginx, or CDN. Examples URIs:
  • https://securities-js.github.io/market-data/us/symbol/AAPL/meta.json
  • https://securities-js.github.io/market-data/us/symbol/AAPL/prices/2025.json
  • https://securities-js.github.io/market-data/us/symbol/AAPL/fundamentals.json
  • https://securities-js.github.io/market-data/au/symbol/CXZ/meta.json
• runs a web UI for browsing, editing, and resolving issuers and symbols

Data model

Jurisdiction-neutral. Three layers:

• Entity — an issuer, keyed by its primary regulator. Its id <cc>/<repo>/<localId> is its path, e.g. us/edgar/0000320193, ca/sedar/00003781, au/asic/68004240313. It holds regulator[], a names[] rename history, fund series[] (name histories), and securities[].
• Security — a tradeable instrument on an entity (a common line, a fund share class, a 529 unit). Carries its isin (one per security, shared across listings), identifiers (CUSIP, …), an optional series reference (funds), an optional underlying cross-reference (a 529 → its SEC fund), and listings[].
• Listing — a {symbol, exchange_mic, currency, is_primary?} on one venue (ISO 10383 MIC).
• Symbol — a ticker, pointing at entities through date-bounded entity[] windows, because tickers get reused). Prices/events/fundamentals/analysis hang off it.

Symbol addressing. URLs are uniformly /<cc>/symbol/<TICKER>/. A same-country two-exchange collision — rare — would disambiguate as /<cc>/<exchange>/symbol/<ticker>.)

Symbol status is derived (see src/lifecycle.js) from the presence of an end on a listing/window/termination.

See examples/ for a fully-formed meta.json of each shape (US company, fund trust, foreign cross-listing, 529, dual-class, Australian) and each per-symbol artifact.

Regional support

The path's second segment is the authority-or-repository that owns the identifier. Adding a region is a drop-in file — reference/country/<cc>/exchanges.json (its exchanges + regulators); no code change.

| Region | Regulator / repository | Path | Automated feed? | | --- | --- | --- | --- | | US | SEC / EDGAR | us/edgar/<CIK> | Yes — fundamentals (EDGAR), ticker→CIK resolution, N-CEN fund series/classes, full-text search | | US (municipal) | MSRB / EMMA | us/msrb/<id> | No — 529 holdings; manual, with EMMA lookup links | | Canada | CSA / SEDAR+ | ca/sedar/<profile> | No public API — manual entry, SEDAR+ search links in the resolver | | Australia | ASIC | au/asic/<ABN> | No — manual | | United Kingdom | FCA | gb/fca/<id> | No — seeded, manual | | Europe | ESMA | eu/esma/<LEI> | No — seeded, manual |

Limitations.

• Only SEC/EDGAR has an automated issuer feed. Every non-US entity is created and edited by hand through the web UI (POST /api/entity), which the resolver assists per jurisdiction (SEDAR+ links for Canada, EMMA links for 529s). There is no public SEDAR+/ASIC/FCA/ESMA API to poll.
• Price feeds are ticker-keyed. Prices/splits/dividends come from Yahoo et al. by bare ticker, so US listings (including OTC cross-listings like MJGCF) collect normally; a foreign primary listing may need an exchange-suffixed symbol the feeds aren't yet mapped to. Entity/listing metadata is fully international; automated price collection is only reliable where a feed serves the ticker.
• Currency is recorded, not converted — no FX. A listings[].currency says what the prices are in.

Data formats

Files are pretty-printed for line-oriented git diffs, with top-level keys sorted (stable diffs). Optional fields are emitted only when present. The canonical shapes — the store layout, every meta.json variant, and the prices/<year>.json / events.json / fundamentals.json / analysis.json artifacts — are documented with worked examples in examples/ (examples/README.md). The entity/symbol examples are round-trip-tested against the shapers, so they never drift from the code.

analysis.json is derived from fundamentals.json + prices + split events (P/E, ROE, ROIC, current ratio, owner earnings, growth trends) — no feed; recomputed when fundamentals change. close is the raw, as-traded price — canonical and immutable; split-adjusted prices are derived from events.json. Artifacts carry a provenance[] of {source, at?} so we can traces its origin.

Data sources & metering

src/contract.js registers each source. auto sources (Yahoo prices, SEC EDGAR fundamentals) may be polled on the daily schedule. metered sources (Tiingo, Stooq, FMP, Alpha Vantage) have API quotas and are only called explicitly.

CLIs

node bin/supervisor.js     # the deployable daemon: HTTP + scheduled collection (see Deployment)
node bin/serve.js          # HTTP server only (control plane + static reads)
node bin/collect.js …      # batch/automation collector (what the supervisor spawns):
  --symbols AAPL,MSFT             # daily prices for these (auto sources)
  --all                           # every symbol in index.json
  --symbols AAPL --fundamentals   # SEC EDGAR fundamentals (auto; also recomputes analysis)
  --cik existing                  # refresh the SEC entities (us/edgar/*) we already track
  --cik all                       # seed all active SEC entities (forces SEC_CIK_ARCHIVE_ACTIVE)
  --force / --out <dir>           # re-fetch up-to-date symbols / alternate store

Per-symbol operations (resolution/pinning, metered backfill, fund series, analysis recompute) live in the web UI, which drives the same library functions.

HTTP API

The server is both a control plane and a static read surface (front it with nginx/CDN if you like). Entity ids contain slashes and are matched as wildcards — pass them un-encoded (/api/entity/us/edgar/0000320193).

Reads

| Method | Path | Purpose | | --- | --- | --- | | GET | /health | liveness | | GET | /api/config | { secConfigured, countries[] } — SEC readiness + the reference jurisdictions (drives the resolver's country/regulator/exchange pickers) | | GET | /api/securities | the whole pool as render-ready rows (symbol, name, type, country, currency, MIC, ISIN, frozen reason, aka search aliases, coverage) | | GET | /api/entity/* | one entity + the symbols pointing at it (* = us/edgar/0000320193) | | GET | /api/symbol/:symbol | one symbol's full inventory + linked entities | | GET | /api/resolve/:symbol | resolver candidates: SEC ticker matches, current links, EMMA (529) / SEDAR+ (CA) lookup links | | GET | /index.json; /<cc>/symbol/<T>/meta.json (e.g. /us/symbol/AAPL/…, /au/symbol/CXZ/…), incl. prices/<year>.json; /<cc>/<repo>/<id>/meta.json (e.g. /us/edgar/0000320193/meta.json) | static read surface (CORS *) |

Resolution & editing (SEC-backed routes require SEC_EDGAR_EMAIL; see Configuration)

| Method | Path | Purpose | | --- | --- | --- | | GET | /api/resolve/:symbol/by-name, /by-fulltext, /probe/:cik | SEC name / full-text search; probe a trust's classes | | POST | /api/entity | create an entity — {cik} (EDGAR build) or {regulator[], names[], securities[]} (any jurisdiction) | | PUT | /api/entity/* | edit an entity (names/regulators/series/securities/termination) | | DELETE | /api/entity/* | delete an entity (refuses to orphan linked symbols) | | PUT | /api/symbol/:symbol | edit a symbol's entity[] windows | | DELETE | /api/symbol/:symbol/entity/* | unlink one entity window (any jurisdiction) | | POST | /collect/:symbol/cik | pin a symbol to a CIK (± series/class) | | POST | /collect/:symbol/secmeta, /collect/:symbol/fundamentals, /collect/:symbol/analysis | per-symbol SEC metadata / fundamentals / analysis | | POST | /collect/fund-series | resolve a trust's N-CEN series/classes | | POST | /collect/:symbol | daily price refresh (?force=true to bypass the up-to-date skip) | | POST | /backfill/:symbol | metered backfill; body { source, token? } | | POST | /collect/bulk | run secmeta/analysis over a list |

curl localhost:8081/api/securities | jq '.[] | select(.symbol=="AAPL")'
curl localhost:8081/api/entity/us/edgar/0000320193          # un-encoded slashes
curl -X POST localhost:8081/api/entity -H 'content-type: application/json' \
  -d '{"regulator":[{"authority":"ASIC","country":"AU","primary_repository":"ASIC","local_identifier_type":"ABN","local_identifier":"68004240313"}],
       "names":[{"name":"Connexion Mobility Ltd"}],
       "securities":[{"id":"common","type":"common","isin":"AU000000CXZ8",
         "listings":[{"symbol":"CXZ","exchange_mic":"XASX","currency":"AUD","is_primary":true}]}]}'

Configuration (env)

| Var | Default | Meaning | | --- | --- | --- | | SECURITIES_DATA | ../../securities-data | the artifact store directory | | SECURITIES_PORT | 8081 | HTTP port | | SEC_EDGAR_EMAIL | — | contact email, required by SEC. When unset, SEC-backed actions are disabled | | TIINGO_TOKEN, FMP_TOKEN, ALPHAVANTAGE_TOKEN | — | metered-source API keys | | SEC_CIK_ARCHIVE_ACTIVE | falsy | truthy → the SEC-metadata sweep archives every active SEC CIK; See below | | SECURITIES_CLOSE_BUFFER_MS | 1800000 (30m) | delay after market close before collecting prices | | SECURITIES_MAX_BACKOFF_MS | 21600000 (6h) | cap on the price no-data retry backoff | | SECURITIES_FUNDAMENTALS_CHECK_MS | 86400000 (24h) | how often the daemon re-evaluates which symbols are due for a 10-K | | SECURITIES_SECMETA_CHECK_MS | 86400000 (24h) | how often the daemon runs the SEC ticker→CIK metadata sweep | | SECURITIES_SERVER_MAX_BACKOFF_MS | 30000 (30s) | cap on the supervisor's restart backoff for the HTTP server |

SEC_CIK_ARCHIVE_ACTIVE

By default the SEC-metadata sweep tracks only the SEC entities you actually hold (any linked from a symbol) plus any a run requests, so us/edgar/ mirrors your portfolio. Setting it truthy archives every CIK on SEC's active ticker lists (~8000+).

• Pros: a durable record of the full active universe; a git diff shows exactly when a CIK joins or leaves SEC's active list.
• Cons: large, noisy daily diffs — thousands of unrelated entities churn on every sweep.

The artifact store is its own git repo

SECURITIES_DATA points at a separate repo, so the data and the daily collector's commits stay out of the code history. The chunked, immutable-historical layout means a daily run only rewrites the current-year price chunks + updated symbols' events/meta + index.json, so commits stay small. A cron alternative to the supervisor:

The store is seeded from the MySQL DB of the legacy Matt::Finance app, using the monorepo's tools/migrate-from-mysql (dev tooling, not part of this package). Thereafter the service owns the artifacts via its feeds.

History

This tool was originally written in Perl (2007–2015) to collect and serve US market data for the unpublished Matt::Finance app. It collected price data from Yahoo and clumsy-but-effective web scrapers to collect fundaments from SEC EDGAR, Yahoo Finance, Google Finance, MSN Money, DailyFinance, and Reuters.