congressional-disclosures

Build a local, queryable database of U.S. congressional financial disclosures from the official House Clerk and Senate eFD sources.

This is the open-source congressional data engine behind NexusTrade. The package gives you the audited files, the extraction pipeline, and a local database you control. NexusTrade adds market data, natural-language research, stock screening, backtesting, portfolio generation, optimization, and deployment on top of the same congressional tables.

The package discovers filings, downloads their original documents, decrypts House PDFs, OCRs scans, extracts transactions with independent model reads, reconciles disagreements, and writes a resumable SQLite data lake. It covers both the House and Senate and retains source URLs, document hashes, extraction status, repeated reports, and amendment history.

• Want the data immediately? Run npx congressional-disclosures@latest download --sqlite. It downloads the current audited Congressional Stock Trades dataset from Hugging Face and creates a ready-to-query SQLite database without model or OCR credentials.
• Want your own local lake? Run the CLI against the official sources.
• Building another product? Install the library and replace only the model, OCR, cache, or storage adapters you need to own.

What can you investigate?

Once the lake exists, ordinary SQL can answer questions such as:

• Which members disclosed the most purchases in a given year?
• Which stocks attracted purchases from the most distinct members?
• How long did members wait between a transaction and its disclosure?
• How do House and Senate trading patterns differ?
• Which filings failed extraction, and why?
• What did the public know on a particular date, before a later amendment?

Performance questions require market prices in addition to disclosure data. For example, “Which politician's disclosed purchases performed best?” needs an explicit return horizon, weighting method, and decision about whether returns begin on the transaction date or the public disclosure date. The lake preserves both dates so that analysis can state that choice instead of hiding it.

Get a queryable SQLite database

This is the shortest path from nothing to SQL:

npx congressional-disclosures@latest download --sqlite

That one command downloads the checksum-verified Parquet snapshot and creates:

./congressional-stock-trades/congressional-disclosures.sqlite

No Docker, Python, DuckDB, API keys, OCR tools, or paid model calls are needed. The CLI checks disk space separately for the compressed download and for the larger SQLite database before it starts either stage.

How can npx do all of that?

npx asks npm for the published congressional-disclosures package, caches it, and runs the package's CLI without making you clone this repository or install anything globally. The download --sqlite command then:

1. Reads snapshot.json from the public Hugging Face dataset.
2. Calculates the required download and SQLite space before writing data.
3. Downloads the published Parquet files, checking every byte count and SHA-256.
4. Reuses files that are already present and still match their checksum.
5. Builds a temporary SQLite database, verifies all three table counts, and only then moves the finished database into place.

It downloads the already-built public snapshot. It does not rerun OCR or model extraction on your laptop. Use sync when you deliberately want to rebuild the lake from the official House and Senate sources yourself.

Confirm the database works without writing any SQL:

npx congressional-disclosures@latest status \
  --db ./congressional-stock-trades/congressional-disclosures.sqlite

Then run a real query. This example shows the latest currently active Nancy Pelosi events in the snapshot:

sqlite3 -header -column \
  ./congressional-stock-trades/congressional-disclosures.sqlite \
  "SELECT ticker, action, amount_low, amount_high, first_available_at
   FROM political_trade_events
   WHERE filer_last = 'Pelosi' AND superseded_at IS NULL
   ORDER BY first_available_at DESC
   LIMIT 10;"

Choose another database path when needed:

npx congressional-disclosures@latest download --sqlite ./data/congress.sqlite

Export any query to CSV with SQLite itself:

sqlite3 -header -csv \
  ./congressional-stock-trades/congressional-disclosures.sqlite \
  "SELECT * FROM political_trade_events WHERE superseded_at IS NULL;" \
  > congressional-trade-events.csv

Download Parquet only

If your tool already reads Parquet, omit --sqlite:

npx congressional-disclosures@latest download

The default destination is ./congressional-stock-trades. Before downloading, the CLI reads the public snapshot.json, calculates the exact selected download size, and prints both required and available disk space in GB. It stops before fetching data files if the destination does not have enough room. Every Parquet file is checked against its published byte size and SHA-256; reruns verify and reuse files that are already correct.

Download a smaller slice when you do not need the complete history:

npx congressional-disclosures download --table political_trade_events --year 2026
npx congressional-disclosures download --out ./data/congressional-trades

The download contains Parquet files plus snapshot.json. All three public tables include sourceUrl, which links each filing, printed transaction, or reconciled event back to the official House or Senate record.

The hosted snapshot has a target refresh interval of 20 hours. That is an operating target, not a guarantee: an upstream outage or failed audit can delay publication. Read generatedAt in the downloaded snapshot.json when freshness matters; it records the snapshot you actually received.

Rebuild the lake from official sources

1. Check the machine

The one-command public snapshot above does not need this setup. Use this section only when you want to discover and extract the official filings yourself.

Node.js 22.5 or newer is required. Use a stable Node release rather than an alpha or nightly build. Install Poppler and Tesseract first:

# macOS
brew install poppler tesseract

# Debian or Ubuntu
apt-get install poppler-utils tesseract-ocr

Then provide your extraction credentials and run the preflight check:

export OPENROUTER_API_KEY=...  # model extraction
export MISTRAL_API_KEY=...     # OCR for scanned/image filings

npx congressional-disclosures doctor

doctor reports whether each executable and credential is present without printing credential values.

2. Sync official filings

npx congressional-disclosures sync \
  --db ./congress.db \
  --since 2024 \
  --accept-senate-terms

npx congressional-disclosures status --db ./congress.db
npx congressional-disclosures audit --db ./congress.db

Senate access requires --accept-senate-terms, acknowledging the eFD site's usage agreement. Start with --dry-run or --max-filings 5 when evaluating the workflow. Model and OCR calls can cost money; raw documents and provider responses are cached by content/request hash so reruns do not pay for the same work again.

3. Query it

Recent normalized transaction rows:

sqlite3 ./congress.db \
  "SELECT display_name AS member,
          transaction_date,
          COALESCE(printed_ticker, resolved_ticker) AS ticker,
          action,
          amount_bracket,
          available_at
   FROM political_trades
   ORDER BY available_at DESC
   LIMIT 10;"

Members with the most purchases first disclosed in 2024:

SELECT
  member_id,
  display_name AS member,
  COUNT(*) AS purchases
FROM political_trade_events
WHERE action = 'purchase'
  AND first_available_at >= '2024-01-01'
  AND first_available_at < '2025-01-01'
  AND superseded_at IS NULL
GROUP BY member_id, display_name
ORDER BY purchases DESC
LIMIT 20;

Stocks purchased by the most distinct members:

SELECT
  ticker,
  COUNT(*) AS purchases,
  COUNT(DISTINCT member_id) AS distinct_members
FROM political_trade_events
WHERE action = 'purchase'
  AND ticker IS NOT NULL
  AND superseded_at IS NULL
GROUP BY ticker
ORDER BY distinct_members DESC, purchases DESC
LIMIT 20;

Average disclosure lag by chamber:

SELECT
  chamber,
  ROUND(AVG(julianday(first_available_at) - julianday(transaction_date)), 1)
    AS average_days_to_disclosure
FROM political_trade_events
WHERE transaction_date IS NOT NULL
  AND superseded_at IS NULL
GROUP BY chamber;

Use political_trade_events for counts and aggregate analysis. It consolidates the same economic trade when it is reported repeatedly and versions amendments instead of double-counting them.

What lands in SQLite?

| Table | Grain | Use it for | |---|---|---| | political_filings | One official document | Coverage, provenance, extraction failures, and audit trails | | political_trades | One transaction row printed on a filing | Inspecting exactly what a document reported | | political_trade_events | One version of a consolidated economic event | Counts, aggregates, point-in-time research, and downstream signals |

Event rows include a nullable comment copied from the extracted transaction row. It may contain an option's strike, expiration, or contract count, but it is unparsed filing text and does not establish a unique option contract or holding. Older published snapshots read this field as null.

Private tables record schema migrations, sync runs, and per-filing receipts. Original documents and paid responses live in the content-addressed cache selected by --cache-dir; large blobs are not stored inside SQLite.

Two dates matter:

• transaction_date is when the disclosed transaction occurred.
• available_at/first_available_at is when the information became public.

For a realistic backtest or alert, never make a trade visible before its public availability timestamp. Later corrections are visible only from their own available_at timestamp until superseded_at.

Ticker handling

The package does not guess a missing ticker from an asset description. printed_ticker contains only the symbol printed on the disclosure. When a filing prints a ticker, resolution_status is printed; when it does not, resolution_status is unresolved, resolved_ticker stays null, and resolution_reason explains why.

resolved_ticker is reserved for an application that deliberately adds a resolver. That resolver may use an exchange directory, issuer identifiers, or manual review, but it must preserve the printed value and record its status and reason. This keeps an inferred symbol from being presented as something the member filed.

Commands

| Command | What it does | |---|---| | doctor | Checks Node, PDF/OCR tools, and provider configuration. | | download | Downloads and verifies the audited public snapshot; add --sqlite [FILE] to create a queryable database. | | sync --db FILE --since YEAR | Discovers, extracts, and stores filings. | | status --db FILE | Prints filing, trade, event, and failure counts. | | audit --db FILE | Runs orphan, event, amount, date, freshness, and sanity checks. |

Useful sync options:

--year YEAR
--chamber house|senate|both
--max-filings N
--model MODEL
--ocr-model MODEL
--cache-dir DIR
--dry-run
--accept-senate-terms

--dry-run discovers and plans without downloading filing documents, calling providers, or writing lake rows. A normal rerun resumes from committed filings, retries failed filings, and reuses cached provider responses.

Use it as a library

npm install congressional-disclosures

import {
  LocalCache,
  MistralOcrClient,
  OpenAiCompatibleCompletionClient,
  SQLitePoliticalRepository,
  syncPoliticalDisclosures,
} from "congressional-disclosures";

const cache = new LocalCache("./.congressional-disclosures");
const repository = new SQLitePoliticalRepository("./congress.db");

try {
  const summary = await syncPoliticalDisclosures({
    repository,
    cache,
    completion: new OpenAiCompatibleCompletionClient({
      apiKey: process.env.OPENROUTER_API_KEY!,
      cache,
    }),
    ocr: new MistralOcrClient({
      apiKey: process.env.MISTRAL_API_KEY!,
      cache,
    }),
    sinceYear: 2024,
    acceptSenateTerms: true,
  });
  console.log(summary);
} finally {
  await repository.close();
}

Official-source clients, provider clients, storage, caching, and progress reporting are injectable. The CLI supplies working defaults; an application only replaces a boundary when it has a concrete reason to.

Focused entry points—/extraction, /lake, /sources, /backfill, /integrity, and /storage—let a server import the congressional domain without loading the SQLite runtime.

Member identity

The official indexes spell members inconsistently. The House Clerk has listed Rep. Scott Franklin as Scott, C. Scott, Scott Scott and Scott Mr, and the Senate often files names in capitals. Every filing is therefore matched to a member of Congress using the public-domain congress-legislators data, and every table carries the result:

| Column | Meaning | |---|---| | member_id | Bioguide ID, one per person across name spellings and both chambers | | display_name | The member's official name; filer_first and filer_last keep the name as filed | | filer_key | member:<member_id> for members, the normalized filed name for anyone else | | identity_source | legislators, override, non_member, or unresolved |

A filing matches everyone who had served in its seat (House) or chamber (Senate) by the filing date, after accents, honorifics and credentials are removed from both names; ties break on given names, then on the most recent service. Nothing is matched by similarity. What the rules cannot decide is settled by a reviewed override that cites the filing proving it, such as a member filing under a married name, or a committee employee whose report reached the member index.

Only members of Congress have events. Filings by people who never served stay in political_filings and political_trades with identity_source = 'non_member', so the tables still account for every official document.

sync downloads the member data once per upstream commit and caches it beside the documents; pin a commit with --legislators-commit to reproduce a run. download never needs it, because the published tables already carry identity.

Version stability

The CLI command names, documented flags, SQLite table grains, and published JavaScript entry points are covered by SemVer starting with 1.0. A breaking change to one of those contracts requires a new major version. 2.0 was one: filer_key now names a member of Congress, events exist only for members, SyncOptions requires a resolver, and PoliticalRepository gained reidentify. A 1.x SQLite database is migrated in place on its next sync. New columns, new optional flags, extraction improvements, and additional audit findings may ship in minor releases when existing callers keep working.

Official sites and provider responses can change independently of this package. The live canaries detect that drift; they are evidence about the current source path, not a substitute for deterministic tests.

Production Parquet lakes

SQLite is the turnkey local path. A production application can retain its own S3-compatible object store and Parquet writer by supplying a ParquetLakePort to ManifestParquetPoliticalRepository:

import {
  ManifestParquetPoliticalRepository,
  syncPoliticalDisclosures,
} from "congressional-disclosures";

const repository = new ManifestParquetPoliticalRepository(parquetLakePort);
await syncPoliticalDisclosures({
  repository,
  cache,
  completion,
  ocr,
  sinceYear: 2012,
  acceptSenateTerms: true,
});

The package owns discovery, downloads, PDF handling, OCR validation, extraction, normalization, event construction, and integrity rules. The host application keeps its credentials, billing, physical Parquet implementation, scheduling, alerts, and application-specific enrichment.

Analyze the data in NexusTrade

Use this repository when you want the raw dataset, a local SQLite database, or the extraction code itself. Use NexusTrade when you want the application layer: ask questions about politicians and securities, combine disclosures with price history, screen stocks, generate competing politician-driven portfolios, backtest them, optimize them, and deploy the one you choose.

NexusTrade runs this package in production. The open-source release is not a toy export or a separate parser—it is the congressional disclosure engine the product consumes.

Reliability and limitations

The reliability work is part of the product:

• House PDFs are classified using text Poppler actually extracts. Encrypted documents are rewritten before any page is copied or split.
• Scans are rendered page by page and checked at independent resolutions.
• Independent model reads must agree; disagreements trigger reconciliation against the filed pages.
• One filing is replaced atomically. Failed filings remain present with a named reason and can be retried.
• A completed sync checks coverage against the official filings in that run; audit checks orphan rows, consolidated events, dates, amounts, and freshness.

The source records still impose unavoidable limits:

• Filings are self-reported and may be late, incomplete, or amended.
• Dollar values are statutory ranges, not exact position sizes or profits.
• A reported owner may be the member, spouse, joint account, or dependent child.
• Options and private assets do not behave like ordinary stock purchases.
• Extraction is probabilistic. Preserve provenance and inspect source filings before treating an individual row as definitive.

This software and dataset are for research and informational use, not investment advice. Review the dataset's statutory-use notice before redistributing records.

Development and release evidence

npm install
npm run typecheck
npm test
npm run build
npm run verify:release
npm run smoke:fixture
npm run smoke:live-senate -- 2024 /tmp/congressional-senate-canary

The fixture smoke test runs the compiled package against a checked-in encrypted filing with scripted provider responses. It proves packaging and deterministic pipeline behavior; live official-source canaries are separate release gates.

Contributing

Found a bad filing result or a package defect? Use the bug-report form and include the package version, exact command, public document id or URL, and the named failure. For a new model, OCR, storage, or decision integration, read CONTRIBUTING.md before writing an adapter; it maps proposals to the package's actual extension points and explains the offline-test requirement. Security reports follow SECURITY.md.

License

Code is released under the MIT License. Government disclosure records may carry separate statutory restrictions; consult the notice distributed with the public dataset.