@developerzava/stock-crypto-sentiment-analyzer

v2.0.0

Published

9 days ago

Sentiment and regime overlay for trading bots responding to policy-driven shocks.

0High
0Medium
0Low

developerzava

sentiment trading regime typescript

Stock Crypto Sentiment Analyzer

Sentiment-regime overlay for algorithmic trading bots. This package ingests tariff-related tweets, policy news, and market data; classifies narratives; scores crash/surge probability; detects market regimes; and exposes guardrail middleware so technical-analysis (TA) strategies can derisk automatically.

Features

Feed ingestion: Twitter/X, NewsAPI, generic market data, and explicitly enabled Alpaca/Benzinga news + multi-symbol bars auto-register from consumer configuration.
Narrative classification: A taxonomy keyword match decides whether text is relevant. Relevant text is labelled threat, escalation, de-escalation, or noise by regex heuristics. An optional transformer (via @xenova/transformers, off by default) can replace that label. It reads the polarity (positive or negative tone) of the text, not the direction of a policy. See "What Runs by Default" below.
Narrative taxonomy & overrides: Six-category taxonomy assets (assets/taxonomy/**) support consumer-provided ticker/sector vocabulary. Consumers must benchmark recall against their own universe; the package does not claim a universal recall floor.
Consumer-owned symbol sentiment: Consumers provide their own ticker/name/sector definitions; entity extraction + the rolling symbol store expose intraday crash/surge/confidence snapshots without a package-owned watchlist.
Sector intelligence: Consumer-owned sector/benchmark definitions provide deduplicated sector sentiment and transparent symbol-versus-sector/peer performance without changing guardrail or order behavior.
Catalyst ingestors: Earnings calendar, pre/post-market gapper feeds, and the macro calendar adapter normalize catalysts into sentiment events so guardrails can differentiate planned beats/misses from CPI/FOMC/jobs shocks.
Optional transformer blend: Off by default and not installed with the package: add @xenova/transformers to your project first (npm install @xenova/transformers). With SENTIMENT_SCORING_ENABLE=true, a text-classification model that you name (SENTIMENT_SCORING_MODEL) is blended into the lexicon score with weight 0.35. The three bundled presets (finbert, deberta-v3-large, fingpt) are all marked unavailable because @xenova/transformers 2.x cannot load them. If no model is named, selecting one falls back to the lexicon only; a model named in SENTIMENT_SCORING_MODEL takes precedence over the preset. Nothing on this path is benchmarked. See docs/models/transformer-presets.md.
Custom feed hooks: Built-in Alpaca adapters, custom-provider examples, and logging guidance let you plug proprietary catalysts directly into the overlay (docs/feeds/custom-feed-hooks.md).
Symbol metrics & alerts: /metrics exports per-symbol threat/escalation/de-escalation counters with Alertmanager/Grafana recipes (docs/alerts/symbol-metrics.md, docs/dashboards/symbol-heatmap.json).
Sentiment scoring: Computes (crashProb, surgeProb, confidence) from a hand-weighted 24-token lexicon (not a trained or calibrated model; see docs/models/finbert-tariff.md), persists scores with TTL, and logs context for audits. confidence is max(crashProb, surgeProb).
Regime detection: Rolling volatility/liquidity metrics classify regimes (fragile, trend, range, calm).
Guardrail engine: Middleware that modulates TA signals, blocks high-risk trades, caps leverage after de-escalations, and emits structured decision logs.
Auditability & governance: Encrypted decision evidence vault with role-gated exports ensures every trade links back to raw sentiment events and executed orders.
Tooling: Annotation CLI, backtest helper, feed connectivity scripts, and staging configuration guide.

Installation

yarn add @developerzava/stock-crypto-sentiment-analyzer
# or
yarn add ../path/to/stock-crypto-sentiment-analyzer/tmp/analyzer.tgz

Environment Variables

Copy .env.example and fill in credentials. Keys that switch on a live feed are commented out in the template, because a placeholder value would register that feed with a dummy credential. Uncomment them only with real values:

# SENTIMENT_TWITTER_BEARER=...        # optional; the feed is skipped with a warning when unset
# SENTIMENT_NEWS_API_KEY=...          # optional; the feed is skipped with a warning when unset
SENTIMENT_NEWS_MAX_AGE_DAYS=2
SENTIMENT_ALPACA_NEWS_ENABLED=false
SENTIMENT_ALPACA_BARS_ENABLED=false
# APCA_API_KEY_ID=...                 # needed when either Alpaca feed is enabled
# APCA_API_SECRET_KEY=...
SENTIMENT_ALPACA_NEWS_LOOKBACK_MS=3600000
SENTIMENT_ALPACA_NEWS_POLL_INTERVAL_MS=60000
SENTIMENT_ALPACA_NEWS_PAGE_LIMIT=50
SENTIMENT_ALPACA_NEWS_MAX_PAGES=10
SENTIMENT_ALPACA_NEWS_MAX_SEEN_IDS=1000
SENTIMENT_ALPACA_BARS_TIMEFRAME=1Min
SENTIMENT_ALPACA_BARS_FEED=sip
SENTIMENT_ALPACA_BARS_ADJUSTMENT=all
SENTIMENT_ALPACA_BARS_LOOKBACK_MS=3600000
SENTIMENT_ALPACA_BARS_END_LAG_MS=60000
SENTIMENT_ALPACA_BARS_POLL_INTERVAL_MS=60000
SENTIMENT_ALPACA_BARS_PAGE_LIMIT=1000
SENTIMENT_ALPACA_BARS_MAX_PAGES=10
SENTIMENT_OFFICIAL_SOCIAL_ENABLED=true
# SENTIMENT_OFFICIAL_SOCIAL_REQUIRED=true   # set true to fail startup when the bearer is missing
# SENTIMENT_MARKET_DATA_URL=https://...
# Optional transformer path, OFF by default (see "What Runs by Default"). SST-2 is a generic
# movie-review polarity model, not a financial model, and is not recommended for trading decisions.
# SENTIMENT_CLASSIFIER_ENABLE=true
# SENTIMENT_CLASSIFIER_MODEL=Xenova/distilbert-base-uncased-finetuned-sst-2-english
# SENTIMENT_SCORING_ENABLE=true
# SENTIMENT_SCORING_MODEL=Xenova/distilbert-base-uncased-finetuned-sst-2-english
# SENTIMENT_ALERT_WEBHOOK=https://discord.com/api/webhooks/<id>/<token>   # Discord payload, not Slack
SENTIMENT_ALERT_MAX_PER_MINUTE=30
SENTIMENT_ALERT_HISTORICAL_LOOKBACK_MINUTES=60
SENTIMENT_ALERT_SUPPRESS_HISTORICAL_SLA=true
SENTIMENT_ALERT_MAX_RETRIES=3
# SENTIMENT_REDIS_URL=redis://127.0.0.1:6379/0   # optional; unset keeps regime state in memory
SENTIMENT_REDIS_DB=0
SENTIMENT_REDIS_PREFIX=sentiment:
# The next four are read by your bot and the repository scripts, not by the library. Pass them to
# DecisionAuditTrail yourself (see Quick Start). Use your own key; do not copy a documented value.
SENTIMENT_STRATEGY_ID=spy-0-dte
SENTIMENT_EVIDENCE_ENCRYPTION_KEY=<64-hex-character-key>
SENTIMENT_EVIDENCE_ROLE=compliance
SENTIMENT_EVIDENCE_PATH=./data/evidence/decisions
SENTIMENT_SYMBOL_SENTIMENT_PATH=./data/symbol-sentiment.json
# Each URL below registers a live feed. Uncomment only with a real endpoint and key.
# SENTIMENT_EARNINGS_API_URL=https://<your-earnings-provider>/earnings
# SENTIMENT_EARNINGS_API_KEY=...
SENTIMENT_EARNINGS_POLL_INTERVAL_MS=300000
# SENTIMENT_GAPPERS_API_URL=https://<your-gappers-provider>/gappers
# SENTIMENT_GAPPERS_API_KEY=...
SENTIMENT_GAPPERS_SESSION=pre
SENTIMENT_GAPPERS_POLL_INTERVAL_MS=60000
SENTIMENT_GAPPERS_MIN_GAP_PCT=3
# SENTIMENT_MACRO_CALENDAR_URL=https://<your-calendar-provider>/macro
# SENTIMENT_MACRO_CALENDAR_API_KEY=...
SENTIMENT_MACRO_CALENDAR_POLL_INTERVAL_MS=300000

When running the overlay in Docker while Redis lives on the host, point SENTIMENT_REDIS_URL at redis://host.docker.internal:6379 (or any hostname reachable from both host and container) and pair it with SENTIMENT_REDIS_DB to pick a DB index that does not collide with other apps. Use SENTIMENT_REDIS_PREFIX to namespace keys (defaults to sentiment:). See docs/staging-config.md for provisioning details and docs/integration.md for full variable descriptions.

Guardrail and risk-profile configuration

Guardrail overrides are passed as an option, not read from the environment. The library does not read a guardrail config path or a risk-profile config path from env variables; only the repository scripts do. Set them in code:

createSentimentOverlay({ overridesPath }) loads a guardrail JSON file. The overlay uses it for the kill switch (killSwitch, vixThreshold) and maxRecentEvents.
GuardrailEngine does not inherit the overlay's file. It uses baseConfig when you pass one and the bundled defaults otherwise, so thresholds such as crashProbCutoff and confidenceFloor only apply to trade decisions if you pass baseConfig.
Load the file once with loadGuardrailConfig(path) and pass the result to both: overridesPath to the overlay and baseConfig to GuardrailEngine.
createSentimentOverlay({ riskProfileConfigPath, riskProfileId }) selects a custom risk-profile file and profile (riskProfileId falls back to SENTIMENT_RISK_PROFILE, then default).
The JSON files shipped under config/ are compiled into dist at build time. They are templates: editing the copies in node_modules has no effect.

What Runs by Default

The default decision path is not a language model. Narrative labels come from taxonomy keywords plus regex heuristics, and the crash/surge scores come from a hand-weighted 24-token lexicon (docs/models/finbert-tariff.md). The lexicon asset is still called finbert-tariff-sentiment, but it uses no FinBERT encoder or embeddings. Its probabilities are not calibrated against market outcomes, and the guardrail cutoffs (0.7 crash, 0.6 surge) are defaults, not findings from outcome data. Accuracy of this path is not measured.

| Configuration | Crash/surge scoring | Narrative label | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Defaults | Lexicon only | Taxonomy match, then regex heuristics | | SENTIMENT_SCORING_ENABLE=true | Lexicon plus a 0.35 blend (SENTIMENT_SCORING_BLEND_WEIGHT) of the named model's P(negative) into crashProb and P(positive) into surgeProb | Unchanged | | SENTIMENT_CLASSIFIER_ENABLE=true | Unchanged | For taxonomy-matched text only, the model's top label decides: positive becomes de-escalation, negative becomes threat. Anything else falls back to the heuristic |

If the blend is on and SENTIMENT_SCORING_MODEL is unset, the default preset (finbert) is used. It cannot be loaded, so scoring stays on the lexicon only. The transformer must be a model you name yourself, and the common example (an SST-2 movie-review model) is a generic polarity model, not a financial one.

A generic polarity model cannot tell a tariff increase from a tariff rollback. Both are mapped by tone, not by what the policy does. Do not read the blended probabilities or the mapped narrative as a measurement of policy direction. See docs/models/transformer-presets.md for details, the cache-directory limitation and licences.

Quick Start

import {
  createSentimentOverlay,
  DecisionAuditTrail,
  GuardrailEngine,
  SentimentGuardrail,
} from '@developerzava/stock-crypto-sentiment-analyzer';
import { createStructuredLogger } from '@developerzava/stock-crypto-sentiment-analyzer/logging';

const logger = createStructuredLogger({ serviceName: 'spy-0-dte' });
const overlay = await createSentimentOverlay({
  env: process.env,
  logger,
  symbols: [
    {
      ticker: 'AAPL',
      names: ['Apple', 'Apple Inc.'],
      sector: 'technology',
      alpacaBars: { assetClass: 'us_equity', providerSymbol: 'AAPL' },
    },
    {
      ticker: 'MSFT',
      names: ['Microsoft'],
      sector: 'technology',
      alpacaBars: { assetClass: 'us_equity', providerSymbol: 'MSFT' },
    },
    {
      ticker: 'NVDA',
      names: ['Nvidia'],
      sector: 'technology',
      alpacaBars: { assetClass: 'us_equity', providerSymbol: 'NVDA' },
    },
  ],
  sectors: [
    {
      id: 'technology',
      displayName: 'Technology',
      benchmarkSymbol: 'XLK',
      benchmarkAlpacaBars: { assetClass: 'us_equity', providerSymbol: 'XLK' },
    },
  ],
  marketPerformance: {
    windowMs: 60 * 60 * 1000,
    maxStalenessMs: 180 * 1000,
    maxTimestampSkewMs: 60 * 1000,
    minPeerCount: 2,
  },
});

// The earnings-beat relief is OFF by default. Opting in (earningsReliefEnabled: true) lets a beat
// reported by any ticker lift crash/confidence blocks for 90 minutes; read docs/integration.md first.
const guardrails = new GuardrailEngine({ overlay, logger });
const encryptionKey = process.env.SENTIMENT_EVIDENCE_ENCRYPTION_KEY;
if (!encryptionKey) throw new Error('SENTIMENT_EVIDENCE_ENCRYPTION_KEY must be set');
const auditTrail = new DecisionAuditTrail({
  strategyId: process.env.SENTIMENT_STRATEGY_ID ?? 'spy-0-dte',
  encryptionKey,
  logger,
});

export async function executeTaSignal(signal: TradeSignal) {
  const decision = await guardrails.evaluate(signal);
  if (!decision.allowed) {
    logger.warn({
      category: LogCategory.DECISION,
      action: 'trade-blocked',
      message: 'Guardrail blocked TA signal',
      details: { reason: decision.reason },
    });
    return;
  }

  const execution = await orderRouter.submit(decision.adjustedSignal);
  await auditTrail.record({
    decision,
    requestedSignal: signal,
    executedOrders: execution.orders, // array of { orderId, symbol, side, quantity, executedAt }
    orderBatchId: execution.batchId,
  });
}

const appleContext = overlay.getSymbolContext('AAPL');
logger.info({
  category: LogCategory.DECISION,
  action: 'symbol-context-snapshot',
  message: 'Latest read-only symbol and sector context',
  details: { symbol: 'AAPL', context: appleContext },
});

The package ships no symbol watchlist. symbols is the authoritative per-symbol universe for that overlay instance. Tickers are normalized case-insensitively and may use non-equity formats such as BRK.B, BTC-USD, or BTC/USD; names and sectors are optional. Omitting symbols keeps global sentiment and regime processing active but intentionally disables per-symbol attribution.

sectors is also consumer-owned. The package never guesses an official sector from a ticker. getSectorSentiment(sector) aggregates each event once per sector—even when the story names several configured constituents—and distinguishes direct sector news from constituent-attributed news. getSymbolContext(symbol) combines symbol sentiment, sector sentiment, and read-only relative performance. These outputs are not connected to GuardrailEngine, position sizing, order routing, or strategy thresholds.

Relative performance requires an explicit marketPerformance contract. Returns use the configured window and aligned samples from the same provider/feed/adjustment/timeframe. The subject is compared with the consumer-configured benchmark and with other configured symbols in the same sector, excluding both the subject and benchmark. Missing, future-dated, stale, misaligned, mixed-provenance, or insufficient data produces an explicit non-available status instead of a directional label. Set minPeerCount to 0 for symbol-versus-benchmark output without requiring peer statistics.

Alpaca integrations are opt-in. Enabling Alpaca news replaces the default NewsAPI registration. Alpaca stock bars are an additional read-only relative-performance source; they do not replace the generic market-data feed or update its global regime/risk path. The stock-bars adapter queries only symbols and benchmarks with an explicit alpacaBars/benchmarkAlpacaBars binding whose assetClass is us_equity, and maps provider symbols back to the configured canonical ticker. Crypto and unbound identifiers never reach /v2/stocks/bars; a crypto-only bars configuration fails before feed registration. Use the generic/custom market-data path for crypto observations. Missing credentials, an eligible US-equity subject, marketPerformance, or required paging/window/feed settings fail overlay initialization. With both flags unset or false, the package makes no Alpaca requests.

The overlay automatically:

Registers feeds from the env you pass in. A missing Twitter/X bearer token or NewsAPI key does not stop startup: that feed is skipped and a warning is logged, and news-feeds-active or no-news-feed-active states which news sources are running. Set SENTIMENT_OFFICIAL_SOCIAL_REQUIRED=true or SENTIMENT_NEWS_API_REQUIRED=true to fail startup instead. Market data, earnings, gapper and macro-calendar feeds register only when their URL is set. The Treasury feed is always registered. Alpaca feeds need their opt-in flags.
Buffers events, classifies narratives, runs sentiment scoring, and emits structured events.
Updates overlay context (crash/surge probability, confidence, regime, recent events).
GuardrailEngine pulls that context for every TA decision.

Runtime compatibility

The package ships both ESM and CJS bundles; asset resolution (tariff lexicon config, taxonomy) uses Node's module resolver via require.resolve(), ensuring reliable path resolution in all installation scenarios (npm/yarn/pnpm, monorepos, Docker, etc.).
Logs must include action, category, and message. The provided logger wrapper will map legacy msg to action/message, but downstream bots should emit structured events directly to avoid validator rejections.

CLI & Scripts

yarn label:events — append labeled rows to data/annotations.csv.
yarn evaluate:keywords — measure per-category taxonomy recall and report false negatives or configured target misses from data/taxonomy-annotations.csv, applying the consumer-style ticker overrides in data/taxonomy-eval-overrides.json (the report also lists generic-only recall without them).
yarn backtest:sentiment — run a simple crash-detection backtest using data/backtest/sample-events.json.
yarn verify:feeds — ensure required environment variables are set.
yarn ping:twitter|news|market — smoke test external APIs.
yarn export:evidence --from=2025-03-01 --to=2025-03-07 --role=compliance — decrypts decision logs for investigations (requires SENTIMENT_EVIDENCE_ENCRYPTION_KEY + authorized role).
yarn symbol:sentiment --symbol=SPY — log the latest crash/surge/confidence snapshots for a consumer-configured ticker from the persisted store.

Development

yarn install
yarn lint
yarn test
yarn build

Publishing (dry-run):

npm publish --dry-run

prepublishOnly runs lint→test→build automatically.

Architecture Overview

Feeds (Twitter/News/Market) -> NarrativeClassifier -> SentimentScoring -> ScoreStore
                                                \-> AlertDispatcher (threat webhooks)
Overlay Context (crashProb, surgeProb, confidence, regime, latest events)
         ^
         |
  RegimeDetector <- MarketDataPoller
         |
GuardrailEngine + SentimentGuardrail -> Adjusted TA Signals -> Order Router

Testing

Unit tests with Vitest live under tests/unit/**. Run yarn test for the full suite.
Mock overlay (createMockOverlay) and GuardrailEngine tests ensure risk rules work without live feeds.

Documentation

docs/integration.md — installation, env vars, guardrail usage, logging/metrics.
docs/staging-config.md — credential provisioning checklist.
internal/stories/user-stories.md (repo-only) — roadmap and status of major features.
docs/data-lineage.md / docs/security/data-access.md — evidence lineage diagram plus access-control policy for decision logs.

License

Licensing note

The package itself is MIT. It ships no model weights. If you enable the optional transformer path, the weights are downloaded at runtime and each model keeps its own licence. FinBERT (ProsusAI/finbert) is licensed CC BY-NC-SA 4.0, which is non-commercial. Earlier documentation called it Apache-2.0, which was wrong. Check the licence of any model you load before using it in a commercial product. See docs/models/transformer-presets.md.