Dromanis Finora Market Data Common

A comprehensive TypeScript library for fetching, processing, and caching financial market data from multiple sources including Yahoo Finance and AlphaVantage APIs.

🚀 Features

Core Processors

• 🔵 InstrumentPriceProcessor - Real-time stock prices with OHLCV data
• 🟡 InstrumentRatingsProcessor - Analyst ratings and price targets
• 🔴 InstrumentRiskProfileProcessor - Advanced risk analysis (beta, volatility, Sharpe ratio)
• 🟣 InstrumentDowngradeUpgradeProcessor - Analyst upgrade/downgrade history
• 🔵 InstrumentFinancialMetricProcessor - Financial statements and metrics
• 🟠 FxRatesProcessor - Foreign exchange rates for 80+ currencies
• 📰 MarketNewsAndTrendsProcessor - Market news sentiment and trends
• 🟢 MarketDataUtility - System/user instrument discovery utilities

Key Features

• ✅ Comprehensive Market Coverage - 8 specialized processors for different data types
• ✅ Multi-Source Integration - Yahoo Finance + AlphaVantage APIs
• ✅ Intelligent Caching - Database-backed caching with configurable cache usage
• ✅ Robust Error Handling - Graceful degradation and detailed error reporting
• ✅ TypeScript First - Full type safety with comprehensive type definitions
• ✅ 100% Test Coverage - 126 unit tests across all processors
• ✅ Enterprise Ready - Production-grade reliability and performance
• ✅ Flexible Configuration - Environment-based API key management

📦 Installation

npm install dromanis.finora.functions.market-data.common

Dependencies

# Core dependencies (automatically installed)
npm install dromanis.finora.types dromanis.finora.common dromanis.finora.db
npm install axios yahoo-finance2

# Development dependencies
npm install --save-dev typescript @types/node jest ts-jest

🔧 Setup

Environment Variables

# Required for AlphaVantage-based processors
export ALPHA_VANTAGE_API_KEY=your_alphavantage_api_key

# Optional: Database connection settings (if using custom database)
export DB_HOST=localhost
export DB_PORT=3306
export DB_NAME=market_data
export DB_USER=username
export DB_PASSWORD=password

Database Schema

The library requires specific database tables. See the Database Schema section for complete table definitions.

🚀 Quick Start

import {
  InstrumentPriceProcessor,
  InstrumentRatingsProcessor,
  MarketNewsAndTrendsProcessor,
  FxRatesProcessor
} from 'dromanis.finora.functions.market-data.common';

// Set up environment
process.env['ALPHA_VANTAGE_API_KEY'] = 'your-api-key';

// Initialize processors
const priceProcessor = new InstrumentPriceProcessor();
const newsProcessor = new MarketNewsAndTrendsProcessor();

// Fetch real-time prices
const prices = await priceProcessor.getLivePrices(['AAPL', 'GOOGL'], true);
for (const [symbol, result] of prices) {
  if (!(result instanceof Error)) {
    console.log(`${symbol}: $${result.price}`);
  }
}

// Fetch market news and trends
const marketData = await newsProcessor.getLatestNewsAndTrends(true);
if (!(marketData instanceof Error)) {
  console.log('News available:', !!marketData.news);
  console.log('Trends available:', !!marketData.trends);
}

📚 API Documentation

InstrumentPriceProcessor

Fetches real-time stock prices with comprehensive OHLCV data.

const processor = new InstrumentPriceProcessor();

// Get live prices for multiple instruments
const results = await processor.getLivePrices(
  ['AAPL', 'GOOGL', 'MSFT'], // Instrument IDs (symbols)
  true                        // Use cache if available
);

// Process results
for (const [instrumentId, result] of results) {
  if (result instanceof Error) {
    console.error(`Error for ${instrumentId}:`, result.message);
  } else {
    // result is PricePoint
    console.log(`${instrumentId}: $${result.price} (${result.volume} volume)`);
  }
}

PricePoint Interface:

interface PricePoint {
  instrumentId: string;
  price: number;
  open: number;
  high: number;
  low: number;
  close: number;
  volume: number;
  preMarketChange: number;
  preMarketChangePercent: number;
  sharesOutstanding: number;
  marketCap: number;
  epsForward: number;
  forwardPE: number;
  asOfDateTime: string;
}

InstrumentRatingsProcessor

Fetches analyst ratings and price targets.

const processor = new InstrumentRatingsProcessor();

const results = await processor.getLiveRatings(['AAPL', 'GOOGL'], true);

for (const [instrumentId, result] of results) {
  if (!(result instanceof Error)) {
    console.log(`${instrumentId} Rating:`);
    console.log(`  Strong Buy: ${result.strongBuy}`);
    console.log(`  Price Target: $${result.priceTargetMean}`);
    console.log(`  Total Analysts: ${result.totalAnalysts}`);
  }
}

InstrumentRiskProfileProcessor

Calculates advanced risk metrics including beta, volatility, and Sharpe ratio.

const processor = new InstrumentRiskProfileProcessor();

const results = await processor.getLiveRiskProfiles(['AAPL', 'GOOGL'], true);

for (const [instrumentId, result] of results) {
  if (!(result instanceof Error)) {
    console.log(`${instrumentId} Risk Profile:`);
    console.log(`  Beta: ${result.beta}`);
    console.log(`  Volatility: ${(result.volatility * 100).toFixed(2)}%`);
    console.log(`  Risk Level: ${result.riskLevel}`);
    console.log(`  Sharpe Ratio: ${result.sharpeRatio}`);
  }
}

InstrumentDowngradeUpgradeProcessor

Fetches analyst upgrade/downgrade history.

const processor = new InstrumentDowngradeUpgradeProcessor();

const results = await processor.getLiveUpgradeDowngrades(['AAPL', 'GOOGL'], true);

for (const [instrumentId, result] of results) {
  if (!(result instanceof Error)) {
    console.log(`${instrumentId} has ${result.length} upgrade/downgrade records`);
    
    result.forEach(upgrade => {
      console.log(`  ${upgrade.firm}: ${upgrade.fromGrade} → ${upgrade.toGrade}`);
      console.log(`  Action: ${upgrade.action} on ${upgrade.gradeDate}`);
    });
  }
}

InstrumentFinancialMetricProcessor

Fetches comprehensive financial metrics from income statements, balance sheets, and cash flow statements.

// Requires ALPHA_VANTAGE_API_KEY environment variable
const processor = new InstrumentFinancialMetricProcessor();

const results = await processor.getLiveFinancialMetrics(['AAPL', 'GOOGL'], true);

for (const [instrumentId, result] of results) {
  if (!(result instanceof Error)) {
    console.log(`${instrumentId} has ${result.length} financial metrics`);
    
    // Group by statement type
    const income = result.filter(m => m.metric.startsWith('income_'));
    const balance = result.filter(m => m.metric.startsWith('balance_'));
    const cashflow = result.filter(m => m.metric.startsWith('cashflow_'));
    
    console.log(`  Income Statement: ${income.length} metrics`);
    console.log(`  Balance Sheet: ${balance.length} metrics`);
    console.log(`  Cash Flow: ${cashflow.length} metrics`);
  }
}

FxRatesProcessor

Fetches foreign exchange rates for 80+ major world currencies against USD.

const processor = new FxRatesProcessor();

// No currency pairs needed - uses internal list of 80 currencies
const results = await processor.getLiveFxRates(true);

console.log(`Retrieved ${results.size} FX rates`);

// Check specific currency pairs
const eurRate = results.get('USDEUR');
const gbpRate = results.get('USDGBP');

if (!(eurRate instanceof Error)) {
  console.log(`USD to EUR: ${eurRate.rate}`);
}

MarketNewsAndTrendsProcessor

Fetches market-wide news sentiment and trend data (gainers/losers).

// Requires ALPHA_VANTAGE_API_KEY environment variable
const processor = new MarketNewsAndTrendsProcessor();

const result = await processor.getLatestNewsAndTrends(true);

if (!(result instanceof Error)) {
  if (result.news) {
    const newsData = JSON.parse(result.news.data);
    console.log(`News feed: ${newsData.feed?.length || 0} articles`);
  }
  
  if (result.trends) {
    const trendsData = JSON.parse(result.trends.data);
    console.log(`Top gainers: ${trendsData.top_gainers?.length || 0}`);
    console.log(`Top losers: ${trendsData.top_losers?.length || 0}`);
  }
}

MarketDataUtility

Utility functions for discovering instruments in the system.

const utility = new MarketDataUtility();

// Get all unique instrument IDs in the system
const systemInstruments = await utility.getSystemUniqueInstrumentIds();
console.log(`System has ${systemInstruments.length} unique instruments`);

// Get instruments for a specific user
const userInstruments = await utility.getUserUniqueInstrumentIds('user123');
console.log(`User has ${userInstruments.length} unique instruments`);

🔄 Caching Strategy

All processors support intelligent caching:

• Cache Hit: Returns cached data immediately
• Cache Miss: Fetches fresh data from APIs and stores in database
• Cache Control: Use useCache: boolean parameter to control caching behavior
• Cache Invalidation: Fresh API calls automatically update cached data

// Always use cache if available (recommended for production)
const cachedResults = await processor.getLivePrices(['AAPL'], true);

// Always fetch fresh data (useful for critical real-time applications)
const freshResults = await processor.getLivePrices(['AAPL'], false);

🧪 Testing

Running Tests

# Run all tests
npm test

# Run tests with coverage
npm run test:coverage

# Run tests in watch mode
npm run test:watch

# Build and test
npm run build && npm test

Test Coverage

• Test Suites: 8 (one per processor)
• Total Tests: 126
• Coverage: 100%
• Test Types: Unit tests, integration tests, error handling tests

Test Structure

src/__tests__/
└── market-data/
    ├── InstrumentPriceProcessor.test.ts           (13 tests)
    ├── InstrumentRatingsProcessor.test.ts         (15 tests)
    ├── InstrumentRiskProfileProcessor.test.ts     (16 tests)
    ├── InstrumentDowngradeUpgradeProcessor.test.ts (16 tests)
    ├── InstrumentFinancialMetricProcessor.test.ts (19 tests)
    ├── FxRatesProcessor.test.ts                   (11 tests)
    ├── MarketNewsAndTrendsProcessor.test.ts       (18 tests)
    └── MarketDataUtility.test.ts                  (18 tests)

🗄️ Database Schema

Core Tables

instrument_market_data_prices

CREATE TABLE `instrument_market_data_prices` (
  `asOfDateTime` varchar(255) NOT NULL,
  `instrumentId` varchar(255) NOT NULL,
  `price` double NOT NULL,
  `open` double NOT NULL,
  `high` double NOT NULL,
  `low` double NOT NULL,
  `volume` double NOT NULL,
  `close` double NOT NULL,
  `preMarketChange` double NOT NULL,
  `preMarketChangePercent` double NOT NULL,
  `sharesOutstanding` double NOT NULL,
  `marketCap` double NOT NULL,
  `epsForward` double NOT NULL,
  `forwardPE` double NOT NULL,
  PRIMARY KEY (`instrumentId`)
);

instrument_market_ratings

CREATE TABLE `instrument_market_ratings` (
  `instrumentId` varchar(255) NOT NULL,
  `averageRating` double DEFAULT NULL,
  `strongBuy` int DEFAULT '0',
  `buy` int DEFAULT '0',
  `hold` int DEFAULT '0',
  `sell` int DEFAULT '0',
  `strongSell` int DEFAULT '0',
  `totalAnalysts` int DEFAULT '0',
  `priceTarget` double DEFAULT NULL,
  `priceTargetHigh` double DEFAULT NULL,
  `priceTargetLow` double DEFAULT NULL,
  `priceTargetMean` double DEFAULT NULL,
  `priceTargetMedian` double DEFAULT NULL,
  `recommendationTrend` varchar(255) DEFAULT NULL,
  `asOfDateTime` varchar(255) NOT NULL,
  PRIMARY KEY (`instrumentId`)
);

instrument_market_risk_profile

CREATE TABLE `instrument_market_risk_profile` (
  `instrumentId` varchar(255) NOT NULL,
  `beta` double DEFAULT NULL,
  `volatility` double DEFAULT NULL,
  `sharpe_ratio` double DEFAULT NULL,
  `risk_score` int DEFAULT NULL,
  `risk_level` varchar(50) DEFAULT NULL,
  `asOfDateTime` varchar(255) NOT NULL,
  PRIMARY KEY (`instrumentId`)
);

instrument_market_downgrade_upgrade_profile

CREATE TABLE `instrument_market_downgrade_upgrade_profile` (
  `instrumentId` varchar(255) NOT NULL,
  `firm` varchar(255) NOT NULL,
  `toGrade` varchar(255) DEFAULT NULL,
  `fromGrade` varchar(255) DEFAULT NULL,
  `action` varchar(255) NOT NULL,
  `epochGradeDate` bigint DEFAULT NULL,
  `gradeDate` varchar(255) DEFAULT NULL,
  `priceTarget` double DEFAULT NULL,
  `priceTargetChange` double DEFAULT NULL,
  `asOfDateTime` varchar(255) NOT NULL,
  PRIMARY KEY (`instrumentId`,`firm`)
);

instrument_market_financial_metrics

CREATE TABLE `instrument_market_financial_metrics` (
  `instrumentId` varchar(255) NOT NULL,
  `metric` varchar(255) NOT NULL,
  `year` varchar(255) NOT NULL,
  `value` double DEFAULT NULL,
  `asOfDateTime` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`instrumentId`,`metric`,`year`)
);

market_data_fx_rates

CREATE TABLE `market_data_fx_rates` (
  `asOfDateTime` varchar(255) NOT NULL,
  `baseCurrency` varchar(255) NOT NULL,
  `toCurrency` varchar(255) NOT NULL,
  `rate` double NOT NULL,
  PRIMARY KEY (`baseCurrency`,`toCurrency`)
);

market_data_news

CREATE TABLE `market_data_news` (
  `asOfDateTime` varchar(255) NOT NULL,
  `data` longtext
);

market_data_trends

CREATE TABLE `market_data_trends` (
  `asOfDateTime` varchar(255) NOT NULL,
  `data` longtext
);

trade_transactions (for utility functions)

CREATE TABLE `trade_transactions` (
  `id` varchar(255) NOT NULL,
  `instrumentId` varchar(255) NOT NULL,
  `direction` varchar(255) NOT NULL,
  `quantity` double NOT NULL,
  `price` double NOT NULL,
  `exchangeRateToPortfolioCurrency` double NOT NULL,
  `transactionDateTime` varchar(255) NOT NULL,
  `transactionFee` double NOT NULL,
  `portfolioId` varchar(255) NOT NULL,
  `brokerId` varchar(255) DEFAULT NULL,
  `brokerTransactionId` varchar(255) DEFAULT NULL,
  `notes` text,
  `userId` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
);

🏗️ Project Structure

dromanis.finora.functions.market-data.common/
├── src/
│   ├── index.ts                                 # Main exports
│   ├── market-data/                            # Core processors
│   │   ├── InstrumentPriceProcessor.ts         # Real-time prices
│   │   ├── InstrumentRatingsProcessor.ts       # Analyst ratings
│   │   ├── InstrumentRiskProfileProcessor.ts   # Risk analysis
│   │   ├── InstrumentDowngradeUpgradeProcessor.ts # Upgrades/downgrades
│   │   ├── InstrumentFinancialMetricProcessor.ts # Financial metrics
│   │   ├── FxRatesProcessor.ts                 # Foreign exchange
│   │   ├── MarketNewsAndTrendsProcessor.ts     # News & trends
│   │   └── MarketDataUtility.ts               # Utility functions
│   └── __tests__/                             # Test suite
│       └── market-data/                       # Processor tests
├── dist/                                      # Compiled JavaScript
├── package.json                              # Dependencies & scripts
├── tsconfig.json                             # TypeScript config
├── jest.config.js                           # Jest test config
├── .husky/                                   # Git hooks
└── README.md                                 # This file

🔧 Development

Prerequisites

• Node.js 18+
• TypeScript 5.0+
• MySQL/MariaDB database
• AlphaVantage API key

Setup Development Environment

# Clone repository
git clone https://github.com/tariqkarim/dromanis.finora.functions.market-data.common.git
cd dromanis.finora.functions.market-data.common

# Install dependencies
npm install

# Set up environment variables
cp .env.example .env
# Edit .env with your API keys and database settings

# Run tests
npm test

# Build project
npm run build

# Start development
npm run dev

Available Scripts

npm run build      # Compile TypeScript to JavaScript
npm run dev        # Start in development mode with hot reload
npm run test       # Run all tests
npm run clean      # Clean build artifacts
npm start          # Start compiled application
npm run prepare    # Set up Husky git hooks

Git Hooks

The project uses Husky for git hooks:

• pre-commit: Runs build and tests
• pre-push: Runs tests

📊 Performance Considerations

API Rate Limits

• Yahoo Finance: No explicit rate limits, but implement delays between requests
• AlphaVantage: 25 requests/day (free tier), 500-1200 requests/minute (paid tiers)

Optimization Strategies

1. Intelligent Caching: Use cache for frequently accessed data
2. Batch Processing: Process multiple instruments in single requests where possible
3. Error Recovery: Graceful degradation when APIs are unavailable
4. Connection Pooling: Database connections are properly managed
5. Memory Management: Large datasets are processed in streams

Production Deployment

# Build for production
npm run build

# Set production environment variables
export NODE_ENV=production
export ALPHA_VANTAGE_API_KEY=your_production_key

# Start application
npm start

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Development Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass (npm test)
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Code Standards

• TypeScript: Strict mode enabled
• Testing: 100% test coverage required
• Linting: ESLint with strict rules
• Formatting: Prettier for consistent code style
• Documentation: Comprehensive JSDoc comments

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Yahoo Finance API for real-time market data
• AlphaVantage API for financial metrics and news data
• Dromanis Finora platform ecosystem
• Open source community for inspiration and tools

📞 Support

• Documentation: This README and inline code comments
• Issues: GitHub Issues
• Discussions: GitHub Discussions

🔮 Roadmap

Upcoming Features

• [ ] WebSocket support for real-time streaming
• [ ] Additional data sources (IEX Cloud, Polygon.io)
• [ ] Advanced analytics and technical indicators
• [ ] GraphQL API interface
• [ ] Redis caching layer
• [ ] Kubernetes deployment manifests

Version History

• v1.0.5 - Current version with 8 processors and 126 tests
• v1.0.0 - Initial release with core functionality

Built with ❤️ by the Dromanis Finora Team

For more information about the Dromanis Finora ecosystem, visit dromanis.finora.com.