last-redoc-date: 2025-10-08

Time Series Forecasting Specification Framework

BMAD Module Code: ts-forecast-spec Version: 1.0.0-alpha.1 Status: 🎉 Phase 1 Complete - Production Ready Author: Philippe Béliveau (Newcode) License: MIT

📥 Installation & Quick Start

NPM Installation (Recommended)

# Install globally
npm install -g @new-code/spec-forecasting

# Or use with npx (no installation)
npx @new-code/spec-forecasting

# Or install in your BMAD project
cd your-bmad-project
npm install @new-code/spec-forecasting

Manual Installation (BMAD Integration)

👉 Installation Guide - Full installation instructions 👉 Quick Start Guide - Get started in 5 minutes 👉 Roadmap - See what's built and what's coming

TL;DR Installation:

1. Add ts-forecast-spec to bmad/_cfg/manifest.yaml
2. Add 3 agents to bmad/_cfg/agent-manifest.csv
3. Run: bmad business-context-architect

See _module-installer/MANUAL_INSTALL.md for step-by-step instructions.

🎯 Mission

Prevent forecasting failures by forcing rigorous problem specification BEFORE any modeling begins.

Most forecasting projects fail not because of bad models, but because teams skip fundamental diagnostic questions and specification steps. This BMAD module provides a framework of specialized AI agents that act as "blocking gatekeepers" — refusing to let you proceed until critical diagnostics are addressed.

Core Philosophy:

"Most forecasting failures stem from specification problems, not model problems." — Inspired by Valeriy Manokhin's "5% models, 95% everything else"

💥 The Story: The MAPE Disaster

Real Corporate Failure (Valeriy Manokhin's Case Study):

Two forecasting projects at a large corporation optimized for MAPE (Mean Absolute Percentage Error) without understanding its destructiveness for their data. The result:

• Teams had a "contract" to deliver the best possible MAPE
• Expensive models performed well on MAPE... but failed in production
• Customer service collapsed
• Billion-dollar clients walked away

The Problem: MAPE explodes when actual values are near zero (division by small numbers). The models optimized to avoid zeros, not to make good business decisions.

This module prevents this exact disaster — and many others like it.

🏗️ Module Architecture

✅ Phase 1: Built & Ready (3 Agents, 2 Workflows, 4 Tasks)

Phase 1 is complete and production-ready. The following components are fully implemented and tested:

🤖 Built Agents (Phase 1)

1. Business Context Architect ✅ (Elena 🎯)

• Role: Frame forecasting problem in business terms FIRST
• Personality: Strategic, pragmatic, asks "so what?" relentlessly
• Blocks Until:
  • You articulate what decision this forecast drives
  • You define cost of over-forecasting vs under-forecasting
  • You identify stakeholders and their uncertainty tolerance
• File: agents/business-context-architect.md
• Usage: bmad business-context-architect

2. Metrics Guardian ✅ (Marcus 🛡️)

• Role: Prevent metric disasters and align evaluation with business reality
• Personality: Battle-scarred, skeptical, references real failures (MAPE disaster)
• Blocks Until:
  • MAPE challenged if near-zero data detected
  • Metric reflects business cost function
  • Asymmetric costs addressed if relevant
  • Point forecasts vs prediction intervals decided
• Famous Reference: The billion-dollar MAPE disaster
• File: agents/metrics-guardian.md
• Usage: bmad metrics-guardian

3. Data Diagnostician ✅ (Dr. Sarah Chen 🔬)

• Role: Force rigorous statistical analysis of data characteristics
• Personality: Obsessed with fundamentals, refuses shortcuts, speaks in tests (ADF, KPSS, ACF)
• Blocks Until:
  • Stationarity checked with evidence (ADF, KPSS, PP tests)
  • Trend, seasonality, cyclical components identified
  • Autocorrelation structure assessed
• Will NOT discuss models until data characteristics understood
• File: agents/data-diagnostician.md
• Usage: bmad data-diagnostician

📋 Built Workflows (Phase 1)

1. Problem Specification ✅ Lead: Business Context Architect (Elena) Output: 790-line Problem Specification Document with business context, stakeholders, cost asymmetry, viability Path: workflows/problem-specification/

2. Metric Selection ✅ Lead: Metrics Guardian (Marcus) Output: Metric Justification Document with primary metric, supporting metrics, avoided metrics Path: workflows/metric-selection/

🔧 Built Tasks (Phase 1)

All 4 core tasks are fully implemented:

1. Conformal Prediction Validator ✅ - tasks/conformal-prediction-validator.xml
2. Calibration Monitor ✅ - tasks/calibration-monitor.xml
3. Uncertainty Diagnostics ✅ - tasks/uncertainty-diagnostics.xml
4. Antipattern Detector ✅ - tasks/antipattern-detector.xml

🔮 Phase 2+: Coming Soon (4 Agents, 4 Workflows)

The following components are planned for future releases (see ROADMAP.md):

Agents (Phase 2+)

• Signal Architect - Feature engineering and signal design
• Missing Data & Anomaly Specialist - MCAR/MAR/MNAR analysis
• Method Advisor - Model selection with NIXTLA ecosystem
• Validation & Deployment Strategist - Production readiness

Workflows (Phase 2+)

• Data Diagnostics & Quality - Full data diagnostic workflow
• Signal Design & Feature Engineering
• Model Selection & Approach
• Validation & Deployment Strategy

🔧 Core Tasks (From Valeriy Manokhin's Methodology)

Tasks are reusable diagnostic utilities that agents invoke. All 4 tasks are fully implemented.

1. Conformal Prediction Validator ✅

• Purpose: Validates CP implementation, checks coverage guarantees
• Validates:
  • Prediction intervals present (not just point estimates)
  • Empirical coverage matches stated confidence (95% intervals contain 95% actuals)
  • Calibration set exists and sized appropriately
  • CP method appropriate for time series (EnbPI, adaptive methods)
• Blocks If: No intervals, coverage <85% for 95% PI, no calibration set
• Reference: Valeriy Batch 1 & 3 (Conformal Prediction for Time Series)

2. Calibration Monitor ✅

• Purpose: Monitors calibration of probabilistic forecasts
• Calculates:
  • ECE (Expected Calibration Error): Predicted confidence vs empirical accuracy
  • Log Loss: Penalizes confident wrong predictions
  • Brier Score: Decomposed into calibration + refinement loss
• Detects: Calibration drift over time (distribution shift warning)
• Blocks If: ECE >0.15, Brier >0.30, severe drift detected
• Reference: Valeriy Batch 4 (Advanced Topics - Calibration)

3. Uncertainty Diagnostics ✅

• Purpose: Distinguish aleatoric (irreducible) vs epistemic (reducible) uncertainty
• Decomposes: Total uncertainty into components
• Strategic Guidance:
  • Aleatoric-dominated: Accept and manage (robust planning)
  • Epistemic-dominated: Reduce through better data/models
• Blocks If: Epistemic >80% at high-stakes + long horizons
• Reference: Valeriy Batch 1 (Foundation - Uncertainty Types)

4. Antipattern Detector ✅

• Purpose: Detect forecasting antipatterns that cause real-world disasters
• Detects:
  • MAPE on near-zero data (the billion-dollar disaster)
  • SMOTE + Conformal Prediction (destroys coverage guarantees)
  • Metric-business misalignment (symmetric metrics, asymmetric costs)
  • Ignored stationarity (ARIMA without testing)
  • Point forecasts only (no uncertainty quantification)
  • Temporal leakage (random splits on time series)
• Blocks If: Critical antipatterns detected (MAPE+zeros, SMOTE+CP, random splits)
• Reference: Valeriy MAPE case study + Batch 4 (SMOTE antipattern)

🚀 Quick Start

See QUICKSTART.md for a complete 5-minute getting started guide.

Installation (2 minutes)

Full Instructions: INSTALL.md

Quick Install:

1. Edit bmad/_cfg/manifest.yaml - add ts-forecast-spec to modules list
2. Edit bmad/_cfg/agent-manifest.csv - add 3 agent entries (see _module-installer/MANUAL_INSTALL.md)
3. Test: bmad business-context-architect

Your First Workflow (3 minutes)

Scenario: You need to forecast daily sales for inventory planning

# Step 1: Activate Business Context Architect
bmad business-context-architect

# Step 2: Select Problem Specification Workflow
> 1  # or type: problem-spec

# Step 3: Answer Elena's diagnostic questions
Elena will guide you through:
- What decision does this forecast drive?
- Who makes the decision?
- What happens if forecast is wrong?
- Cost asymmetry analysis (over vs under-forecast)
- Stakeholder mapping
- Viability checks

# Output: Problem Specification Document (790 lines)
# Saved to: output/forecasting/problems/problem-specification-{date}.md

Challenge Your Metrics (2 minutes)

# Activate Metrics Guardian
bmad metrics-guardian

# Quick MAPE disaster check
> 2  # or type: mape-check

# Marcus will ask about near-zero values
# If detected: 🚨 CRITICAL WARNING - MAPE is BLOCKED
# Recommendation: Switch to MAE or sMAPE

# Full metric selection workflow
> 1  # or type: metric-selection

# Output: Metric Justification Document
# Includes: Primary metric, supporting metrics, avoided metrics

Diagnose Your Data (2 minutes)

# Activate Data Diagnostician
bmad data-diagnostician

# Check stationarity
> 1  # or type: stationarity

# Dr. Sarah Chen will guide you through:
# - ADF and KPSS test interpretation
# - Differencing recommendations
# - Implications for model selection

# Other options:
> 2  # Decomposition (trend, seasonality)
> 3  # ACF/PACF analysis
> 4  # Missing data classification
> 5  # Signal-to-noise analysis

Recommended Workflow Sequence

1. Problem Specification (Business Context Architect) → Outputs cost asymmetry
2. Data Diagnostics (Data Diagnostician - consulting mode) → Outputs data characteristics
3. Metric Selection (Metrics Guardian) → Uses outputs from #1 and #2
4. (Phase 2) Model Selection → Coming soon

📊 Real-World Failure Patterns Prevented

1. The MAPE Disaster

Symptom: Optimizing MAPE with near-zero values Prevention: Antipattern Detector + Metrics Guardian block MAPE usage with zeros Outcome: Billion-dollar client loss avoided

2. The False Precision Trap

Symptom: Point forecasts without uncertainty for high-stakes decisions Prevention: Metrics Guardian mandates prediction intervals Outcome: Risk-aware decision making, contingency planning enabled

3. The Data Leakage Disaster

Symptom: Random train/test split on time series (future trains the model) Prevention: Antipattern Detector catches temporal violations Outcome: Valid performance metrics, production success

4. The SMOTE Coverage Violation

Symptom: Synthetic oversampling destroys conformal prediction guarantees Prevention: Antipattern Detector blocks SMOTE + CP combination Outcome: Mathematically valid prediction intervals

5. The Ignored Stationarity Sin

Symptom: ARIMA on trending data without differencing (spurious regression) Prevention: Data Diagnostician mandates stationarity tests Outcome: Reliable statistical forecasts

6. The Metric Misalignment Failure

Symptom: Symmetric metrics (RMSE) with asymmetric costs (stockout ≠ overstock) Prevention: Metrics Guardian forces business cost function alignment Outcome: Optimized for real business decisions, not academic metrics

🧠 Methodology Foundation

Valeriy Manokhin's Principles

This module is built on the rigorous forecasting methodology of Valeriy Manokhin (PhD Machine Learning, "Black Belt 🥋 in Time Series"):

Key Insights:

• The 5% Rule: Only 5% of success is about models. 95% is data, metrics, stakeholder alignment, and specification.
• Conformal Prediction Advocacy: Distribution-free uncertainty estimates that work with data "as it really is"
• Failure Analysis: Learn from real disasters (MAPE case study)
• Academic Rigor: PhD-level discipline applied to real-world problems

Referenced Works:

• "Mastering Modern Time Series Forecasting"
• "Applied Conformal Prediction"
• "Mastering Forecasting Metrics & Accuracy"
• LinkedIn posts on forecasting methodology (300+ likes, engaged audience)

Philippe's Experience Integration

Real-World Context from:

• Videns Analytics: Built 3 cash flow forecasting approaches (statistical + MDP) for corporate treasurers at daily/weekly horizons. Key challenge: distinguishing signal from noise at granular time scales.
• CroesusLab: Developed GANs for synthetic relational databases, focusing on preserving temporal correlation structures.
• Groupe Azur: Led user discovery with corporate treasurers, translating business needs to data science teams.

Newcode Philosophy:

"If you learn to ask the right question, then you can build anything with AI agents."

📁 Module Structure

bmad/ts-forecast-spec/
├── agents/                          # ✅ 3/7 agents (Phase 1 complete)
│   ├── business-context-architect.md     # ✅ Elena 🎯
│   ├── metrics-guardian.md               # ✅ Marcus 🛡️
│   ├── data-diagnostician.md             # ✅ Dr. Sarah Chen 🔬
│   ├── signal-architect.md               # 📋 Phase 2
│   ├── missing-data-specialist.md        # 📋 Phase 2
│   ├── method-advisor.md                 # 📋 Phase 2
│   └── validation-strategist.md          # 📋 Phase 2
│
├── workflows/                       # ✅ 2/6 workflows (Phase 1 complete)
│   ├── problem-specification/            # ✅ Full workflow
│   │   ├── workflow.yaml
│   │   ├── instructions.md
│   │   ├── template.md
│   │   └── checklist.md
│   ├── metric-selection/                 # ✅ Full workflow
│   │   ├── workflow.yaml
│   │   ├── instructions.md
│   │   ├── template.md
│   │   └── checklist.md
│   ├── data-diagnostics/                 # 📋 Phase 2
│   ├── signal-design/                    # 📋 Phase 2
│   ├── model-selection/                  # 📋 Phase 2
│   └── validation-deployment/            # 📋 Phase 2
│
├── tasks/                           # ✅ 4/4 core tasks (ALL IMPLEMENTED)
│   ├── conformal-prediction-validator.xml
│   ├── calibration-monitor.xml
│   ├── uncertainty-diagnostics.xml
│   └── antipattern-detector.xml
│
├── _module-installer/               # ✅ Installation infrastructure
│   ├── install.py                        # Python installer
│   ├── install.sh                        # Shell installer
│   ├── MANUAL_INSTALL.md                 # Manual installation guide
│   └── README.md
│
├── config.yaml                      # ✅ Module configuration
├── README.md                        # ✅ This file
├── INSTALL.md                       # ✅ Full installation guide
├── QUICKSTART.md                    # ✅ 5-minute getting started
└── ROADMAP.md                       # ✅ Development roadmap

🛠️ Development Status

✅ Phase 1 Complete (Production Ready)

Delivered:

• Agents (3/3 Phase 1): ✅ ALL BUILT
  • Business Context Architect (Elena 🎯)
  • Metrics Guardian (Marcus 🛡️)
  • Data Diagnostician (Dr. Sarah Chen 🔬)
• Workflows (2/2 Phase 1): ✅ ALL BUILT
  • Problem Specification (9 steps, 790-line output)
  • Metric Selection (8 steps, comprehensive justification)
• Core tasks (4/4): ✅ ALL IMPLEMENTED
  • Conformal Prediction Validator
  • Calibration Monitor
  • Uncertainty Diagnostics
  • Antipattern Detector
• Module infrastructure: ✅ COMPLETE
  • Configuration system
  • Installation system (manual + automated)
  • Complete documentation

Total Delivered: ~7,500 lines of code and documentation

📋 Phase 2+ Planned

See ROADMAP.md for detailed plans:

• Phase 2: Diagnostic Infrastructure (2-3 weeks)

  • Signal Architect, Missing Data Specialist
  • Data Diagnostics, Signal Design workflows

• Phase 3: Complete Framework (2-4 weeks)

  • Method Advisor, Validation Strategist
  • Model Selection, Validation & Deployment workflows

📚 Documentation

Module Documentation

| Document | Description | |----------|-------------| | README.md | This file - Module overview and architecture | | INSTALL.md | Complete installation guide with troubleshooting | | QUICKSTART.md | 5-minute getting started guide with examples | | ROADMAP.md | Development roadmap and phase planning | | config.yaml | Module configuration file |

Installation Documentation

| Document | Description | |----------|-------------| | _module-installer/MANUAL_INSTALL.md | Step-by-step manual installation (recommended) | | _module-installer/README.md | Installer directory overview | | _module-installer/install.py | Automated Python installer |

Agent Documentation

| Agent | File | Description | |-------|------|-------------| | Elena 🎯 | agents/business-context-architect.md | Problem framing specialist | | Marcus 🛡️ | agents/metrics-guardian.md | Metric selection expert | | Dr. Sarah Chen 🔬 | agents/data-diagnostician.md | Data diagnostics specialist |

Workflow Documentation

| Workflow | Path | Description | |----------|------|-------------| | Problem Specification | workflows/problem-specification/ | Business context and viability | | Metric Selection | workflows/metric-selection/ | Metric justification and selection |

Each workflow directory contains:

• workflow.yaml - Configuration
• instructions.md - Execution logic
• template.md - Output template
• checklist.md - Validation checklist

Task Documentation

| Task | File | Purpose | |------|------|---------| | Antipattern Detector | tasks/antipattern-detector.xml | Detect forecasting antipatterns | | Calibration Monitor | tasks/calibration-monitor.xml | Monitor probabilistic calibration | | Conformal Prediction Validator | tasks/conformal-prediction-validator.xml | Validate CP coverage | | Uncertainty Diagnostics | tasks/uncertainty-diagnostics.xml | Decompose uncertainty types |

🎓 Learning Resources

For Understanding This Module

1. Start Here: QUICKSTART.md Get up and running in 5 minutes with example scenarios

2. Deep Dive: Explore agent files

   • business-context-architect.md - See Elena's personality and philosophy
   • metrics-guardian.md - Learn from Marcus's MAPE disaster story
   • data-diagnostician.md - Understand Dr. Sarah's diagnostic toolkit

3. Workflow Details: Read workflow instructions

   • workflows/problem-specification/instructions.md
   • workflows/metric-selection/instructions.md

For Forecasting Fundamentals

Valeriy Manokhin's Books:

• "Mastering Modern Time Series Forecasting"
• "Applied Conformal Prediction"
• "Mastering Forecasting Metrics & Accuracy"

LinkedIn: Follow Valeriy Manokhin for case studies and methodology insights

Classic Textbook:

• "Forecasting: Principles and Practice" (Hyndman & Athanasopoulos)

🤝 Contributing

Current Status: Module in active development by Philippe Béliveau.

Future Collaboration:

• Planned outreach to Valeriy Manokhin with working demo
• Community contributions welcome after Phase 1 complete
• Issues and feedback: GitHub repository

📜 License

MIT License

Copyright (c) 2025 Philippe Béliveau / Newcode

🙏 Acknowledgments

• Valeriy Manokhin: Methodology foundation, conformal prediction expertise, MAPE disaster case study
• BMAD Community: Framework and module builder tools
• Nixtla Team: Comprehensive forecasting library ecosystem
• Corporate Treasurers (Videns Analytics): Real-world problem framing insights

📞 Contact

Philippe Béliveau Co-Founder, Newcode Montréal, Quebec, Canada

Newcode Philosophy: "If you learn to ask the right question, then you can build anything with AI agents"

Built with BMAD V6 Module Builder Making AI agents work together to solve hard problems