Datadog Patrol MCP Server

An automated production error monitoring system for FW/TXM microservices. It connects to Datadog, pulls error logs, tracks known issues in a knowledge base, and sends summary alerts to Slack — all driven by natural-language prompts in Cursor IDE.

Think of it as an automated security guard for your production services — it watches, remembers, classifies, reports, and alerts.

Table of Contents

• What Does It Do?
• Screenshots
• How It Works — The Patrol Flow
• Error Classification
• Noise Filtering Effectiveness
• Slack Message Flow
• Prerequisites
• Setup
• MCP Tools
• Cursor Agent Skills
• Quick Start Prompts
• Knowledge Base
• Monitored Services
• Performance
• Project Structure
• Why Not the Built-in Datadog MCP?
• Troubleshooting
• Improvement Roadmap

What Does It Do?

1. Watches 11 production microservices for errors
2. Remembers every error pattern it has seen before (knowledge base with 37+ tracked issues)
3. Classifies each error as New, Repeated, or Resolved — inline during fetch
4. Reports a clear summary with trends, severity, and action items
5. Alerts your team on Slack (#fw-prod-error-alerts) with a compact, actionable message

Screenshots

Running a Patrol from Cursor

Type a natural-language prompt like "Run prod patrol for last 24 hours" — the MCP server discovers tools, checks Datadog health, loads the KB, and kicks off the patrol automatically.

Patrol Dashboard — Starting

The dashboard shows each pipeline step. Here the patrol has loaded the KB, built the query, and begun fetching pages with inline classification (genuine / suppressed / noise).

Parallel Slice Fetch — Live Progress

For larger time ranges, logs are fetched across multiple concurrent slices. Each slice shows its progress bar, log count, and the live log streams genuine error counts in real time.

Patrol Complete — All Slices Done

All 12 slices complete with 125,952 logs classified in 188 seconds. The pipeline shows every step checked: Load KB → Build Query → Estimate → Fetch Pages → Write Summary → Update History → Complete.

Large Scale Patrol — 15 Slices, 529K Logs

A 6-hour patrol with 15 density-aware slices handling 529K+ logs. Heavy time periods (nightly batch) automatically get more slices.

Pipeline Complete — Full Workflow

A completed 1-hour patrol showing all pipeline steps done, with page-level detail in the live log including genuine/suppressed/noise counts per page.

Patrol Report — Rendered in Cursor

The generated patrol summary renders directly in Cursor chat with new issues, repeated patterns with previous vs. current counts, KB status, and classification breakdown.

Slack Send Prompt — User Approval

After the patrol report is generated, you're asked whether to send it to Slack. The prompt shows key findings (health status, urgent issues, good news) and lets you choose: send to the default channel, skip, or pick a different channel.

Slack Alert — #fw-prod-error-alerts

Compact patrol summaries are forwarded to Slack via the Datadog Events API with new issues, re-appeared warnings, repeated issue trends, and Datadog deep links for instant investigation.

Patrol History — Track Every Run

The History tab shows all past patrol runs with total logs, genuine errors, suppressed counts, noise, and duration — making it easy to spot trends over time.

How It Works — The Patrol Flow

flowchart TD
    A["🧑 You: Run prod patrol\nfor last 12 hours"] --> B["1. Health Check\n— Is Datadog reachable?"]
    B --> C["2. Load KB\n— 37 known error patterns"]
    C --> D["3. Estimate + Plan\n— Analytics API → ~129K logs\n— Density-aware slice planning"]
    D --> E["4. Parallel Fetch +\nInline Classify\n— 8 concurrent slices\n— HTTP 5xx retry"]

    E --> F["genuine-errors/"]
    E --> G["suppressed-known/"]
    E --> H["noise/"]

    F --> I["5. Analyze\n— Match vs KB\n— Count per service"]
    I --> J["6. Classify\n— 🆕 New\n— 🔁 Repeated\n— ✅ Resolved"]
    J --> K["7. Report\n— Full summary in Cursor"]
    K --> L{"8. Send to Slack?"}
    L -- Yes --> M["9. Slack Alert\n→ Datadog Events API\n→ #fw-prod-error-alerts"]
    L -- No --> N["10. Update KB\n— Save new findings"]
    M --> N

Step-by-Step Breakdown

| Step | What Happens | System | |------|-------------|--------| | 1. Health Check | Verify Datadog API is reachable | Datadog API | | 2. Load KB | Read 37 known error patterns from issues-export.csv | Local CSV | | 3. Estimate + Plan | Use Analytics Aggregate API to estimate log volume, plan density-aware slices | Datadog Analytics | | 4. Parallel Fetch | Fetch logs across 8 concurrent slices with inline noise classification | Datadog Logs API v2 | | 5. Analyze | Read only genuine error files, match against KB | Local files | | 6. Classify | Categorize as New / Repeated / Resolved with trend indicators | In-memory | | 7. Report | Generate full markdown summary with action items | Cursor chat | | 8. Ask User | Confirm before sending to Slack (see prompt) | User approval | | 9. Slack Alert | Post compact summary via Datadog Events API | Datadog → Slack | | 10. Update KB | Save new findings, update counts and timestamps | Local CSV |

Error Classification — How It Decides

Classification happens inline during fetch (not after), powered by the NoiseClassifier:

flowchart TD
    A["Log fetched from Datadog"] --> B{"Matches noise filter\nin noise-filters.json?"}
    B -- Yes --> C["🔇 NOISE\n→ written to noise/"]
    B -- No --> D{"Matches KB issue with\nSEPARATE / COUNT_ONLY?"}
    D -- Yes --> E["📦 SUPPRESSED\n→ written to suppressed-known/"]
    D -- No --> F{"INFO level with\nno real error content?"}
    F -- Yes --> C
    F -- No --> G["✅ GENUINE ERROR\n→ written to genuine-errors/"]

    G --> H{"Pattern in KB?"}
    H -- Yes --> I["🔁 REPEATED\n— show trend: 📈 up / 📉 down"]
    H -- No --> J["🆕 NEW\n— add to KB, flag for investigation"]

    K["Pattern in KB but\nNOT in logs"] --> L["✅ RESOLVED\n— mark after consecutive confirmations"]

Suppression Levels

| Level | Behavior | |-------|----------| | NONE | No suppression — error is analyzed as genuine | | SEPARATE | Counted and tracked separately, excluded from the main report | | COUNT_ONLY | Only the count is tracked; individual logs are not stored | | EXCLUDE | Removed from the Datadog query itself, reducing API calls |

Noise Filtering Effectiveness

A typical 12-hour patrol processes ~130K logs but only surfaces ~5K actionable errors:

  Raw logs from Datadog:  128,959 (100%)
       │
       ├── Noise (filtered):     80,965 (62.8%)   ← saved from manual review
       ├── Suppressed (known):   42,372 (32.9%)   ← tracked separately
       └── Genuine (actionable):  5,622 ( 4.4%)   ← what you actually analyze

The noise filter rules live in noise-filters.json and can be tuned using the noise-filter-tuning skill.

Slack Message Flow

flowchart LR
    A["Patrol Report\n(full markdown)"] --> B["buildCompactMessage()\n— Extract sections\n— Compute trends\n— Add deep links\n— UTC → IST"]
    B --> C["Datadog Events API\nPOST /api/v1/events\n— 4000 char limit\n— Smart truncation"]
    C --> D["Datadog → Slack\nIntegration forwards to\n#fw-prod-error-alerts"]

What the Slack Message Contains

• Header — Date (IST), services patrolled, total errors with trend vs. previous patrol
• New Issues — Each with severity, hit count, service, and Datadog deep link
• Re-appeared Issues — Previously resolved errors that returned
• Repeated Issues — With trend indicators (📈 up / 📉 down / ➡️ stable)
• Resolved Issues — Errors no longer seen in this patrol window
• Action Items — Prioritized by severity with recommended next steps
• Full Report Link — Reference to the saved patrol summary file

Prerequisites

• Node.js 18+
• Datadog API key and Application key (Datadog → Organization Settings → API/Application Keys)
• Cursor IDE with MCP support

Setup

1. Install and Build

cd datadog-mcp-observability
npm install
npm run build

2. Configure Environment Variables

cp .env.example .env

Edit .env with your credentials and any optional overrides:

Required Variables

| Variable | Description | Default | |----------|-------------|---------| | DATADOG_API_KEY | Datadog API key (Organization Settings → API Keys) | — (required) | | DATADOG_APP_KEY | Datadog Application key (Organization Settings → Application Keys) | — (required) | | DATADOG_SITE | Datadog site URL | datadoghq.com |

Optional Variables

| Variable | Description | Default | |----------|-------------|---------| | SLACK_CHANNEL | Slack channel for patrol alerts (without #) | fw-prod-error-alerts | | PATROL_SERVICES_FILE | Path to JSON file with monitored services | config/services.json | | PATROL_DASHBOARD_PORT | Port for the SSE progress dashboard | 3456 | | DATA_DIR | Directory where patrol reports are stored | reports/ | | QUERY_FILE_PATH | Fallback query file for query_datadog_logs (patrols don't use this) | queries/error-query.txt | | LOG_LEVEL | Winston log level (debug, info, warn, error) | info | | NODE_TLS_REJECT_UNAUTHORIZED | Set to 0 to disable TLS cert verification (dev only) | 1 (enabled) |

Example .env

# Required — Datadog credentials
DATADOG_SITE=datadoghq.com
DATADOG_API_KEY=your-api-key-here
DATADOG_APP_KEY=your-app-key-here

# Optional — overrides
SLACK_CHANNEL=fw-prod-error-alerts
PATROL_DASHBOARD_PORT=3456
LOG_LEVEL=info

3. Register in Cursor

Add to your Cursor MCP settings (Settings → MCP → Add new global MCP server):

Option A: Via npm (recommended — no git clone needed)

{
  "mcpServers": {
    "datadog-patrol": {
      "command": "npx",
      "args": ["-y", "@pardhu186/datadog-patrol-mcp"],
      "env": {
        "DATADOG_API_KEY": "<your-datadog-api-key>",
        "DATADOG_APP_KEY": "<your-datadog-app-key>",
        "DATADOG_SITE": "datadoghq.com",
        "NODE_TLS_REJECT_UNAUTHORIZED": "0",
        "PATROL_SENDER_NAME": "Your Name",
        "PATROL_DASHBOARD_ENABLED": "true"
      }
    }
  }
}

Option B: From local clone

{
  "mcpServers": {
    "datadog-patrol": {
      "command": "node",
      "args": ["/absolute/path/to/datadog-mcp-observability/dist/index.js"],
      "env": {
        "DATADOG_API_KEY": "<your-datadog-api-key>",
        "DATADOG_APP_KEY": "<your-datadog-app-key>",
        "DATADOG_SITE": "datadoghq.com",
        "NODE_TLS_REJECT_UNAUTHORIZED": "0",
        "PATROL_SENDER_NAME": "Your Name",
        "PATROL_DASHBOARD_ENABLED": "true"
      }
    }
  }
}

Tip: Environment variables in the MCP config override values in .env. You can set credentials in either place — the MCP env block takes precedence. Set NODE_TLS_REJECT_UNAUTHORIZED=0 if you're behind a corporate proxy.

Restart Cursor after adding the config.

4. Verify

Type in Cursor chat: "Check Datadog connection" — you should see a health check response confirming the API is reachable.

MCP Tools (11 tools)

| Tool | What It Does | Key Params | |------|-------------|------------| | patrol_production | Full patrol across all services — fetches, classifies, reports | timeRange (default: 6h) | | query_datadog_logs | Custom Datadog log query against flex index | query, timeRange, writeToFiles | | fetch_file_content | Read a log or report file from disk | filePath | | get_datadog_health | Check Datadog API connectivity | — | | update_knowledge_base | Add or update error issues in the KB | issues (array of issue objects) | | get_knowledge_base | Read current KB state | status (optional: OPEN/IN_PROGRESS/RESOLVED/CLOSED) | | get_patrol_history | List past patrol runs | limit (default: 20) | | send_patrol_to_slack | Send patrol summary to Slack via Datadog Events | patrolId + slackMessagePath, or summary, or reportPath; optional: channel, timeRange | | suppress_issue | Change suppression/classification of a KB issue | issueId; optional: suppressionLevel, classification, reason | | save_patrol_report | Save report and generate Slack preview for review before sending | summary; optional: timeRange | | list_services | List all monitored services with sourcecategory mappings | — |

Supported Time Ranges

Accepted values: 1h, 3h, 6h, 12h, 1d, 7d, 30d

Cursor Agent Skills (5 Skills)

Pre-built Cursor Agent Skills that extend the MCP tools with guided workflows:

| Skill | What It Does | |-------|-------------| | datadog-prod-patrol | Guided patrol workflow — fetches, classifies, generates reports, sends Slack alerts | | datadog-log-query | Ad-hoc Datadog log queries for debugging specific errors | | error-kb-management | Suppress, exclude, classify, update, and review KB error patterns | | noise-filter-tuning | Analyze signal-to-noise ratios, identify noise candidates, manage noise-filters.json | | patrol-report-analysis | Compare trends across patrols, spot error spikes, track issue resolution over time |

How Skills Work Together

flowchart LR
    A["Run Patrol"] --> B["Analyze Report"]
    B --> C["Manage KB"]
    C --> D["Tune Noise Filters"]
    D --> E["Next Patrol\nis Cleaner 🎯"]

Installation

Option A: Project-level (recommended)

cp -r skills/* .cursor/skills/

Option B: User-level (all projects)

cp -r skills/* ~/.cursor/skills/

Verify: Ask the agent "What skills are available?" — it should list all 5 patrol skills.

Quick Start Prompts

| What You Want | Type This in Cursor | |---------------|---------------------| | Full patrol | "Run a prod patrol for the last 6 hours" | | FW services only | "Patrol FW services for errors in the last 3 hours" | | TXM services only | "Check TXM services for new errors" | | Specific service | "Show errors for prod1-fw-balance-manager" | | Search by keyword | "Search for NullPointerException in production" | | Trace an account | "Find errors for account 12345" | | View KB | "Show me the knowledge base" | | Update KB | "Mark ERR-001 as RESOLVED" | | Suppress an issue | "Suppress ERR-014" | | Patrol history | "Show patrol history" | | Compare patrols | "Compare the last two patrol reports" | | Noise analysis | "What's the signal-to-noise ratio?" |

Knowledge Base

The knowledge base is the memory of the patrol system. It tracks every error pattern seen across all patrols.

• Location: knowledge-base/issues-export.csv
• Format: CSV with 37+ tracked issues
• Updated automatically after each patrol
• Statuses: OPEN → IN_PROGRESS → RESOLVED → CLOSED

What Gets Tracked Per Issue

| Field | Example | |-------|---------| | Issue ID | ERR-001 | | Error Category | MongoDB Duplicate Key | | Component | prod1-fw-balance-manager | | Error Message | E11000 duplicate key error on balanceTrends | | Severity | CRITICAL / HIGH / MEDIUM / LOW | | Frequency | ~16,423 | | Classification | TRUE_ERROR / OPERATIONAL / EXPECTED_BEHAVIOR / NOISE | | Suppression Level | NONE / SEPARATE / COUNT_ONLY / EXCLUDE | | Affected Accounts | 32694429586090557798837049311 | | Root Cause | Concurrent threads inserting duplicate keys | | Recommended Action | Add upsert logic in DBClient bulk operations | | First Seen | 2026-02-28 12:36:44 | | Last Seen | 2026-03-13 04:42:00 | | Status | OPEN |

Monitored Services (11)

  FW Services (9)                         TXM Services (2)
  ───────────────                         ────────────────
  balance-manager                         alrt-txm-gov
  balance-agg                             alrt-txm-manager
  txm-normalizer
  fi-manager
  txm-reminder
  label-manager
  exp-manager
  asset-mgmt
  daily-balanceTrends-forwardfilling

All services are queried from the Datadog flex index (lifelock-prod-flex) using sourcecategory filters.

Performance

| Time Range | Est. Logs | Slices | Fetch Time | Classification | |------------|-----------|--------|------------|----------------| | 1h | ~5K | 1 | ~30s | Inline | | 6h | ~50K | 4-6 | ~90s | Inline | | 12h | ~130K | 8 | ~3 min | Inline | | 1d | ~250K+ | 8-16 | ~6 min | Inline |

Key Optimizations

• Parallel slicing — 6-8x faster than sequential page fetching
• Inline classification — noise/suppression filtering during fetch, not after
• Smart slice planning — dense time periods (nightly batch) get more slices
• HTTP 5xx retry — failed slices automatically recovered with sequential retry after 5s cooldown
• Query-level exclusion — EXCLUDE suppression patterns removed from Datadog query, reducing API load

Project Structure

datadog-mcp-observability/
├── src/                              Source code (TypeScript)
│   ├── index.ts                      MCP server entry point (11 tools)
│   ├── datadog-client.ts             Datadog Logs API v2 client
│   │                                 • Parallel query execution (8 concurrent slices)
│   │                                 • Smart slice planning (density-aware)
│   │                                 • HTTP 5xx retry + failed-slice recovery
│   │                                 • Inline noise classification during fetch
│   │                                 • Log count estimation via Analytics API
│   ├── noise-classifier.ts           Noise/KB classification engine
│   │                                 • Pattern-based noise filtering
│   │                                 • KB suppression (SEPARATE/COUNT_ONLY/EXCLUDE)
│   │                                 • Heuristic INFO-level detection
│   ├── config.ts                     Service definitions + configuration
│   ├── slack-notifier.ts             Slack message builder + Datadog Events sender
│   │                                 • UTC→IST conversion
│   │                                 • Trend computation (vs previous patrol)
│   │                                 • Service breakdown + noise summary
│   │                                 • Smart truncation (4000-char limit)
│   ├── progress-server.ts            SSE-based progress streaming
│   ├── chart-generator.ts            Parses service errors + top errors from reports
│   └── logger.ts                     Winston logger
│
├── skills/                           Cursor Agent Skills (5 skills)
│   ├── README.md                     Skills overview + installation
│   ├── datadog-prod-patrol/          Patrol workflow skill
│   ├── datadog-log-query/            Ad-hoc log query skill
│   ├── error-kb-management/          KB management skill
│   ├── noise-filter-tuning/          Noise filter tuning skill
│   └── patrol-report-analysis/       Report analysis skill
│
├── knowledge-base/
│   └── issues-export.csv             Error pattern knowledge base (37 issues)
│
├── noise-filters.json                Noise classification rules (regex patterns)
│
├── reports/
│   ├── patrol-logs/                  Classified log files per patrol run
│   │   └── YYYY-MM-DD/
│   │       ├── genuine-errors/       Actionable error logs
│   │       ├── suppressed-known/     Known KB issues (tracked separately)
│   │       ├── noise/                Filtered noise logs
│   │       └── classification-summary.json
│   ├── patrol-summaries/             Full patrol reports (markdown)
│   ├── slack-messages/               Pre-built Slack messages
│   ├── patrol-history.json           History of all patrol runs
│   └── patrol-tracking.json          Tracking metadata
│
├── dashboard-migrations/             Sumo Logic → Datadog dashboard migrations
│
├── docs/
│   └── screenshots/                  README screenshots
│
├── config/
│   └── services.json                 Service configuration (if externalized)
│
├── dist/                             Compiled JavaScript (auto-generated)
├── logs/                             Runtime logs (combined.log, error.log)
│
├── .cursor/
│   └── rules/
│       └── datadog-prod-patrol.mdc   AI patrol workflow rules
│
├── package.json                      Dependencies + scripts
├── tsconfig.json                     TypeScript configuration
├── .env.example                      Credential template
├── IMPROVEMENTS.md                   Roadmap + completed improvements
└── README.md                         This file

Why Not the Built-in Datadog MCP?

The standard Datadog MCP (get_logs) does not support:

• indexes parameter (needed for flex index)
• storage_tier parameter (needed for flex storage)

FW production logs live in the flex index (lifelock-prod-flex) and are invisible to the generic MCP. This server calls the Datadog Logs API v2 directly with full flex support.

Sample Patrol Output

📊 PRODUCTION PATROL SUMMARY — Last 24 Hours

Date: 2026-03-12 04:42 UTC → 2026-03-13 04:42 UTC
Total Logs Fetched: 309,541
- Genuine (actionable): 8,089 (2.6%)
- Suppressed (known KB): 300,671 (97.1%)
- Noise (filtered): 781 (0.3%)

Services Patrolled: 11 (FW + TXM)
Overall Health: 🔴 CRITICAL

🆕 NEW ISSUES (2)
  ERR-036 | Redisson DNS resolution failure | ~5 hits | HIGH | alrt-txm-gov
  ERR-037 | Datadog Trace Agent ConnectException | ~3 hits | LOW | fi-manager

🔁 TOP REPEATED ISSUES
  ERR-001 | MongoDB Duplicate Key | ~300,671 (suppressed) | 📈 +223% | CRITICAL
  ERR-020 | getFiAccountIds Timeout 30s | ~2,000+ | ➡️ Sustained | CRITICAL
  ERR-009 | Invalid Access Token 440 | ~1,500+ | 📈 +650% | CRITICAL

✅ RESOLVED (12 issues disappeared from logs)

🎯 URGENT ACTION ITEMS
  1. 🔴 Investigate fi-manager connection pool — timeout cascade
  2. 🔴 ERR-001 MongoDB duplicate key at 300K+/24h — needs upsert fix
  3. 🔴 ERR-009 + ERR-015 token/auth storm — check Plaid API health

🔇 Signal-to-Noise: 97.4% filtered

@slack-fw-prod-error-alerts

Date (IST): 2026-03-12 10:12 IST → 2026-03-13 10:12 IST
Services: 11 services (FW + TXM)

Total Errors: 🔴 309,541 (📉 95% down rate vs prev 6,804 in 1h)
New: 0 🆕 | Repeated: 0 🔁 | Resolved: 2 ✅

🆕 New Issues
- 🟠 ERR-036: Redisson DNS failure — 5 hits in alrt-txm-gov [Logs →]
- 🟢 ERR-037: Trace Agent ConnectException — 3 hits in fi-manager [Logs →]

🔁 Repeated Issues
- 📈 ERR-001: MongoDB Duplicate Key — +223% (🔴 CRITICAL)
- ➡️ ERR-020: getFiAccountIds Timeout — Sustained (🔴 CRITICAL)
- 📈 ERR-009: Invalid Access Token — +650% (🔴 CRITICAL)

🚨 Action Items
- 1. 🔴 Investigate fi-manager connection pool
- 2. 🔴 ERR-001 needs upsert strategy fix
- 3. 🔴 Token/auth storm — check Plaid API health

📄 Full report: 2026-03-13-1025.md

Troubleshooting

| Problem | Likely Cause | Fix | |---------|-------------|-----| | 403 Forbidden | Invalid API/APP keys | Check keys in .env, verify permissions in Datadog | | No logs returned | Flex index not queried | Built-in — verify service is logging to flex index | | ECONNRESET | Network/VPN issue | Check VPN, retry is built-in (3x with backoff) | | HTTP 500 from Datadog | API overloaded | Auto-retried 3x with backoff; failed slices retried sequentially after 5s cooldown | | Timeout | Firewall blocking | Allow api.datadoghq.com:443 | | MCP not in Cursor | Config path wrong | Verify absolute path in MCP settings, restart Cursor | | Build fails | TypeScript error | Run npx tsc --noEmit for diagnostics | | Empty patrol | No errors in time range | Production is healthy! Try a broader time range | | Slack missing sections | Message over 4000 chars | Sections are prioritized: New → Repeated → Resolved → Top Errors | | Slack not received | Datadog→Slack integration | Check @slack-<channel> handle matches your Datadog notification rule | | Skills not recognized | Skills not installed | Copy skills/* to .cursor/skills/ or ~/.cursor/skills/ |

Improvement Roadmap

See IMPROVEMENTS.md for the full roadmap. Key highlights:

Completed

| Improvement | Impact | |------------|--------| | Parallel query execution + smart slice planning | 6-8x faster fetch | | Inline noise classification during fetch | 95.6% noise filtered automatically | | HTTP 5xx retry + failed-slice recovery | 100% data completeness | | Enhanced Slack notifier with trends | Richer team alerts | | SSE-based progress streaming | Real-time patrol visibility | | Externalized service configuration | Edit config/services.json, no rebuild |

Planned

| Improvement | Priority | |------------|----------| | Auto-clean patrol log files | P0 | | Verify Datadog deep links | P1 | | Richer Slack (Block Kit) | P1 | | Scheduled patrols | P2 | | CI/CD integration | P2 |

License

Internal tooling — FW/TXM team use only.