Sitemap MCP Server

Model Context Protocol (MCP) server for sitemap crawling and change detection - optimized for web scraping workflows.

Features

• 🚀 Fast sitemap fetching with automatic pagination support
• 🎯 Smart filtering - EITHER regex patterns OR LLM-powered intelligent filtering
• 🤖 LLM filtering - AI understands your intent, no regex needed
• 🔍 Change detection - only crawl modified pages (saves 90%+ time on updates)
• 📊 Analytics - understand site structure before crawling
• 🎓 Degree level splitting - automatic UG/PG program classification
• 💾 Lightweight history - single crawl_history.json per project

Installation

cd /home/lyrica/projects/sitemap_MCP
npm install

Add to Claude Code

claude mcp add sitemap node /home/lyrica/projects/sitemap_MCP/index.js

Or manually add to ~/.claude.json:

Without LLM (manual regex patterns only):

{
  "mcpServers": {
    "sitemap": {
      "command": "node",
      "args": ["/home/lyrica/projects/sitemap_MCP/index.js"],
      "transport": "stdio"
    }
  }
}

With LLM filtering (optional):

{
  "mcpServers": {
    "sitemap": {
      "command": "node",
      "args": ["/home/lyrica/projects/sitemap_MCP/index.js"],
      "env": {
        "PROVIDER": "openai",
        "API_KEY": "sk-...",
        "MODEL": "gpt-4o-mini"
      },
      "transport": "stdio"
    }
  }
}

Supported providers:

• openai - OpenAI (gpt-4o, gpt-4o-mini, etc.)
• anthropic - Claude (claude-3-5-sonnet-latest, etc.)
• openrouter - OpenRouter (any model)

Tools

1. fetch_sitemap

Fetch and filter URLs from a website's sitemap.

⚠️ IMPORTANT: Choose ONE filtering mode:

• Manual mode: Use include_patterns + exclude_patterns (regex)
• LLM mode: Use use_llm: true + user_intent (AI-powered)

Parameters:

• url (required): Sitemap URL
• use_llm: Boolean - use LLM for intelligent filtering (requires PROVIDER, API_KEY, MODEL in env)
• user_intent: String - what you're looking for (required if use_llm=true)
• include_patterns: Array of regex patterns to include (ignored if use_llm=true)
• exclude_patterns: Array of regex patterns to exclude (ignored if use_llm=true)
• degree_level: Filter by "both", "pg", or "ug"

Example 1: Manual regex filtering (precise control)

fetch_sitemap({
  url: "https://www.ox.ac.uk/sitemap.xml",
  include_patterns: ["/graduate/courses/", "/postgraduate/"],
  exclude_patterns: ["courses-a-z", "open-days", "departments"],
  degree_level: "pg"
})

Example 2: LLM filtering (intelligent)

fetch_sitemap({
  url: "https://www.ox.ac.uk/sitemap.xml",
  use_llm: true,
  user_intent: "I want postgraduate computer science and AI programs only, exclude short courses and executive education",
  degree_level: "pg"
})

Returns:

{
  "total": 537,
  "pg_count": 486,
  "ug_count": 51,
  "unknown_count": 0,
  "urls": [...],
  "split": {
    "pg": [...],
    "ug": [...]
  }
}

2. detect_changes

Compare current sitemap with previous crawl to detect changes.

Parameters:

• university_key (required): Unique identifier (e.g., "oxford")
• sitemap_url (required): Sitemap URL
• include_patterns: Optional filtering
• exclude_patterns: Optional filtering

Example:

detect_changes({
  university_key: "oxford",
  sitemap_url: "https://www.ox.ac.uk/sitemap.xml",
  include_patterns: ["/courses/"],
  exclude_patterns: ["courses-a-z"]
})

Returns:

{
  "modified_urls": [...],      // Changed since last crawl
  "unchanged_urls": [...],     // No changes
  "is_first_crawl": false,
  "summary": {
    "modified_count": 45,
    "unchanged_count": 492,
    "should_crawl": true
  }
}

Saves to crawl_history.json:

{
  "oxford": {
    "last_crawl": "2025-11-11T15:00:00Z",
    "sitemap_url": "https://www.ox.ac.uk/sitemap.xml",
    "total_urls": 537,
    "pg_urls": 486,
    "ug_urls": 51
  }
}

3. analyze_sitemap

Get statistics about sitemap structure.

Parameters:

• url (required): Sitemap URL

Example:

analyze_sitemap({
  url: "https://www.ox.ac.uk/sitemap.xml"
})

Returns:

{
  "total_urls": 537,
  "by_degree_level": {
    "postgraduate": 486,
    "undergraduate": 51,
    "unknown": 0
  },
  "by_pattern": {
    "/admissions/": 537,
    "/courses/": 486
  },
  "by_last_modified": {
    "2025-10": 120,
    "2025-09": 200
  },
  "priority_stats": {
    "avg": "0.75",
    "distribution": {
      "0.8": 300,
      "1.0": 237
    }
  }
}

Usage Workflow

First-time crawl:

# 1. Analyze sitemap structure
analyze_sitemap({ url: "https://www.ox.ac.uk/sitemap.xml" })

# 2. Fetch filtered URLs
fetch_sitemap({
  url: "https://www.ox.ac.uk/sitemap.xml",
  include_patterns: ["/graduate/courses/"],
  exclude_patterns: ["courses-a-z", "open-days"]
})

# 3. Use URLs in your crawler script
# (537 URLs discovered)

Quarterly update (3 months later):

# 1. Detect changes since last crawl
detect_changes({
  university_key: "oxford",
  sitemap_url: "https://www.ox.ac.uk/sitemap.xml",
  include_patterns: ["/graduate/courses/"]
})

# Returns: modified_count: 45 (only 45/537 changed)
# 2. Only crawl the 45 modified URLs
# Save 91% time! ⚡

How Change Detection Works

The MCP uses sitemap's <lastmod> field to detect changes:

<url>
  <loc>https://www.ox.ac.uk/admissions/graduate/courses/msc-cs</loc>
  <lastmod>2025-10-15</lastmod>  <!-- Page last modified -->
</url>

Logic:

1. First crawl → saves timestamp to crawl_history.json
2. Next crawl → compares page's lastmod with saved timestamp
3. Only returns URLs where lastmod > last_crawl

Benefits:

• 90%+ time savings on quarterly updates
• No need to re-crawl unchanged pages
• Lightweight: single JSON file for all universities

Project Structure

/home/lyrica/projects/sitemap_MCP/
  ├── index.js            # MCP server
  ├── package.json
  └── README.md

# When used in Offer_I project:
/home/lyrica/Offer_I/
  └── crawl_history.json  # Created automatically

Development

# Install dependencies
npm install

# Run in dev mode (auto-reload)
npm run dev

# Test manually
node index.js

License

MIT © kaminoguo