Web Feed (@ai/web-feed)

A local MCP (Model Context Protocol) server that provides browser automation capabilities using Playwright. This server enables AI applications to control a web browser for navigation, content extraction, and interaction with web pages.

Features

• 🌐 Web Navigation: Navigate to any URL with smart waiting
• 📄 Content Extraction: Extract text, HTML, links, and images
• 🖱️ Element Interaction: Click elements and fill forms
• 📸 Screenshots: Capture full page or element-specific screenshots
• 🛡️ Error Handling: Comprehensive error handling and recovery
• 📊 Logging: Detailed logging and performance monitoring
• 🔧 CLI Tool: Command-line interface for direct browser automation

Installation

Global Installation (Recommended)

npm install -g @ai/web-feed

Local Installation

npm install @ai/web-feed

Development Setup

1. Clone the repository:

git clone https://github.com/your-username/web-feed.git
cd web-feed

2. Install dependencies:

npm install

3. Build the project:

npm run build

Configuration

Copy .env.example to .env and adjust settings as needed:

cp .env.example .env

Key configuration options:

• BROWSER_HEADLESS: Run browser in headless mode (default: true)
• BROWSER_TIMEOUT: Page load timeout in milliseconds (default: 30000)
• LOG_LEVEL: Logging level - error, warn, info, debug (default: info)

Usage

MCP Client Configuration

Add to your MCP client configuration (e.g., Claude Desktop):

For globally installed package:

{
  "mcpServers": {
    "web-feed": {
      "command": "web-feed",
      "env": {
        "BROWSER_HEADLESS": "true",
        "LOG_LEVEL": "info"
      }
    }
  }
}

For locally installed package:

{
  "mcpServers": {
    "web-feed": {
      "command": "node",
      "args": ["./node_modules/.bin/web-feed"],
      "env": {
        "BROWSER_HEADLESS": "true",
        "LOG_LEVEL": "info"
      }
    }
  }
}

CLI Usage

The package also provides a CLI tool for direct browser automation:

# Navigate to a URL and take screenshot
web-feed navigate https://example.com --screenshot

# Extract content from a page
web-feed extract https://news.ycombinator.com --type text

# Interactive debugging
web-feed debug https://example.com

Debug

npx @modelcontextprotocol/inspector -- web-feed -e BROWSER_HEADLESS=true -e LOG_LEVEL=debug

Development Mode

Run directly from source:

npm run dev

Production Mode

Run built version:

npm start

Available Tools

browser_navigate

Navigate to a specified URL.

{
  "name": "browser_navigate",
  "arguments": {
    "url": "https://example.com",
    "waitForSelector": "#main-content",
    "timeout": 30000
  }
}

browser_get_content

Extract content from the current page.

{
  "name": "browser_get_content",
  "arguments": {
    "selector": "body",
    "type": "text"
  }
}

Content types:

• text: Plain text content
• html: HTML source
• links: Array of links with text and URLs
• images: Array of images with src and dimensions

browser_click

Click on an element matching the CSS selector.

{
  "name": "browser_click",
  "arguments": {
    "selector": "#submit-button",
    "timeout": 5000
  }
}

browser_fill_form

Fill a form field with the specified value.

{
  "name": "browser_fill_form",
  "arguments": {
    "selector": "#search-input",
    "value": "search query",
    "timeout": 5000
  }
}

browser_screenshot

Take a screenshot of the current page.

{
  "name": "browser_screenshot",
  "arguments": {
    "fullPage": true,
    "selector": "#content"
  }
}

Examples

Basic Navigation

Navigate to https://google.com
Get the page content
Click on the search input
Fill form "search query" into the search input

Data Extraction

Navigate to https://news.ycombinator.com
Get all links from the page
Get the page title
Take a screenshot of the page

Error Handling

The server provides detailed error information:

• BrowserTimeoutError: Operation exceeded timeout
• ElementNotFoundError: Specified element not found
• NavigationError: Failed to navigate to URL
• ValidationError: Invalid input parameters

Logging

Logs are written to both console and files:

• logs/combined.log: All log entries
• logs/error.log: Error-level logs only

Security Considerations

• The browser runs with standard user permissions
• All browsing history and data is cleared when the server stops
• URL validation prevents access to local files (file://)
• Consider running in a sandboxed environment for sensitive operations

Requirements

• Node.js 18+
• Chrome/Chromium browser (Playwright will download if needed)
• 512MB+ RAM available
• 100MB+ disk space

Development

Project Structure

src/
├── server.ts              # MCP server implementation
├── browser/
│   ├── manager.ts         # Browser automation core
│   ├── types.ts           # TypeScript definitions
│   └── actions.ts         # Browser actions
├── tools/                 # MCP tool definitions
├── utils/
│   ├── logger.ts          # Logging utilities
│   ├── config.ts          # Configuration management
│   └── errors.ts          # Error handling
└── index.ts               # Main entry point

Building

npm run build

Testing

npm test

License

MIT License - see LICENSE file for details.

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests if applicable
5. Ensure tests pass (npm test)
6. Commit your changes (git commit -m 'Add some amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Support

For issues and questions, please create an issue in the project repository.

Changelog

v1.0.0

• Initial release
• MCP server implementation with Playwright integration
• Full browser automation capabilities
• CLI tool for direct usage
• Comprehensive error handling and logging