crawl4ai-mcp-server

Overview

Crawl4AI MCP Server is a purpose-built implementation that exposes web scraping and crawling capabilities through the Model Context Protocol. It offers four main tools: scrape for single-page content extraction, crawl for multi-page breadth-first crawling with adaptive stopping, crawl_site for comprehensive site crawling, and crawl_sitemap for sitemap-based crawling. The server is self-hosted and free, providing an alternative to Firecrawl's API while offering similar functionality. It supports various AI development environments including OpenAI Agents SDK, Cursor, and Claude Code.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• scrapescrape(url, output_dir, crawler, browser, script, timeout_sec)

  Fetch a single URL and return markdown content.

  Try:Scrape the content from https://example.comTry:Extract markdown content from https://news.example.com/articleTry:Get the full text content of https://blog.example.com/post
• crawlcrawl(seed_url, max_depth, max_pages, same_domain_only, include_patterns, exclude_patterns, adaptive, output_dir, crawler, browser, script, timeout_sec)

  Multi-page breadth-first crawling with filtering and adaptive stopping.

  Try:Crawl https://docs.example.com with adaptive stoppingTry:Get content from up to 10 pages starting from https://blog.example.comTry:Crawl https://news.example.com but only from technology sections
• crawl_sitecrawl_site(entry_url, output_dir, max_depth, max_pages, ...)

  Comprehensive site crawling with persistence.

  Try:Crawl the entire website at https://example.comTry:Download all pages from https://docs.example.comTry:Archive content from https://blog.example.com to local storage
• crawl_sitemapcrawl_sitemap(sitemap_url, output_dir, max_entries, ...)

  Sitemap-based crawling with persistence.

  Try:Crawl all URLs in https://example.com/sitemap.xmlTry:Process the sitemap at https://blog.example.com/sitemap.xmlTry:Extract content from all pages listed in https://news.example.com/sitemap.xml

Comparable tools

Installation

# Pull pre-built image
docker pull uysalsadi/crawl4ai-mcp-server:latest

# Test the installation
python test-config.py

git clone https://github.com/uysalsadi/crawl4ai-mcp-server.git
cd crawl4ai-mcp-server
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -r requirements.txt
python -m playwright install chromium

{
  "mcpServers": {
    "crawl4ai-mcp": {
      "command": "docker",
      "args": [
        "run", "--rm", "-i",
        "--volume", "/tmp/crawl4ai-crawls:/app/crawls",
        "uysalsadi/crawl4ai-mcp-server:latest"
      ],
      "env": {
        "CRAWL4AI_MCP_LOG": "INFO"
      }
    }
  }
}

FAQ

Is this server similar to Firecrawl?

Yes, it provides similar functionality to Firecrawl's API but is self-hosted and free, with no API key requirements.

What safety features does the server implement?

The server blocks internal networks, localhost, and private IPs by default to prevent scraping of sensitive or private content.

Compare crawl4ai-mcp-server with