crawl4ai-mcp

@wyattowalsh

About crawl4ai-mcp

MCP (Model Context Protocol) server wrapping Crawl4AI library as functions using Python

#python #agent #crawler #scraper #mcp #crawling #python3 #scrape #uv #uvx #agentic-workflow #agentic-ai #crawl4ai #model-context-protocol #mcp-servers #mcp-server #model-context-protocol-servers

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawl4ai-mcp": {
      "command": "docker",
      "args": [
        "build",
        "-t",
        "mcp-crawl4ai",
        "."
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is crawl4ai-mcp?

crawl4ai-mcp is a Model Context Protocol (MCP) server that provides AI systems with access to the live web. Built on FastMCP v3 and Crawl4AI, it exposes 4 tools, 2 resources, and 3 prompts through a standardized MCP interface, backed by a lifespan-managed headless Chromium browser.

How to use crawl4ai-mcp?

Install via pip install mcp-crawl4ai or uv add mcp-crawl4ai, then run mcp-crawl4ai for stdio transport (default) or mcp-crawl4ai --transport http --port 8000 for HTTP. For Claude Desktop, add to MCP settings with the command mcp-crawl4ai and args ["--transport", "stdio"].

Key features of crawl4ai-mcp

Full MCP compliance via FastMCP v3 with tool annotations
4 focused tools for scrape, crawl, session lifecycle, and artifacts
3 prompts for common LLM workflows (summarize, extract schema, compare)
2 resources exposing server configuration and version info
Headless Chromium managed as a lifespan singleton
Multiple transports: stdio (default) and Streamable HTTP
LLM-optimized output in markdown, cleaned HTML, raw HTML, or plain text
Auto browser setup and detection of missing Playwright browsers

Use cases of crawl4ai-mcp

AI agents that need to scrape or crawl web pages for real-time information
Summarizing a page with a specified focus using the summarize_page prompt
Building a CSS extraction schema for structured data from a URL
Comparing two web pages and producing a structured comparison
Deep traversal of a website using recursive BFS/DFS from a single seed

FAQ from crawl4ai-mcp

What is the difference between crawl4ai-mcp and using Crawl4AI directly?

crawl4ai-mcp wraps Crawl4AI in an MCP server with only 2 runtime dependencies (fastmcp and crawl4ai), providing a standardized interface for AI tools and LLM clients. It adds session management, resource endpoints, and prompt templates.

What are the runtime dependencies of crawl4ai-mcp?

The server requires Python 3.13+ and has exactly two runtime dependencies: fastmcp and crawl4ai. Playwright browsers are needed for headless Chromium but are installed automatically or via mcp-crawl4ai --setup.

Does crawl4ai-mcp require a browser to be installed?

Yes, it uses headless Chromium. The server auto-detects missing Playwright browsers and attempts to install them on first startup, or you can run mcp-crawl4ai --setup manually. The Docker image includes browsers pre-installed.

What transports does crawl4ai-mcp support?

Stdio (default) for desktop clients like Claude Desktop and Cursor, and HTTP (Streamable) for remote access. HTTP binds to 127.0.0.1:8000 by default; use --host and a reverse proxy for external exposure.

Are there any known limits in crawl4ai-mcp?

The scrape tool accepts up to 20 URLs in a bounded list. The crawl tool supports both list traversal (bounded) and deep traversal (recursive BFS/DFS) from a single seed. There are configurable limits exposed via the config://server resource.

Frequently asked questions

What is the difference between crawl4ai-mcp and using Crawl4AI directly?

crawl4ai-mcp wraps Crawl4AI in an MCP server with only 2 runtime dependencies (`fastmcp` and `crawl4ai`), providing a standardized interface for AI tools and LLM clients. It adds session management, resource endpoints, and prompt templates.

What are the runtime dependencies of crawl4ai-mcp?

The server requires Python 3.13+ and has exactly two runtime dependencies: `fastmcp` and `crawl4ai`. Playwright browsers are needed for headless Chromium but are installed automatically or via `mcp-crawl4ai --setup`.

Does crawl4ai-mcp require a browser to be installed?

Yes, it uses headless Chromium. The server auto-detects missing Playwright browsers and attempts to install them on first startup, or you can run `mcp-crawl4ai --setup` manually. The Docker image includes browsers pre-installed.

What transports does crawl4ai-mcp support?

Stdio (default) for desktop clients like Claude Desktop and Cursor, and HTTP (Streamable) for remote access. HTTP binds to `127.0.0.1:8000` by default; use `--host` and a reverse proxy for external exposure.

Are there any known limits in crawl4ai-mcp?

The `scrape` tool accepts up to 20 URLs in a bounded list. The `crawl` tool supports both list traversal (bounded) and deep traversal (recursive BFS/DFS) from a single seed. There are configurable limits exposed via the `config://server` resource.