Crawl4ai Rag
@coleam00
About Crawl4ai Rag
No overview available yet
Config
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"crawl4ai-rag": {
"command": "python",
"args": [
"path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
],
"env": {
"TRANSPORT": "stdio",
"OPENAI_API_KEY": "your_openai_api_key",
"SUPABASE_URL": "your_supabase_url",
"SUPABASE_SERVICE_KEY": "your_supabase_service_key",
"USE_KNOWLEDGE_GRAPH": "false",
"NEO4J_URI": "bolt://localhost:7687",
"NEO4J_USER": "neo4j",
"NEO4J_PASSWORD": "your_neo4j_password"
}
}
}
}Tools
No tools detected
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is Crawl4ai Rag?
A Model Context Protocol (MCP) server that integrates Crawl4AI web crawling with Supabase vector storage to provide AI agents and coding assistants with web crawling and RAG (Retrieval-Augmented Generation) capabilities.
How to use Crawl4ai Rag?
Install via Docker or uv (Python 3.12+), configure a .env file with Supabase and OpenAI API key, set up a Supabase database with pgvector, and optionally set up Neo4j for knowledge graph features. The server exposes MCP tools for crawling, searching, and querying crawled content.
Key features of Crawl4ai Rag
- Smart URL detection (regular pages, sitemaps, text files)
- Recursive crawling following internal links
- Parallel processing for efficient multi-page crawling
- Content chunking by headers and size
- Vector search with optional source filtering
- Advanced RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph)
Use cases of Crawl4ai Rag
- AI coding assistants scraping and indexing documentation for code-aware RAG
- Building a knowledge engine for AI agents to crawl and query websites
- Performing semantic search over previously crawled content with source filtering
- Detecting AI hallucinations by validating code against a repository knowledge graph
FAQ from Crawl4ai Rag
What dependencies does Crawl4ai Rag require?
Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.
How are RAG strategies configured?
Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a USE_* environment variable set to "true" in the .env file. All default to "false".
Where is crawled content stored?
Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.
What transport does the server use?
The server uses SSE (Server-Sent Events) transport by default, configurable via the TRANSPORT environment variable.
Are there known limitations or planned improvements?
The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.
Frequently asked questions
What dependencies does Crawl4ai Rag require?
Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.
How are RAG strategies configured?
Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a `USE_*` environment variable set to `"true"` in the `.env` file. All default to `"false"`.
Where is crawled content stored?
Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.
What transport does the server use?
The server uses SSE (Server-Sent Events) transport by default, configurable via the `TRANSPORT` environment variable.
Are there known limitations or planned improvements?
The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.
Basic information
More Memory & Knowledge MCP servers
Memory Bank MCP Server
alioshrA Model Context Protocol (MCP) server implementation for remote memory bank management, inspired by Cline Memory Bank.
Anytype MCP Server
anyprotoAn MCP server enabling AI assistants to interact with Anytype - your encrypted, local and collaborative wiki - to organize objects, lists, and more through natural language.
Mcp Knowledge Graph
shanehollomanMCP server enabling persistent memory for Claude through a local knowledge graph - fork focused on local development
ctxfile
ctxfileLocal-first MCP server that snapshots your project's working state into one context object. Open-core, privacy-first.
The Cloud
TekaThe Cloud is the AI-native workspace that gives every AI — Claude, ChatGPT, Gemini, and other Frontier & Free or Local models — shared memory and real agents. Not cloud computing: thecloud.so is one workspace where your
Comments