webclaw

Overview

WebClaw is a comprehensive web scraping tool that extracts clean content from websites and transforms it into markdown, JSON, and LLM-ready formats. Built in Rust for performance, it offers both local processing capabilities and a hosted API option. The project provides multiple interfaces including a CLI, REST API, and an MCP server for direct integration with AI agents. Its architecture separates core extraction logic from fetching layers, enabling flexibility for different use cases while maintaining high performance.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• scrape

  Extract one URL as markdown, text, JSON, LLM format, or HTML

  Try:Scrape this website for me in markdown formatTry:Extract content from https://example.com as clean text
• crawl

  Follow same-origin links and extract discovered pages

  Try:Crawl this documentation site and extract all pagesTry:Follow links from https://blog.example.com up to 3 levels deep
• map

  Discover URLs without extracting every page

  Try:Map out the structure of this websiteTry:Find all the URLs on this page without extracting content
• batch

  Scrape multiple URLs in parallel

  Try:Scrape these 10 URLs at onceTry:Extract content from these competitor websites in parallel
• extract

  Convert page content into structured data

  Try:Extract product information from this page as JSONTry:Parse this webpage and extract structured data
• summarize

  Summarize a page

  Try:Summarize this article for meTry:Create a brief summary of this documentation page
• diff

  Compare page content snapshots

  Try:Compare this pricing page with a previous versionTry:Show me what changed on this webpage since last time
• brand

  Extract colors, fonts, logos, and metadata

  Try:Extract the brand elements from this company websiteTry:Get the color scheme and logo from this webpage
• search

  Search the web and scrape results

  Try:Search for 'AI news' and scrape the resultsTry:Find and scrape information about the latest tech trends
• research

  Multi-source research workflow

  Try:Research the latest developments in AI across multiple sourcesTry:Conduct a comprehensive research on this topic

Comparable tools

Installation

npx create-webclaw

brew tap 0xMassi/webclaw
brew install webclaw

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

{
  "mcpServers": {
    "webclaw": {
      "command": "~/.webclaw/webclaw-mcp"
    }
  }
}

FAQ

Does WebClaw require an API key for basic use?

No, the CLI and MCP server work locally without an account for core extraction. The API key is only needed when using the hosted service.

Can WebClaw handle JavaScript-rendered content?

The local version doesn't execute JavaScript, but the hosted API at webclaw.io does for dynamic content when needed.

Compare webclaw with