webclaw

概述

WebClaw 是一个全面的网页抓取工具，可以从网站提取干净的内容并将其转换为 Markdown、JSON 和 LLM 就绪的格式。它使用 Rust 构建，具有高性能，同时提供本地处理功能和托管 API 选项。该项目提供了多种接口，包括 CLI、REST API 和直接与 AI 代理集成的 MCP 服务器。其架构将核心提取逻辑与获取层分离，为不同用例提供灵活性，同时保持高性能。

试试问 AI

装完之后，这里有 5 个你可以让 AI 做的事：

• scrape

  Extract one URL as markdown, text, JSON, LLM format, or HTML

  试试:Scrape this website for me in markdown format试试:Extract content from https://example.com as clean text
• crawl

  Follow same-origin links and extract discovered pages

  试试:Crawl this documentation site and extract all pages试试:Follow links from https://blog.example.com up to 3 levels deep
• map

  Discover URLs without extracting every page

  试试:Map out the structure of this website试试:Find all the URLs on this page without extracting content
• batch

  Scrape multiple URLs in parallel

  试试:Scrape these 10 URLs at once试试:Extract content from these competitor websites in parallel
• extract

  Convert page content into structured data

  试试:Extract product information from this page as JSON试试:Parse this webpage and extract structured data
• summarize

  Summarize a page

  试试:Summarize this article for me试试:Create a brief summary of this documentation page
• diff

  Compare page content snapshots

  试试:Compare this pricing page with a previous version试试:Show me what changed on this webpage since last time
• brand

  Extract colors, fonts, logos, and metadata

  试试:Extract the brand elements from this company website试试:Get the color scheme and logo from this webpage
• search

  Search the web and scrape results

  试试:Search for 'AI news' and scrape the results试试:Find and scrape information about the latest tech trends
• research

  Multi-source research workflow

  试试:Research the latest developments in AI across multiple sources试试:Conduct a comprehensive research on this topic

可对比工具

安装

npx create-webclaw

brew tap 0xMassi/webclaw
brew install webclaw

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

{
  "mcpServers": {
    "webclaw": {
      "command": "~/.webclaw/webclaw-mcp"
    }
  }
}

FAQ

WebClaw 是否需要 API 密钥才能进行基本使用？

不需要，CLI 和 MCP 服务器可以在本地使用，无需账户即可进行核心提取。仅在使用托管服务时才需要 API 密钥。

WebClaw 能否处理 JavaScript 渲染的内容？

本地版本不执行 JavaScript，但当需要时，webclaw.io 的托管 API 可以处理动态内容。

webclaw 对比