mcp-server-webscan

Overview

The MCP Webscan Server is a TypeScript-based implementation that provides tools for fetching, analyzing, and extracting information from web pages. It features page fetching with Markdown conversion, link extraction with filtering options, recursive site crawling with depth control, broken link checking, pattern matching for URLs, and XML sitemap generation. The server runs on stdio transport, making it compatible with MCP clients like Claude Desktop. The codebase is well-structured with clear separation between services, tools, and types.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• fetch-pageurl: string, selector?: string

  Fetches a web page and converts it to Markdown

  Try:Fetch the content from https://example.com and convert it to MarkdownTry:Get the main content from https://news.ycombinator.com using the .post CSS selector
• extract-linksurl: string, baseUrl?: string, limit?: number

  Extracts all links from a web page with their text

  Try:Extract all links from https://example.comTry:Get links from https://example.com but only those from example.com domain, limit to 50
• crawl-siteurl: string, maxDepth?: number

  Recursively crawls a website up to a specified depth

  Try:Crawl https://example.com with a maximum depth of 2Try:Perform a deep crawl of https://blog.example.com up to depth 3
• check-linksurl: string

  Checks for broken links on a page

  Try:Check all links on https://example.com for broken onesTry:Verify if there are any dead links on my website https://myblog.com
• find-patternsurl: string, pattern: string

  Finds URLs matching a specific pattern

  Try:Find all product URLs matching the pattern /product/[0-9]+ on https://shop.example.comTry:Search for all email addresses on https://example.com using pattern [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
• generate-site-mapurl: string, maxDepth?: number, limit?: number

  Generates a simple XML sitemap by crawling

  Try:Generate a sitemap for https://example.com with maximum depth of 2Try:Create an XML sitemap for https://blog.example.com with a limit of 100 URLs

Comparable tools

Installation

npx -y @smithery/cli install mcp-server-webscan --client claude

# Clone the repository
git clone <repository-url>
cd mcp-server-webscan

# Install dependencies
npm install

# Build the project
npm run build

{
  "mcpServers": {
    "webscan": {
      "command": "node",
      "args": ["path/to/mcp-server-webscan/build/index.js"],
      "env": {
        "NODE_ENV": "development",
        "LOG_LEVEL": "info"
      }
    }
  }
}

FAQ

What is the maximum depth for site crawling?

The default max depth is 2, but it can be configured between 0-5 for the crawl-site and generate-site-map tools.

Can I filter links by base URL?

Yes, the extract-links tool accepts an optional baseUrl parameter to filter links.

Compare mcp-server-webscan with