mcp-apache-spark-history-server

概述

这是一个双接口工具包，既为 AI 代理提供 MCP 服务器，也为工程师提供 CLI 工具来交互 Spark 历史服务器数据。MCP 服务器提供 21 个工具用于 Spark 应用程序调查，包括作业分析、阶段指标、执行器信息和 SQL 查询分析。它支持多个 Spark 历史服务器配置，并提供比较分析和性能瓶颈检测功能。该项目由 Kubeflow 积极维护，定期更新并拥有全面的文档。

试试问 AI

装完之后，这里有 5 个你可以让 AI 做的事：

• list_applications

  List applications with optional status, date, and limit filters

  试试:Show me all running Spark applications试试:List failed applications from yesterday
• get_application

  Get application detail: status, resources, duration, attempts

  试试:Get details for application 123试试:Show me resource usage for app 456
• list_jobs

  List jobs with status filtering

  试试:List all failed jobs in application 123试试:Show running jobs for app 456
• list_slowest_jobs

  Top N slowest jobs

  试试:Show me the 5 slowest jobs试试:Find the slowest jobs in application 123
• list_stages

  List stages with status filtering

  试试:List failed stages in application 123试试:Show all completed stages for app 456
• get_stage

  Stage detail with attempt and summary metrics

  试试:Get details for stage 7 in application 123试试:Show metrics for stage 5 attempt 2
• get_executor

  Executor detail: resources, task stats, performance

  试试:Show details for executor 3 in application 123试试:Get resource usage for executor 5
• get_sql_execution

  SQL execution detail with optional plan and node metrics

  试试:Show details for SQL query 456试试:Get execution plan for query 789
• compare_job_performance

  Diff performance metrics between two applications

  试试:Compare performance of app 123 with app 456试试:Show differences between yesterday's and today's run
• get_job_bottlenecks

  Identify bottlenecks across stages, tasks, and executors

  试试:Find bottlenecks in application 123试试:What's slowing down job 456?
• aws_analyze_spark_workload

  One-shot root cause analysis of failed/slow Spark workloads

  试试:Analyze why job 123 failed on AWS试试:Diagnose slow performance on EMR
• list_slowest_sql_queries

  Top N slowest SQL executions with metrics

  试试:Show me the 10 slowest SQL queries试试:Find slowest queries in application 123

可对比工具

安装

pip install mcp-apache-spark-history-server
spark-mcp

uvx --from mcp-apache-spark-history-server spark-mcp

servers:
  local:
    default: true
    url: "http://your-spark-history-server:18080"
    auth:
      username: "user"
      password: "pass"
mcp:
  transports:
    - streamable-http
  port: "18888"

{
  "mcpServers": {
    "spark": {
      "command": "spark-mcp",
      "args": []
    }
  }
}

FAQ

MCP 服务器和 CLI 工具有什么区别？

MCP 服务器专为 AI 代理设计，使用自然语言与 Spark 历史服务器交互，而 CLI (shs) 是一个独立的 Go 二进制文件，用于直接终端访问、脚本和 CI/CD 管道。

是否支持 AWS EMR 和 Glue？

是的，该项目包含针对 AWS Glue 和 Amazon EMR 集成的具体示例，以及可选的 AWS Spark 故障排除功能，用于根本原因分析和代码建议。

mcp-apache-spark-history-server 对比