mcp-apache-spark-history-server

Overview

This is a dual-interface toolkit that provides both an MCP Server for AI agents and a CLI tool for engineers to interact with Spark History Server data. The MCP Server exposes 21 tools for Spark application investigation, including job analysis, stage metrics, executor information, and SQL query analysis. It supports multiple Spark History Server configurations and provides features for comparative analysis and performance bottleneck detection. The project is actively maintained by Kubeflow with regular updates and comprehensive documentation.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• list_applications

  List applications with optional status, date, and limit filters

  Try:Show me all running Spark applicationsTry:List failed applications from yesterday
• get_application

  Get application detail: status, resources, duration, attempts

  Try:Get details for application 123Try:Show me resource usage for app 456
• list_jobs

  List jobs with status filtering

  Try:List all failed jobs in application 123Try:Show running jobs for app 456
• list_slowest_jobs

  Top N slowest jobs

  Try:Show me the 5 slowest jobsTry:Find the slowest jobs in application 123
• list_stages

  List stages with status filtering

  Try:List failed stages in application 123Try:Show all completed stages for app 456
• get_stage

  Stage detail with attempt and summary metrics

  Try:Get details for stage 7 in application 123Try:Show metrics for stage 5 attempt 2
• get_executor

  Executor detail: resources, task stats, performance

  Try:Show details for executor 3 in application 123Try:Get resource usage for executor 5
• get_sql_execution

  SQL execution detail with optional plan and node metrics

  Try:Show details for SQL query 456Try:Get execution plan for query 789
• compare_job_performance

  Diff performance metrics between two applications

  Try:Compare performance of app 123 with app 456Try:Show differences between yesterday's and today's run
• get_job_bottlenecks

  Identify bottlenecks across stages, tasks, and executors

  Try:Find bottlenecks in application 123Try:What's slowing down job 456?
• aws_analyze_spark_workload

  One-shot root cause analysis of failed/slow Spark workloads

  Try:Analyze why job 123 failed on AWSTry:Diagnose slow performance on EMR
• list_slowest_sql_queries

  Top N slowest SQL executions with metrics

  Try:Show me the 10 slowest SQL queriesTry:Find slowest queries in application 123

Comparable tools

Installation

pip install mcp-apache-spark-history-server
spark-mcp

uvx --from mcp-apache-spark-history-server spark-mcp

servers:
  local:
    default: true
    url: "http://your-spark-history-server:18080"
    auth:
      username: "user"
      password: "pass"
mcp:
  transports:
    - streamable-http
  port: "18888"

{
  "mcpServers": {
    "spark": {
      "command": "spark-mcp",
      "args": []
    }
  }
}

FAQ

What's the difference between the MCP Server and the CLI tool?

The MCP Server is designed for AI agents to interact with Spark History Server using natural language, while the CLI (shs) is a standalone Go binary for direct terminal access, scripts, and CI/CD pipelines.

Does this support AWS EMR and Glue?

Yes, the project includes specific examples for AWS Glue and Amazon EMR integration, plus optional AWS Spark troubleshooting features for root cause analysis and code recommendations.

Compare mcp-apache-spark-history-server with