ingero

Overview

Ingero traces the full chain from Linux kernel events through CUDA API calls to Python source lines with less than 2% overhead. It correlates CPU scheduler events, CUDA Runtime API calls, and CUDA Driver API calls by timestamp and PID, then generates causal chain explanations. The MCP server component allows AI assistants like Claude Code and Cursor to ask natural language questions about GPU performance issues and receive resolved causal chains with Python source lines, no SQL or hex addresses required.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• mcp

  Expose Ingero as a Model Context Protocol server for AI-assisted GPU observability

  Try:What caused the GPU stall?Try:Explain the GPU performance issuesTry:Analyze the CUDA events from the last 5 minutes
• trace

  Trace GPU workloads and capture CUDA events in real-time

  Try:Start tracing the GPU eventsTry:Capture CUDA API calls for PID 4821Try:Begin live event stream monitoring
• explain

  Generate an automated diagnosis of GPU performance issues

  Try:Explain the GPU issues from the last 5 minutesTry:Analyze the causal chain of GPU eventsTry:Provide a root cause analysis for GPU stalls
• check

  Verify system readiness for GPU tracing

  Try:Check if the system is ready for GPU tracingTry:Verify eBPF kernel supportTry:Check if Ingero can trace the GPU
• query

  Query the GPU trace data for specific time periods

  Try:Query GPU events from the last hourTry:Show me high-latency CUDA operationsTry:Find GPU preemption events
• demo

  Run demonstration scenarios with synthetic data

  Try:Run the incident demo scenarioTry:Show a GPU performance issue demoTry:Run the demo without a GPU

Comparable tools

Installation

VERSION=0.16.0
curl -fsSL "https://github.com/ingero-io/ingero/releases/download/v${VERSION}/ingero_${VERSION}_linux_amd64.tar.gz" | tar xz
sudo mv ingero /usr/local/bin/

ingero mcp

{
  "mcpServers": {
    "ingero": {
      "command": "ingero",
      "args": ["mcp"]
    }
  }
}

FAQ

What does the MCP server expose?

The MCP server exposes tools for AI assistants to query GPU performance data, causal chains, and diagnostics through natural language questions.

What is the overhead of using Ingero?

Less than 2% overhead as measured on GPU hardware (0.4-1.7% across RTX 3090 through H100 with stack tracing).

Compare ingero with