OmniMCP

Overview

OmniMCP is a MCP server that bridges AI models with user interfaces by using Microsoft OmniParser for visual analysis. It implements a perceive-plan-act loop where the system captures screenshots, plans actions using LLMs, and executes precise mouse/keyboard inputs. The server supports both real UI interactions and synthetic UI simulations, with optional auto-deployment to AWS EC2 and comprehensive debugging capabilities.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• capture_screen

  Captures the current screen state for UI analysis

  Try:Take a screenshot of my current screenTry:Show me what's on the display
• parse_ui

  Analyzes UI elements using OmniParser to understand the interface

  Try:Identify all clickable elements on this screenTry:Describe what UI elements are visible
• execute_action

  Performs mouse or keyboard actions on UI elements

  Try:Click on the calculator buttonTry:Type 'Hello World' in the text field
• deploy_omniparser

  Deploys OmniParser server on AWS EC2 with auto-shutdown

  Try:Start the OmniParser server on AWSTry:Deploy the parser to the cloud
• stop_omniparser

  Stops the deployed OmniParser server and cleans up AWS resources

  Try:Stop the OmniParser serverTry:Clean up the cloud deployment
• ui_interaction

  Performs a complete perceive-plan-act cycle for UI interaction

  Try:Calculate 5*9 using the calculator appTry:Log into my account on this website

Note: Tools were inferred from code architecture descriptions and functionality mentions rather than an explicit MCP tools section. The experimental MCP server exists but no specific MCP tools are documented in the README.

Comparable tools

Installation

# Clone and install
git clone https://github.com/OpenAdaptAI/OmniMCP.git
cd OmniMCP
./install.sh

# Configure environment
cp .env.example .env
# Edit .env with your API keys

# Activate environment
source .venv/bin/activate

{
  "mcpServers": {
    "omnimcp": {
      "command": "uv",
      "args": ["run", "python", "path/to/omnimcp/mcp_server.py"]
    }
  }
}

FAQ

What operating systems are supported?

Currently supports Linux with X11/Wayland graphical sessions. macOS support is partially implemented with display scaling dependencies handled automatically. Windows support is not explicitly mentioned in the documentation.

How does the MCP server relate to the main CLI functionality?

The MCP server (OmniMCP class in omnimcp/mcp_server.py) is experimental and separate from the primary cli.py/AgentExecutor workflow. The main CLI provides a complete perceive-plan-act loop while the MCP server is intended for integration with other MCP-compatible systems.

Compare OmniMCP with