llm-app-exploration

Overview

This repository presents a pattern for LLM agents to systematically explore any application (mobile or desktop) through device access. The approach uses accessibility APIs rather than vision-based methods, treating apps as graphs where each screen is a node and interactive elements are edges. The process involves three phases: exploration (capturing every screen and element), organization (building route maps and transition tables), and analysis (using the complete data for user flows, competitive analysis, and UX audits). Two MCP servers implement this pattern: mobile-mcp for Android/iOS and desktop-mcp for macOS native apps and Electron applications.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• mobile_do

  Reads the screen and executes actions on mobile devices using accessibility tree

  Try:Tap the button labeled 'Save'Try:Open the dropdown menu and select 'Option 1'Try:Scroll to the bottom of the list and capture all elements
• desktop_do

  Reads the screen and executes actions on desktop apps via accessibility API or Chrome DevTools Protocol

  Try:Click on the menu item labeled 'File'Try:Type 'test@example.com' in the email input fieldTry:Switch to the settings app and find the network preferences

Comparable tools

Installation

{
  "mcpServers": {
    "mobile": {
      "command": "node",
      "args": ["/path/to/mobile-mcp/index.js"]
    }
  }
}

{
  "mcpServers": {
    "desktop": {
      "command": "node",
      "args": ["/path/to/desktop-mcp/index.js"]
    }
  }
}

FAQ

What platforms are supported?

The pattern supports mobile (Android, iOS) and desktop (macOS native apps, Electron apps). Each platform has its own MCP server implementation.

How does the element identification work?

Elements are identified with stable IDs in the format 'TYPE:Label' (e.g., 'BUTTON:Save'). This format is consistent across sessions and platforms, making targeting deterministic.

Compare llm-app-exploration with