Android Use

Overview

This skill enables Claude to control Android devices using ADB (Android Debug Bridge) and the Android Accessibility API (uiautomator). It works by capturing the device's UI hierarchy as structured XML text, parsing interactive elements with their coordinates, and executing actions like tapping, typing, swiping, and navigation.

IMPORTANT: This skill uses TEXT-BASED UI automation, NOT image/screenshot processing.

Why Text-Based (Not Screenshots)?

Approach	Cost	Speed	Accuracy
Screenshots + Vision	~$0.15/action	3-5s	70-80%
Text UI Dump	~$0.01/action	<1s	99%+

The Android Accessibility API provides structured XML with:

Element text content (exact, no OCR errors)
Precise coordinates (deterministic)
Clickable/focusable state
Resource IDs for identification

Only use screenshots as a LAST RESORT when:

UI text is ambiguous or unclear
You need to verify visual state (colours, images)
Text dump is empty but screen clearly has content

Prerequisites

Before using this skill, ensure:

ADB installed: brew install android-platform-tools (macOS) or equivalent
Device connected: Via USB with "USB debugging" enabled in Developer Options
Device authorised: Accept the RSA key prompt on the device when first connecting

To verify connection:

adb devices -l

Quick Start Workflow

To control an Android device:

Get screen state - Dump UI hierarchy as text
Parse elements - Extract interactive components with coordinates
Decide action - Based on text content, find target element
Execute action - Tap, type, swipe, back, home
Wait & repeat - Allow UI to update, then reassess

Core Commands

Device Management

# List all connected devices
adb devices -l

# Target specific device (when multiple connected)
adb -s <DEVICE_ID> <command>

UI Inspection (Text-Based)

# Dump UI hierarchy to device
adb -s <DEVICE_ID> shell uiautomator dump /sdcard/window_dump.xml

# Pull to local machine
adb -s <DEVICE_ID> pull /sdcard/window_dump.xml /tmp/screen.xml

# Parse and extract interactive elements
python3 scripts/parse_ui.py /tmp/screen.xml

One-liner for convenience:

adb -s <DEVICE_ID> shell uiautomator dump /sdcard/window_dump.xml && \
adb -s <DEVICE_ID> pull /sdcard/window_dump.xml /tmp/screen.xml && \
python3 scripts/parse_ui.py /tmp/screen.xml

The parser outputs interactive elements like:

TAPPABLE:
  👆 "Submit" @ (540, 1200)
  👆 "Cancel" @ (200, 1200)
  👆 [search_button] @ (980, 156)

INPUT FIELDS:
  ⌨️ "Enter email" @ (540, 600)

TEXT/INFO:
  👁️ "Welcome back, John" @ (540, 300)

Actions

# Tap at coordinates
adb shell input tap <x> <y>

# Type text (use %s for spaces)
adb shell input text "hello%sworld"

# Press keys
adb shell input keyevent KEYCODE_HOME      # Home button
adb shell input keyevent KEYCODE_BACK      # Back button
adb shell input keyevent KEYCODE_ENTER     # Enter key
adb shell input keyevent KEYCODE_DEL       # Backspace

# Swipe (x1, y1 to x2, y2 over duration_ms)
adb shell input swipe <x1> <y1> <x2> <y2> <duration_ms>

# Long press (swipe with same start/end)
adb shell input swipe <x> <y> <x> <y> 1000

App Management

# Launch app by package name (simplest)
adb shell monkey -p <package> -c android.intent.category.LAUNCHER 1

# Launch app with specific activity
adb shell am start -n <package>/<activity>

# List installed packages
adb shell pm list packages | grep <search>

# Force stop app
adb shell am force-stop <package>

Standard Workflow

Step 1: Get Current Screen State (TEXT ONLY)

adb -s <DEVICE_ID> shell uiautomator dump /sdcard/window_dump.xml && \
adb -s <DEVICE_ID> pull /sdcard/window_dump.xml /tmp/screen.xml && \
python3 scripts/parse_ui.py /tmp/screen.xml

This gives you a text list of all interactive elements with coordinates.

Step 2: Find Target Element

From the parsed output, identify the element you need:

Match by text content: "Submit", "Login", "Search"
Match by resource ID: [login_button], [search_field]
Match by type: Button, EditText, TextView

Step 3: Execute Action

# To tap "Submit" button at (540, 1200)
adb -s <DEVICE_ID> shell input tap 540 1200

# To type in a field - first tap it, then type
adb -s <DEVICE_ID> shell input tap 540 600
adb -s <DEVICE_ID> shell input text "user@email.com"

Step 4: Wait and Repeat

# Wait 1-2 seconds for UI to update
sleep 2

# Dump again and reassess
adb -s <DEVICE_ID> shell uiautomator dump /sdcard/window_dump.xml && \
adb -s <DEVICE_ID> pull /sdcard/window_dump.xml /tmp/screen.xml && \
python3 scripts/parse_ui.py /tmp/screen.xml

When to Use Screenshots (LAST RESORT ONLY)

Only take a screenshot if:

UI dump returns empty/incomplete but you know the screen has content
You need to verify something visual (image loaded, colour state)
Text elements are ambiguous and you need visual context
User explicitly asks to "see" the screen

# Screenshot command (USE SPARINGLY - costs ~15x more to process)
adb -s <DEVICE_ID> shell screencap -p /sdcard/screen.png && \
adb -s <DEVICE_ID> pull /sdcard/screen.png /tmp/screen.png

Common Patterns

Opening an App and Navigating

# 1. Launch app
adb -s DEVICE shell monkey -p com.example.app -c android.intent.category.LAUNCHER 1

# 2. Wait for launch
sleep 2

# 3. Get UI state (text)
adb -s DEVICE shell uiautomator dump /sdcard/window_dump.xml
adb -s DEVICE pull /sdcard/window_dump.xml /tmp/screen.xml
python3 scripts/parse_ui.py /tmp/screen.xml

# 4. Find and tap target element based on text output
adb -s DEVICE shell input tap <x> <y>

Searching in an App

# 1. Find search field in UI dump
# Output shows: ⌨️ "Search" @ (540, 200)

# 2. Tap search field
adb shell input tap 540 200

# 3. Type search query
adb shell input text "cheesecake"

# 4. Press enter
adb shell input keyevent KEYCODE_ENTER

Scrolling to Find Elements

If an element isn't visible in the UI dump, scroll and re-dump:

# Scroll down
adb shell input swipe 540 1500 540 500 300

# Wait and re-dump
sleep 1
adb shell uiautomator dump /sdcard/window_dump.xml
# ... parse again

Handling Popups/Dialogs

Popups appear in UI dump with their own elements:

👆 "Allow" @ (700, 1200)
👆 "Deny" @ (300, 1200)

Just tap the appropriate button coordinates.

Parser Output Reference

The parse_ui.py script outputs elements grouped by action type:

TAPPABLE (clickable=true)

Elements that respond to taps: buttons, links, icons

👆 "Button Text" @ (x, y)
👆 [resource_id] @ (x, y)

INPUT FIELDS (EditText)

Text input fields

⌨️ "Placeholder text" @ (x, y)

TEXT/INFO (readable)

Non-interactive text elements (limited to 10)

👁️ "Display text" @ (x, y)

JSON Output

For programmatic use:

python3 scripts/parse_ui.py /tmp/screen.xml --json

Returns:

[
  {
    "id": "com.app:id/submit_btn",
    "text": "Submit",
    "type": "Button",
    "bounds": "[400,1100][680,1300]",
    "center": [540, 1200],
    "clickable": true,
    "action": "tap"
  }
]

Troubleshooting

"error: device not found"

Check USB connection
Run adb devices to verify
Try adb kill-server && adb start-server

UI dump returns empty/incomplete

Screen may be loading - wait and retry
Some apps have accessibility restrictions
Only then consider a screenshot to diagnose

Taps not registering

Verify coordinates are within screen bounds
Check if element is actually clickable
Some overlays may intercept touches

Text input issues

Replace spaces with %s
Special characters may need escaping
For passwords, some apps block programmatic input

Element not found in dump

It may be off-screen - try scrolling
It may be in a WebView (limited accessibility)
It may be loading - wait and retry

Reference

Common Package Names

App	Package
Uber Eats	com.ubercab.eats
DoorDash	com.dd.dasher
Chrome	com.android.chrome
Settings	com.android.settings
Messages	com.google.android.apps.messaging

Common Keycodes

Key	Code
Home	KEYCODE_HOME
Back	KEYCODE_BACK
Enter	KEYCODE_ENTER
Delete	KEYCODE_DEL
Tab	KEYCODE_TAB
Escape	KEYCODE_ESCAPE

Resources

scripts/parse_ui.py - Parses Android UI XML and outputs interactive elements
Based on android-action-kernel approach

android-use