rdp-cli
Android Virtual Display Automation for AI Agents
Android virtual display automation gives an AI agent a separate display target for operating and testing an Android application.
Instead of controlling only the phone’s physical screen, the agent can create a session-owned virtual display, set its size and DPI, launch an app and verify each action with a fresh screenshot.
With rdp-cli, an AI agent can connect to Android devices over USB or Wi-Fi through ADB, choose a virtual display, launch an installed app and follow a screenshot-based automation workflow.
Quick answer: Android virtual display automation lets an AI agent run GUI tasks on a separate Android display. The reliable workflow is: create the display, capture the screen, perform one action, capture again and verify the result.
Android Virtual Display vs Main Display
rdp-cli supports two Android display targets:
main: controls the phone’s existing physical display.virtual: creates a session-owned virtual display for the automation task.
A virtual display is useful when the agent needs a controlled screen size and DPI. The agent, GUI viewer and Web Viewer use the selected display for the session. Application behavior and system permissions can still vary by device and app.
A virtual display is not a cloud Android emulator. The application still runs on an authorized Android phone, and its behavior depends on the Android version, device manufacturer and application.
When Should You Use a Virtual Display?
Android virtual display automation is useful for:
- Repeating Android GUI tests
- Testing applications without a usable API
- Running forms, login flows and approval workflows
- Using a fixed screen size for visual testing
- Capturing screenshots as test evidence
- Running an app on an isolated display target
For basic device setup, first follow Control an Android phone with an AI agent over USB or Wi-Fi. After the device is authorized, continue with the virtual display workflow below.
Requirements
Prepare:
- Android 12 or later
- USB debugging or Wireless debugging enabled
- ADB authorization confirmed on the phone
- The target app already installed
- An Android-enabled rdp-cli build
- A screen size and DPI suitable for the test
rdp-cli lists Android 12+ as the support baseline. Google Pixel 3 running Android 12 is a verified reference device, but compatibility with other devices may vary.
Create a Virtual Android Display
After discovering and authorizing the device, create a virtual display session:
rdp-cli connect \
--protocol adb \
--device usb:DEVICE_ID \
--transport usb \
--android-display virtual \
--android-virtual-size 1080x1920 \
--android-virtual-dpi 420 \
--android-start-app com.example.app
Replace DEVICE_ID with the authorized device ID, 1080x1920 with the required display size, 420 with the required DPI and com.example.app with the installed app package name.
The command returns a session ID. Save it for later operations.
The available virtual display options are documented in the rdp-cli command reference.
Use the Observe, Act and Verify Workflow
Never send an action using an old screenshot.
Capture the current virtual display:
rdp-cli screenshot \
--session SESSION \
--out before.png
Inspect the screenshot, locate the target control and send one action using the latest observation:
rdp-cli android touch \
--session SESSION \
--kind tap \
--x X \
--y Y \
--observation OBSERVATION
Capture the display again:
rdp-cli screenshot \
--session SESSION \
--out after.png
The second screenshot confirms what actually happened. A successful input response only means that the command was dispatched; it does not prove that the app reached the expected state.
Transfer Test Files and Add Human Review
For Unicode or multiline text, use the Android clipboard method:
rdp-cli android text \
--session SESSION \
--method clipboard \
--text "Test order completed." \
--observation OBSERVATION
Files can be transferred through the session file root:
rdp-cli files upload \
--session SESSION \
--local ./test-data.json \
--path test-data.json
Download the processed result with rdp-cli files download, wait for the transfer to complete and verify the returned file.
When human review is needed, open the same session in a native window or authorized browser:
rdp-cli watch --session SESSION --viewer web
A human can pause agent input, handle a permission prompt or correct an unexpected screen, then return control to the agent. After a control change, the agent should capture a new screenshot before continuing.
Limitations
- ADB authorization is required.
- Device and Android-version compatibility may vary.
- Some applications may behave differently on a virtual display.
- System permissions and audio behavior still apply.
- Secure application windows may restrict screen capture.
- rdp-cli provides device control but does not include an AI model.
- The external AI agent remains responsible for planning and decision-making.
For critical applications, combine virtual display automation with application-level tests and screenshot verification.
FAQ
Does Android virtual display automation require root?
The documented rdp-cli workflow does not require root access. It requires an authorized ADB connection over USB or Wi-Fi.
Can I automate any Android app on a virtual display?
No. Some applications may depend on the physical display, device-specific services or system permissions. Test the target app before production use.
How can I verify that an Android automation action succeeded?
Capture a screenshot before the action, send one input command, capture a fresh screenshot and inspect the visible result. Command dispatch alone is not task completion.
Does rdp-cli include an AI agent?
No. rdp-cli provides the connection, screenshots, input commands, session control and structured results. An external AI agent decides what to do.
Final Takeaway
Android virtual display automation is useful when an AI agent needs a controlled display for GUI testing or repeated Android workflows.
The reliable pattern is:
Create the virtual display → capture the screen → perform one action → capture again → verify the result.
With rdp-cli, an agent can launch an Android app on a virtual display, interact with visible controls, transfer test files and pause for human review when necessary.