rdp-cli

Your AI Agent Can Write Code. Can It Use a Real Desktop? A Computer Use Agent Guide

A coding agent can edit files, run tests and create pull requests. But many important business tasks still happen inside graphical applications.

They may involve Excel, ERP software, accounting tools, PDF applications, legacy Windows programs or Android apps.

So, can an AI agent control a real desktop instead of only working in a chat window or terminal?

Yes. A computer use agent can inspect a screen, click controls, type text, handle dialogs and verify visible results. With rdp-cli, the agent can perform these actions on an authorized RDP, VNC, X11 or Android session without taking over your local computer.

What Is a Computer Use Agent?

A computer use agent operates software through its graphical interface. It uses screenshots and input actions instead of relying only on APIs or command-line tools.

It can:

  • Read screenshots
  • Click buttons and menus
  • Type into forms
  • Scroll through windows
  • Use keyboard shortcuts
  • Open files and dialogs
  • Verify visible results

This is the foundation of AI desktop automation and AI GUI automation.

The basic workflow is: observe the current screen, perform one action, capture a fresh screenshot and verify the result.

The AI model supplies the reasoning. rdp-cli supplies the remote session and desktop control.

Why Does an AI Agent Need a Real Desktop?

APIs and MCP tools are useful when an application exposes structured operations. However, many desktop workflows depend on interface elements that are not available through an API.

For example, an agent may need to open a Windows application, sign in through a native dialog, select a value, confirm a warning, export a report and check that the output file exists.

These steps depend on windows, buttons, focus, pop-ups and file dialogs. A computer use agent can interact with them like a human operator.

Where rdp-cli Fits: The Missing Session Layer

Computer-use workflows contain different layers:

Layer Main responsibility What it does not solve
Model-native computer use Understand screenshots and choose actions Remote connection, session ownership and human handoff
RPA Execute predefined selectors or workflows Flexible visual reasoning and remote session lifecycle
MCP client Expose desktop actions as tools The complete lifecycle of every remote session
rdp-cli Connect, maintain and share remote sessions AI reasoning and task planning

Model-native computer use is the reasoning layer. It can interpret a screenshot and propose a click or keystroke, but it does not necessarily provide a connection to an existing RDP target, session recovery, file transfer or human control of the same session.

RPA remains useful for stable, repetitive and high-volume workflows. It is less suitable when the interface changes, the application has no API or a human must intervene.

MCP provides an agent-to-tool interface. An MCP client can expose actions such as screenshot, click, type and wait. But MCP itself does not define how a remote session is created, maintained, shared or handed back to a person.

rdp-cli is the session layer. It connects an agent to an existing authorized environment and lets the agent and human operator work on the same session.

It provides:

  • RDP, VNC, X11 and Android connections
  • Session IDs for addressing the correct remote desktop
  • Headless operation without opening a local viewer
  • Screenshots and observation-based input
  • File transfer on supported protocols
  • State waiting and result verification
  • GUI or Web Viewer access to the same session
  • Pause, human takeover and handoff back to the agent

rdp-cli uses the same session-oriented CLI model across RDP, VNC, X11 and Android ADB. Each connection still has its own session ID; the benefit is a consistent control pattern across different target types.

How AI GUI Automation Works with rdp-cli

The agent should always use a fresh screenshot before sending input.

Capture the screen with rdp-cli screenshot --session SESSION --out before.png. After locating the target control, send one action with rdp-cli click --session SESSION --desktop --x X --y Y --observation OBSERVATION. Then capture the screen again with rdp-cli screenshot --session SESSION --out after.png.

The second screenshot verifies the result. A successful input command only confirms that the action was sent; it does not prove that the application completed the operation.

After clicking Save, the agent should check for a confirmation message, a new record number, an updated status, a changed table row or a generated file. See the rdp-cli command documentation for current syntax.

What Can a Computer Use Agent Control?

RDP-based AI desktop automation is useful for legacy ERP systems, accounting software, warehouse applications, Excel workflows, PDF tools, internal business applications, desktop testing and installers.

With X11, the agent can interact with supported Linux graphical applications. VNC provides screen, keyboard and mouse control for an existing VNC desktop. With ADB, rdp-cli can control authorized Android devices over USB or Wi-Fi.

See the rdp-cli Quick Start Guide for setup details.

Can an Agent Control a Computer Without Taking Over Your PC?

Yes. The agent can operate a separate remote session while you continue using your local computer. You can open the same session through the GUI or authorized Web Viewer only when you need to inspect or control it.

For uncertain states, a safe handoff is: pause agent input, take control, correct the desktop, release control and require the agent to capture a fresh screenshot. Read Watch, Pause and Take Over an AI Agent’s Remote Desktop for details.

What rdp-cli Does Not Do

rdp-cli does not provide the AI model, decide which action to perform or create a remote computer automatically. You need an existing authorized device, the required credentials and an AI agent that can inspect screenshots and run commands.

Protocol capabilities depend on the server. For example, RDP may support file transfer and multiple displays when enabled by the remote server. rdp-cli also does not replace unit tests, API tests or backend validation.

FAQ

What is a computer use agent?

A computer use agent is an AI system that operates software through a graphical interface using screenshots, mouse actions, keyboard input and visual verification.

Can AI agents control Windows desktop applications?

Yes. An AI agent can control Windows applications through an authorized RDP session, including legacy software without an API or SDK.

Does rdp-cli include an AI agent?

No. rdp-cli provides the remote session, screenshots, input, file transfer and control handoff. Your selected AI agent provides the reasoning.

Does rdp-cli require MCP?

No. Sessions can be driven entirely through the rdp-cli CLI. Agents that support MCP can also use the available MCP interface.

Why use rdp-cli instead of model-native computer use?

Use rdp-cli when the target desktop already exists remotely, must remain separate from the user’s computer or needs shared access between an AI agent and a human operator.

Why use rdp-cli instead of an MCP-native remote desktop client?

An MCP-native client can be a reasonable fit when the agent works entirely through MCP and needs one supported protocol. rdp-cli is useful when shell scripts, CI jobs or non-MCP orchestration must control existing RDP, VNC, X11 and Android ADB sessions through a consistent session-oriented CLI model.

Final Takeaway

A model that can write code is useful. A computer use agent that can operate a real desktop can complete workflows that APIs and terminals cannot reach.

The unique role of rdp-cli is not AI reasoning or predefined automation. It is the session layer between the agent and the real remote environment.

With rdp-cli, an agent can connect to an existing desktop, work headlessly, verify screenshots, transfer files and hand the same session to a human when needed.

Download rdp-cli and give your AI agent controlled access to the desktop workflows where the real work happens.