How to Let an AI Agent Operate a Remote Windows Desktop

An AI agent can write scripts, but many real tasks still require a desktop: a Windows-only application, a document editor, or a file export dialog. The reliable way to automate that work is to make the agent follow a short feedback loop: observe the screen, perform one action, and verify the result.

1. Prepare the Windows host

Enable Remote Desktop on the target computer. Download the rdp-cli file for the computer running the agent, then check it in PowerShell:

.\rdp-cli.exe --help

The agent does not need to run on the Windows host. It only needs network access to the RDP service.

2. Connect without exposing the password

Pass the password through standard input instead of putting it in shell history:

$secret = Read-Host 'Password' -AsSecureString
try {
  [System.Net.NetworkCredential]::new('', $secret).Password |
    .\rdp-cli.exe connect --host 192.0.2.20 --user USER --password-stdin
} finally {
  $secret.Dispose()
}

Save the session value returned in JSON. It identifies the remote desktop for the next commands.

3. Observe before acting

Do not guess coordinates. Capture the current screen first:

.\rdp-cli.exe screenshot --session SESSION --out .\before.png

The agent can inspect the image, find the target control, and click it using the observation returned with the screenshot:

.\rdp-cli.exe click --session SESSION --desktop --x 420 --y 260 --observation OBSERVATION

4. Verify every important step

After a click, capture a fresh screen before typing or submitting anything:

.\rdp-cli.exe screenshot --session SESSION --out .\focused.png
.\rdp-cli.exe type --session SESSION --text "Remote task completed." --observation NEW_OBSERVATION
.\rdp-cli.exe screenshot --session SESSION --out .\after.png

This catches moved windows, loading states, and misdirected clicks. Never assume that a command worked just because it returned without an error.

5. Keep a human in control

A person can watch the same session in a window or browser:

.\rdp-cli.exe watch --session SESSION --viewer window

The viewer starts read-only and supports pausing the agent, taking control, and handing the session back. Use it for login prompts, confirmation dialogs, and irreversible actions.

Use RDP over a private network or VPN when possible, store credentials securely, and configure certificate validation for your environment. The practical pattern is simple: observe, act, verify.