rdp-cli

AI Agent Desktop Automation with RDP Screenshots and Input

Applies to: 0.1.2 · Windows / macOS · RDP

Start with the result you want

Give your agent the executable location, an authorized target or existing session, and a concrete outcome: for example, enter a short note in a remote editor and verify that it appears. The tool does not contain an AI model; your agent decides the next step from the remote screen.

First connect to the server and save data.session. All commands below use SESSION as a placeholder for this returned value.

Look before sending input

For Windows PowerShell, run these examples from the executable’s folder and replace rdp-cli with .\rdp-cli.exe.

For the Calc example below, the remote Windows computer needs LibreOffice Calc and RDP drive redirection. Download the sample workbook locally as service-activity.xlsx. Upload it, save data.transfer_id as TRANSFER_ID, and wait for the transfer’s completed state. Then open that file from the redirected drive in remote Calc and select A20. The full Calc tutorial explains the workbook and expected results.

rdp-cli files upload --session SESSION --local ./service-activity.xlsx --path service-activity.xlsx
rdp-cli files status --session SESSION --transfer-id TRANSFER_ID
rdp-cli screenshot --session SESSION --out before.png

Open the PNG and inspect the desktop. Save data.observation from the response and use it for the next input. Coordinates refer to the remote desktop, not a local viewer window. If you need to focus a field, choose its location from this image:

rdp-cli click --session SESSION --desktop --x 420 --y 315 --observation OBSERVATION

The numbers are only an example. Read the screen and choose actual coordinates. Capture again if the application moves or a new dialog appears.

Enter a note and verify it

In the example below, open the prepared workbook in remote Calc and select cell A20. Inspect a fresh screenshot and use its observation token. Paste the completion note, save with the application’s Save button, and verify the displayed result.

Use data.observation from before.png’s response as OBSERVATION. After clicking, inspect clipboard-ready.png and use its response’s data.observation as NEW_OBSERVATION. Finally inspect result.png to verify the text; a successful command alone does not prove the application accepted it.

Choose a unique PASTE_REQUEST for this paste operation and keep it with the response. If it times out, query that same ID; do not create a new ID and paste again before checking the screen.

PNG data.observation
note-focused.png FOCUSED_OBSERVATION
clipboard-ready.png NEW_OBSERVATION
rdp-cli screenshot --session SESSION --out note-focused.png
rdp-cli clipboard set --session SESSION --text "Result: report completed and ready for review." --observation FOCUSED_OBSERVATION
rdp-cli screenshot --session SESSION --out clipboard-ready.png
rdp-cli key --session SESSION --keys CTRL+V --observation NEW_OBSERVATION --request-id PASTE_REQUEST
rdp-cli screenshot --session SESSION --out result.png

A remote Calc report saved after rdp-cli pasted the completion note.

The Calc example uses fictional service data. The completed workbook below corresponds to the selected language and theme.

Download the result shown in the screenshots: Light theme · Dark theme

A remote Calc report saved after rdp-cli pasted the completion note.

View original screenshot — Light ↗ · View original screenshot — Dark ↗

Recover without duplicating work

If an observation expires, capture and inspect again. If a person pauses or takes control, stop sending input. After reconnect or handoff, use a new screenshot instead of an old token.

If an operation times out, query its original request or operation before retrying:

rdp-cli operations status --session SESSION --request-id PASTE_REQUEST

An uncertain result does not prove nothing happened. The same caution applies to file transfers: query files status and verify the completed result. Do not repeat an upload just because its first reply was lost.

Keep the local desktop available

Normal screenshot and input commands travel through the remote protocol. They do not need a local viewer or move your local pointer. A person can still open the session to watch or take over. See the feature guide for clipboard, files and protocol conditions.

Try it on your remote desktop

Download rdp-cli · Follow the platform-specific quick start

Related guides

Enhanced commands in rdp-cli

Run rdp-cli --version first. The commands below require agent_execution.contract_version to be 1; older builds can continue using the single-step workflow above.

Replace uppercase placeholders with actual returned values. In Windows PowerShell, use .\rdp-cli.exe from its folder; on macOS or Linux, use the installed command or the executable’s actual path.

See the part that matters

Request a screen region or a smaller image. Compare it with an earlier observation and skip the image when nothing changed; coordinates remain tied to the remote screen.

Wait for a change or a completed transfer

Wait for a region to change or settle, or for a connection, operation or file transfer to reach a specified state. Verify the actual outcome before continuing.

Send a short sequence, with a place to stop

Combine actions your agent has chosen, include a new observation and stop at a checkpoint. Errors or a control change stop later steps; already sent actions are not rolled back.

Keep a record and continue deliberately

Save task progress, checkpoints and selected images. Export a local report, inspect what completed after an interruption, and start new steps without replaying uncertain actions.

Send the JSON below to batch through standard input. This example only observes and stops at a checkpoint; inspect the current image before adding input actions. Use the task ID returned by tasks create as TASK_ID to keep this batch in the same task record.

rdp-cli screenshot --session SESSION --max-width 640
rdp-cli screenshot --session SESSION --region X,Y,WIDTH,HEIGHT --max-width 640
rdp-cli wait --session SESSION --condition changed --region X,Y,WIDTH,HEIGHT --timeout-ms 10000
rdp-cli tasks create --session SESSION --title "Review result"
rdp-cli batch --session SESSION --task TASK_ID --steps-stdin --request-id UNIQUE_REQUEST_ID
{"version":1,"steps":[{"id":"look","command":"observe","arguments":{"max-width":640}},{"id":"review","command":"checkpoint"}]}