One session for your agent and you
rdp-cli lets an AI agent see and operate a Linux application through commands. The agent captures a screen, acts on the observed target and checks the next screen. You can open a native window or an authorized web viewer to watch, pause or take control of that same session.
Prepare an authorized target
The Linux computer must already provide SSH access and an installed X11-compatible application. Replace the example address, account, private-key path and independently verified known-hosts file. In Windows PowerShell, use .\rdp-cli.exe instead of rdp-cli. Check that --version reports embedded_ssh, x11_desktop and x11_application as true.
Open one application
This example opens an already installed Gedit in an isolated display managed inside rdp-cli. It opens no local viewer by default. Save data.session from the response as SESSION. A host whose normal desktop uses Wayland can still forward an X11-compatible application this way.
rdp-cli connect --protocol x11 --mode application --host 192.0.2.20 --user USER --ssh-key PRIVATE_KEY --ssh-known-hosts KNOWN_HOSTS --linux-file-root FILE_ROOT --application /usr/bin/gedit --app-args '["--standalone","--new-window"]'
Or join an existing X11 desktop
Use the actual display and its existing authorization-file path. The SSH account needs permission to access that display; SSH authentication alone does not grant desktop access. This mode joins the existing desktop rather than creating another login session.
rdp-cli connect --protocol x11 --mode desktop --host 192.0.2.20 --user USER --ssh-key PRIVATE_KEY --ssh-known-hosts KNOWN_HOSTS --linux-file-root FILE_ROOT --display DISPLAY --xauthority XAUTHORITY_PATH
Observe, act, verify
Open the returned PNG and save its data.observation. Replace X and Y with coordinates from that image. After focusing the field, capture again and use the new token. The explicit clipboard method replaces the remote clipboard; it does not change the computer clipboard. An input response confirms dispatch, so verify the visible result afterward.
rdp-cli screenshot --session SESSION --out before.png
rdp-cli click --session SESSION --x X --y Y --observation OBSERVATION
rdp-cli screenshot --session SESSION --out focused.png
rdp-cli linux text --session SESSION --method clipboard --text 'Review complete.' --observation OBSERVATION
rdp-cli screenshot --session SESSION --out after.png
Bring files back
Open the upload result’s remote_path in the Linux application, edit and save it, then download the result. Keep each transfer_id and wait for completed before using the file. Save files before transferring; existing names are not overwritten by default. Check the returned task state after an error instead of blindly repeating a transfer.
Set FILE_ROOT to an existing remote directory that this SSH account can read and write. The file commands stay inside that directory; setting a root does not create the directory or grant new permissions.
rdp-cli files list --session SESSION
rdp-cli files upload --session SESSION --local ./report.txt --path report.txt
rdp-cli files status --session SESSION --transfer-id UPLOAD_TRANSFER
rdp-cli files download --session SESSION --path report.txt --local ./result.txt
rdp-cli files status --session SESSION --transfer-id DOWNLOAD_TRANSFER
Let a person join
Open this viewer only when you want to watch. It starts read-only. Take control to stop agent input, make a correction, and release control to hand the session back. An authorized web viewer joins the same session; only one viewer can hold human control. Closing a read-only viewer keeps the backend connection.
rdp-cli watch --session SESSION --viewer window
Know the protocol boundary
Native X11 has no playback or microphone redirection. A whole Wayland desktop requires an already configured compatible RDP or VNC endpoint, using its actual negotiated capabilities. rdp-cli does not install or enable a remote service. An application can exit when its display connection is disconnected or lost; save first, and do not assume automatic relaunch or a preserved workspace.
Wayland text input depends on the existing server. GNOME 46 and Krfb 23.08 may not map every Unicode key; verify the resulting text. Use explicit clipboard paste only when it works on that endpoint: Krfb on Plasma Wayland can advertise clipboard support without delivering it to applications. For sensitive input or rapid sequences, send smaller steps and check the next screenshot.
Tested desktop endpoints: Ubuntu 24.04.5 · GNOME 46 / GNOME Remote Desktop 46.3; KDE Plasma 5.27.12 / Krfb 23.08.5.