sandbox-agent/.context/plans/desktop-computer-use-api-enhancements.md

# Desktop Computer Use API Enhancements

## Context

Competitive analysis of Daytona, Cloudflare Sandbox SDK, and CUA revealed significant gaps in our desktop computer use API. Both Daytona and Cloudflare have or are building screenshot compression, hotkey combos, mouseDown/mouseUp, keyDown/keyUp, per-component process health, and live desktop streaming. CUA additionally has window management and accessibility trees. We have none of these. This plan closes the most impactful gaps across 7 tasks.

## Execution Order

```
Sprint 1 (parallel, no dependencies):  Tasks 1, 2, 3, 4
Sprint 2 (foundational refactor):      Task 5
Sprint 3 (parallel, depend on #5):     Tasks 6, 7
```

---

## Task 1: Unify keyboard press with object modifiers

**What**: Change `DesktopKeyboardPressRequest` to accept a `modifiers` object instead of requiring DSL strings like `"ctrl+c"`.

**Files**:
- `server/packages/sandbox-agent/src/desktop_types.rs` — Add `DesktopKeyModifiers { ctrl, shift, alt, cmd }` struct (all `Option<bool>`). Add `modifiers: Option<DesktopKeyModifiers>` to `DesktopKeyboardPressRequest`.
- `server/packages/sandbox-agent/src/desktop_runtime.rs` — Modify `press_key_args()` (~line 1349) to build xdotool key string from modifiers object. If modifiers present, construct `"ctrl+shift+a"` style string. `cmd` maps to `super`.
- `server/packages/sandbox-agent/src/router.rs` — Add `DesktopKeyModifiers` to OpenAPI schemas list.
- `docs/openapi.json` — Regenerate.

**Backward compatible**: Old `{"key": "ctrl+a"}` still works. New form: `{"key": "a", "modifiers": {"ctrl": true}}`.

**Test**: Unit test that `press_key_args("a", Some({ctrl: true, shift: true}))` produces `["key", "--", "ctrl+shift+a"]`. Integration test with both old and new request shapes.

---

## Task 2: Add mouseDown/mouseUp and keyDown/keyUp endpoints

**What**: 4 new endpoints for low-level press/release control.

**Endpoints**:
- `POST /v1/desktop/mouse/down` — `xdotool mousedown BUTTON` (optional x,y moves first)
- `POST /v1/desktop/mouse/up` — `xdotool mouseup BUTTON`
- `POST /v1/desktop/keyboard/down` — `xdotool keydown KEY`
- `POST /v1/desktop/keyboard/up` — `xdotool keyup KEY`

**Files**:
- `server/packages/sandbox-agent/src/desktop_types.rs` — Add `DesktopMouseDownRequest`, `DesktopMouseUpRequest` (x/y optional, button optional), `DesktopKeyboardDownRequest`, `DesktopKeyboardUpRequest` (key: String).
- `server/packages/sandbox-agent/src/desktop_runtime.rs` — Add 4 public methods following existing `click_mouse()` / `press_key()` patterns.
- `server/packages/sandbox-agent/src/router.rs` — Add 4 routes, 4 handlers with utoipa annotations.
- `sdks/typescript/src/client.ts` — Add `mouseDownDesktop()`, `mouseUpDesktop()`, `keyDownDesktop()`, `keyUpDesktop()`.
- `docs/openapi.json` — Regenerate.

**Test**: Integration test: mouseDown → mousemove → mouseUp sequence. keyDown → keyUp sequence.

---

## Task 3: Screenshot compression

**What**: Add format, quality, and scale query params to screenshot endpoints.

**Params**: `format` (png|jpeg|webp, default png), `quality` (1-100, default 85), `scale` (0.1-1.0, default 1.0).

**Files**:
- `server/packages/sandbox-agent/src/desktop_types.rs` — Add `DesktopScreenshotFormat` enum. Add `format`, `quality`, `scale` fields to `DesktopScreenshotQuery` and `DesktopRegionScreenshotQuery`.
- `server/packages/sandbox-agent/src/desktop_runtime.rs` — After capturing PNG via `import`, pipe through ImageMagick `convert` if format != png or scale != 1.0: `convert png:- -resize {scale*100}% -quality {quality} {format}:-`. Add a `run_command_with_stdin()` helper (or modify existing `run_command_output`) to pipe bytes into a command's stdin.
- `server/packages/sandbox-agent/src/router.rs` — Modify screenshot handlers to pass format/quality/scale, return dynamic `Content-Type` header.
- `sdks/typescript/src/client.ts` — Update `takeDesktopScreenshot()` to accept format/quality/scale.
- `docs/openapi.json` — Regenerate.

**Dependencies**: ImageMagick `convert` already installed in Docker. Verify WebP delegate availability.

**Test**: Integration tests: request `?format=jpeg&quality=50`, verify `Content-Type: image/jpeg` and JPEG magic bytes. Verify default still returns PNG. Verify `?scale=0.5` returns a smaller image.

---

## Task 4: Window listing API

**What**: New endpoint to list open windows.

**Endpoint**: `GET /v1/desktop/windows`

**Files**:
- `server/packages/sandbox-agent/src/desktop_types.rs` — Add `DesktopWindowInfo { id, title, x, y, width, height, is_active }` and `DesktopWindowListResponse`.
- `server/packages/sandbox-agent/src/desktop_runtime.rs` — Add `list_windows()` method using xdotool (already installed):
  1. `xdotool search --onlyvisible --name ""` → window IDs
  2. `xdotool getwindowname {id}` + `xdotool getwindowgeometry {id}` per window
  3. `xdotool getactivewindow` → is_active flag
  4. Add `parse_window_geometry()` helper.
- `server/packages/sandbox-agent/src/router.rs` — Add route, handler, OpenAPI annotations.
- `sdks/typescript/src/client.ts` — Add `listDesktopWindows()`.
- `docs/openapi.json` — Regenerate.

**No new Docker dependencies** — xdotool already installed.

**Test**: Integration test: start desktop, verify `GET /v1/desktop/windows` returns 200 with a list (may be empty if no GUI apps open, which is fine).

---

## Task 5: Unify desktop processes into process runtime with owner flag

**What**: Desktop processes (Xvfb, openbox, dbus) get registered in the general process runtime with an `owner` field, gaining log streaming, SSE, and unified lifecycle for free.

**Files**:

- `server/packages/sandbox-agent/src/process_runtime.rs`:
  - Add `ProcessOwner` enum: `User`, `Desktop`, `System`.
  - Add `RestartPolicy` enum: `Never`, `Always`, `OnFailure`.
  - Add `owner: ProcessOwner` and `restart_policy: Option<RestartPolicy>` to `ProcessStartSpec`, `ManagedProcess`, and `ProcessSnapshot`.
  - Modify `list_processes()` to accept optional owner filter.
  - Add auto-restart logic in `watch_exit()`: if restart_policy is Always (or OnFailure and exit code != 0), re-spawn the process using stored spec. Need to store the original `ProcessStartSpec` on `ManagedProcess`.

- `server/packages/sandbox-agent/src/router/types.rs`:
  - Add `owner` to `ProcessInfo` response.
  - Add `ProcessListQuery { owner: Option<ProcessOwner> }`.

- `server/packages/sandbox-agent/src/router.rs`:
  - Modify `get_v1_processes` to accept `Query<ProcessListQuery>` and filter.
  - Pass `ProcessRuntime` into `DesktopRuntime::new()`.
  - Add `ProcessOwner`, `RestartPolicy` to OpenAPI schemas.

- `server/packages/sandbox-agent/src/desktop_runtime.rs` — **Major refactor**:
  - Remove `ManagedDesktopChild` struct.
  - `DesktopRuntime` takes `ProcessRuntime` as constructor param.
  - `start_xvfb_locked()` and `start_openbox_locked()` call `process_runtime.start_process(ProcessStartSpec { owner: Desktop, restart_policy: Some(Always), ... })` instead of spawning directly.
  - Store returned process IDs in state instead of `Child` handles.
  - `stop` calls `process_runtime.stop_process()` / `kill_process()`.
  - `processes_locked()` queries process runtime for desktop-owned processes.
  - dbus-launch remains a direct one-shot spawn (it's not a long-running process, just produces env vars).

- `sdks/typescript/src/client.ts` — Add `owner` filter option to `listProcesses()`.
- `docs/openapi.json` — Regenerate.

**Risks**:
- Lock ordering: desktop runtime holds Mutex, process runtime uses RwLock. Release desktop Mutex before calling process runtime, or restructure.
- `log_path` field in `DesktopProcessInfo` no longer applies (logs are in-memory now). Remove or deprecate.

**Test**: Integration: start desktop, `GET /v1/processes?owner=desktop` returns Xvfb+openbox. `GET /v1/processes?owner=user` excludes them. Desktop process logs are streamable via `GET /v1/processes/{id}/logs?follow=true`. Existing desktop lifecycle tests still pass.

---

## Task 6: Screen recording API (ffmpeg x11grab)

**What**: 6 endpoints for recording the desktop to MP4.

**Endpoints**:
- `POST /v1/desktop/recording/start` — Start ffmpeg recording
- `POST /v1/desktop/recording/stop` — Stop recording (SIGTERM → wait → SIGKILL)
- `GET /v1/desktop/recordings` — List recordings
- `GET /v1/desktop/recordings/{id}` — Get recording metadata
- `GET /v1/desktop/recordings/{id}/download` — Serve MP4 file
- `DELETE /v1/desktop/recordings/{id}` — Delete recording

**Files**:
- **New**: `server/packages/sandbox-agent/src/desktop_recording.rs` — Recording state, ffmpeg process management. `start_recording()` spawns ffmpeg via process runtime (owner=Desktop): `ffmpeg -f x11grab -video_size WxH -i :99 -c:v libx264 -preset ultrafast -r 30 {path}`. Recordings stored in `{state_dir}/recordings/`.
- `server/packages/sandbox-agent/src/desktop_types.rs` — Add recording request/response types.
- `server/packages/sandbox-agent/src/desktop_runtime.rs` — Wire recording manager, expose through desktop runtime.
- `server/packages/sandbox-agent/src/router.rs` — Add 6 routes + handlers.
- `server/packages/sandbox-agent/src/desktop_install.rs` — Add `ffmpeg` to dependency detection (soft: only error when recording is requested).
- `docker/runtime/Dockerfile` and `docker/test-agent/Dockerfile` — Add `ffmpeg` to apt-get.
- `sdks/typescript/src/client.ts` — Add 6 recording methods.
- `docs/openapi.json` — Regenerate.

**Depends on**: Task 5 (ffmpeg runs as desktop-owned process).

**Test**: Integration: start desktop → start recording → wait 2s → stop → list → download (verify MP4 magic bytes) → delete.

---

## Task 7: Neko WebRTC desktop streaming + React component

**What**: Integrate neko for WebRTC desktop streaming, mirroring the ProcessTerminal + Ghostty pattern.

### Server side

- **New**: `server/packages/sandbox-agent/src/desktop_streaming.rs` — Manages neko process via process runtime (owner=Desktop). Neko connects to existing Xvfb display, runs GStreamer pipeline for H.264 encoding.
- `server/packages/sandbox-agent/src/router.rs`:
  - `GET /v1/desktop/stream/ws` — WebSocket proxy to neko's internal WebSocket. Upgrade request, bridge bidirectionally.
  - `POST /v1/desktop/stream/start` / `POST /v1/desktop/stream/stop` — Lifecycle control.
- `docker/runtime/Dockerfile` and `docker/test-agent/Dockerfile` — Add neko binary + GStreamer packages (`gstreamer1.0-plugins-base`, `gstreamer1.0-plugins-good`, `gstreamer1.0-x`, `libgstreamer1.0-0`). Consider making this an optional Docker stage to avoid bloating the base image.

### TypeScript SDK

- **New**: `sdks/typescript/src/desktop-stream.ts` — `DesktopStreamSession` class ported from neko's `base.ts` (~500 lines):
  - WebSocket for signaling (SDP offer/answer, ICE candidates)
  - `RTCPeerConnection` for video stream
  - `RTCDataChannel` for binary input (mouse: 7 bytes, keyboard: 11 bytes)
  - Events: `onTrack(stream)`, `onConnect()`, `onDisconnect()`, `onError()`
- `sdks/typescript/src/client.ts` — Add `connectDesktopStream()` returning `DesktopStreamSession`, `buildDesktopStreamWebSocketUrl()`, `startDesktopStream()`, `stopDesktopStream()`.
- `sdks/typescript/src/index.ts` — Export `DesktopStreamSession`.

### React SDK

- **New**: `sdks/react/src/DesktopViewer.tsx` — Following `ProcessTerminal.tsx` pattern:
  ```
  Props: client (Pick<SandboxAgent, 'connectDesktopStream'>), height, className, style, onConnect, onDisconnect, onError
  ```
  - `useEffect` → `client.connectDesktopStream()` → wire `onTrack` to `<video>.srcObject`
  - Capture mouse events on video element → scale coordinates to desktop resolution → send via DataChannel
  - Capture keyboard events → send via DataChannel
  - Connection state indicator
  - Cleanup: close RTCPeerConnection, close WebSocket
- `sdks/react/src/index.ts` — Export `DesktopViewer`.

**Depends on**: Task 5 (neko runs as desktop-owned process).

**Test**: Server integration: start stream, connect WebSocket, verify signaling messages flow. React: component mounts/unmounts without errors. Full E2E requires browser (manual initially).

---

## Verification

After all tasks:
1. `cargo test` — All Rust unit tests pass
2. `cargo test --test v1_api` — All integration tests pass (requires Docker)
3. Regenerate `docs/openapi.json` and verify it reflects all new endpoints
4. Build TypeScript SDK: `cd sdks/typescript && pnpm build`
5. Build React SDK: `cd sdks/react && pnpm build`
6. Manual: start desktop, take JPEG screenshot, list windows, record 5s video, stream desktop via DesktopViewer component