mirror of
https://github.com/harivansh-afk/sandbox-agent.git
synced 2026-04-16 02:01:35 +00:00
Add desktop runtime API and SDK support
This commit is contained in:
parent
3d9476ed0b
commit
641597afe6
27 changed files with 5881 additions and 21 deletions
30
docs/cli.mdx
30
docs/cli.mdx
|
|
@ -37,6 +37,36 @@ Notes:
|
|||
- Set `SANDBOX_AGENT_LOG_STDOUT=1` to force stdout/stderr logging.
|
||||
- Use `SANDBOX_AGENT_LOG_DIR` to override log directory.
|
||||
|
||||
## install
|
||||
|
||||
Install first-party runtime dependencies.
|
||||
|
||||
### install desktop
|
||||
|
||||
Install the Linux desktop runtime packages required by `/v1/desktop/*`.
|
||||
|
||||
```bash
|
||||
sandbox-agent install desktop [OPTIONS]
|
||||
```
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--yes` | Skip the confirmation prompt |
|
||||
| `--print-only` | Print the package-manager command without executing it |
|
||||
| `--package-manager <apt\|dnf\|apk>` | Override package-manager detection |
|
||||
| `--no-fonts` | Skip the default DejaVu font package |
|
||||
|
||||
```bash
|
||||
sandbox-agent install desktop --yes
|
||||
sandbox-agent install desktop --print-only
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Supported on Linux only.
|
||||
- The command detects `apt`, `dnf`, or `apk`.
|
||||
- If the host is not already running as root, the command requires `sudo`.
|
||||
|
||||
## install-agent
|
||||
|
||||
Install or reinstall a single agent.
|
||||
|
|
|
|||
|
|
@ -21,6 +21,27 @@ docker run --rm -p 3000:3000 \
|
|||
sandbox-agent server --no-token --host 0.0.0.0 --port 3000"
|
||||
```
|
||||
|
||||
If you also want the desktop API inside the container, install desktop dependencies before starting the server:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 3000:3000 \
|
||||
-e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
|
||||
-e OPENAI_API_KEY="$OPENAI_API_KEY" \
|
||||
node:22-bookworm-slim sh -c "\
|
||||
apt-get update && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y curl ca-certificates bash libstdc++6 && \
|
||||
rm -rf /var/lib/apt/lists/* && \
|
||||
curl -fsSL https://releases.rivet.dev/sandbox-agent/0.3.x/install.sh | sh && \
|
||||
sandbox-agent install desktop --yes && \
|
||||
sandbox-agent server --no-token --host 0.0.0.0 --port 3000"
|
||||
```
|
||||
|
||||
In a Dockerfile:
|
||||
|
||||
```dockerfile
|
||||
RUN sandbox-agent install desktop --yes
|
||||
```
|
||||
|
||||
## TypeScript with dockerode
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ console.log(url);
|
|||
- Event JSON inspector
|
||||
- Prompt testing
|
||||
- Request/response debugging
|
||||
- Desktop panel for status, remediation, start/stop, and screenshot refresh
|
||||
- Process management (create, stop, kill, delete, view logs)
|
||||
- Interactive PTY terminal for tty processes
|
||||
- One-shot command execution
|
||||
|
|
@ -49,3 +50,16 @@ console.log(url);
|
|||
The Inspector includes an embedded Ghostty-based terminal for interactive tty
|
||||
processes. The UI uses the SDK's high-level `connectProcessTerminal(...)`
|
||||
wrapper via the shared `@sandbox-agent/react` `ProcessTerminal` component.
|
||||
|
||||
## Desktop panel
|
||||
|
||||
The `Desktop` panel shows the current desktop runtime state, missing dependencies,
|
||||
the suggested install command, last error details, process/log paths, and the
|
||||
latest captured screenshot.
|
||||
|
||||
Use it to:
|
||||
|
||||
- Check whether desktop dependencies are installed
|
||||
- Start or stop the managed desktop runtime
|
||||
- Refresh desktop status
|
||||
- Capture a fresh screenshot on demand
|
||||
|
|
|
|||
1100
docs/openapi.json
1100
docs/openapi.json
File diff suppressed because it is too large
Load diff
|
|
@ -224,6 +224,16 @@ icon: "rocket"
|
|||
If agents are not installed up front, they are lazily installed when creating a session.
|
||||
</Step>
|
||||
|
||||
<Step title="Install desktop dependencies (optional, Linux only)">
|
||||
If you want to use `/v1/desktop/*`, install the desktop runtime packages first:
|
||||
|
||||
```bash
|
||||
sandbox-agent install desktop --yes
|
||||
```
|
||||
|
||||
Then use `GET /v1/desktop/status` or `sdk.getDesktopStatus()` to verify the runtime is ready before calling desktop screenshot or input APIs.
|
||||
</Step>
|
||||
|
||||
<Step title="Create a session">
|
||||
```typescript
|
||||
import { SandboxAgent } from "sandbox-agent";
|
||||
|
|
|
|||
|
|
@ -177,6 +177,44 @@ const writeResult = await sdk.writeFsFile({ path: "./hello.txt" }, "hello");
|
|||
console.log(health.status, agents.agents.length, entries.length, writeResult.path);
|
||||
```
|
||||
|
||||
## Desktop API
|
||||
|
||||
The SDK also wraps the desktop host/runtime HTTP API.
|
||||
|
||||
Install desktop dependencies first on Linux hosts:
|
||||
|
||||
```bash
|
||||
sandbox-agent install desktop --yes
|
||||
```
|
||||
|
||||
Then query status, surface remediation if needed, and start the runtime:
|
||||
|
||||
```ts
|
||||
const status = await sdk.getDesktopStatus();
|
||||
|
||||
if (status.state === "install_required") {
|
||||
console.log(status.installCommand);
|
||||
}
|
||||
|
||||
const started = await sdk.startDesktop({
|
||||
width: 1440,
|
||||
height: 900,
|
||||
dpi: 96,
|
||||
});
|
||||
|
||||
const screenshot = await sdk.takeDesktopScreenshot();
|
||||
const displayInfo = await sdk.getDesktopDisplayInfo();
|
||||
|
||||
await sdk.moveDesktopMouse({ x: 400, y: 300 });
|
||||
await sdk.clickDesktop({ x: 400, y: 300, button: "left", clickCount: 1 });
|
||||
await sdk.typeDesktopText({ text: "hello world", delayMs: 10 });
|
||||
await sdk.pressDesktopKey({ key: "ctrl+l" });
|
||||
|
||||
await sdk.stopDesktop();
|
||||
```
|
||||
|
||||
Screenshot helpers return `Uint8Array` PNG bytes. The SDK does not attempt to install OS packages remotely; callers should surface `missingDependencies` and `installCommand` from `getDesktopStatus()`.
|
||||
|
||||
## Error handling
|
||||
|
||||
```ts
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue