harivansh-afk/sandbox-agent
1
0
Fork 0
mirror of https://github.com/harivansh-afk/sandbox-agent.git synced 2026-04-15 06:04:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add desktop runtime API and SDK support

Browse source
This commit is contained in:
Nathan Flurry 2026-03-07 23:32:49 -08:00
parent 3d9476ed0b
commit 641597afe6
27 changed files with 5881 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

30
docs/cli.mdx
View file

@ -37,6 +37,36 @@ Notes:
- Set `SANDBOX_AGENT_LOG_STDOUT=1` to force stdout/stderr logging.
- Use `SANDBOX_AGENT_LOG_DIR` to override log directory.
## install
Install first-party runtime dependencies.
### install desktop
Install the Linux desktop runtime packages required by `/v1/desktop/*`.
```bash
sandbox-agent install desktop [OPTIONS]
```
| Option | Description |
|--------|-------------|
| `--yes` | Skip the confirmation prompt |
| `--print-only` | Print the package-manager command without executing it |
| `--package-manager <apt\|dnf\|apk>` | Override package-manager detection |
| `--no-fonts` | Skip the default DejaVu font package |
```bash
sandbox-agent install desktop --yes
sandbox-agent install desktop --print-only
```
Notes:
- Supported on Linux only.
- The command detects `apt`, `dnf`, or `apk`.
- If the host is not already running as root, the command requires `sudo`.
## install-agent
Install or reinstall a single agent.

21
docs/deploy/docker.mdx
View file

@ -21,6 +21,27 @@ docker run --rm -p 3000:3000 \
sandbox-agent server --no-token --host 0.0.0.0 --port 3000"
```
If you also want the desktop API inside the container, install desktop dependencies before starting the server:
```bash
docker run --rm -p 3000:3000 \
-e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
-e OPENAI_API_KEY="$OPENAI_API_KEY" \
node:22-bookworm-slim sh -c "\
apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y curl ca-certificates bash libstdc++6 && \
rm -rf /var/lib/apt/lists/* && \
curl -fsSL https://releases.rivet.dev/sandbox-agent/0.3.x/install.sh | sh && \
sandbox-agent install desktop --yes && \
sandbox-agent server --no-token --host 0.0.0.0 --port 3000"
```
In a Dockerfile:
```dockerfile
RUN sandbox-agent install desktop --yes
```
## TypeScript with dockerode
```typescript

14
docs/inspector.mdx
View file

@ -34,6 +34,7 @@ console.log(url);
- Event JSON inspector
- Prompt testing
- Request/response debugging
- Desktop panel for status, remediation, start/stop, and screenshot refresh
- Process management (create, stop, kill, delete, view logs)
- Interactive PTY terminal for tty processes
- One-shot command execution
@ -49,3 +50,16 @@ console.log(url);
The Inspector includes an embedded Ghostty-based terminal for interactive tty
processes. The UI uses the SDK's high-level `connectProcessTerminal(...)`
wrapper via the shared `@sandbox-agent/react` `ProcessTerminal` component.
## Desktop panel
The `Desktop` panel shows the current desktop runtime state, missing dependencies,
the suggested install command, last error details, process/log paths, and the
latest captured screenshot.
Use it to:
- Check whether desktop dependencies are installed
- Start or stop the managed desktop runtime
- Refresh desktop status
- Capture a fresh screenshot on demand

1100
docs/openapi.json
View file

File diff suppressed because it is too large Load diff

10
docs/quickstart.mdx
View file

@ -224,6 +224,16 @@ icon: "rocket"
If agents are not installed up front, they are lazily installed when creating a session.
</Step>
<Step title="Install desktop dependencies (optional, Linux only)">
If you want to use `/v1/desktop/*`, install the desktop runtime packages first:
```bash
sandbox-agent install desktop --yes
```
Then use `GET /v1/desktop/status` or `sdk.getDesktopStatus()` to verify the runtime is ready before calling desktop screenshot or input APIs.
</Step>
<Step title="Create a session">
```typescript
import { SandboxAgent } from "sandbox-agent";

38
docs/sdk-overview.mdx
View file

@ -177,6 +177,44 @@ const writeResult = await sdk.writeFsFile({ path: "./hello.txt" }, "hello");
console.log(health.status, agents.agents.length, entries.length, writeResult.path);
```
## Desktop API
The SDK also wraps the desktop host/runtime HTTP API.
Install desktop dependencies first on Linux hosts:
```bash
sandbox-agent install desktop --yes
```
Then query status, surface remediation if needed, and start the runtime:
```ts
const status = await sdk.getDesktopStatus();
if (status.state === "install_required") {
console.log(status.installCommand);
}
const started = await sdk.startDesktop({
width: 1440,
height: 900,
dpi: 96,
});
const screenshot = await sdk.takeDesktopScreenshot();
const displayInfo = await sdk.getDesktopDisplayInfo();
await sdk.moveDesktopMouse({ x: 400, y: 300 });
await sdk.clickDesktop({ x: 400, y: 300, button: "left", clickCount: 1 });
await sdk.typeDesktopText({ text: "hello world", delayMs: 10 });
await sdk.pressDesktopKey({ key: "ctrl+l" });
await sdk.stopDesktop();
```
Screenshot helpers return `Uint8Array` PNG bytes. The SDK does not attempt to install OS packages remotely; callers should surface `missingDependencies` and `installCommand` from `getDesktopStatus()`.
## Error handling
```ts

93
frontend/packages/inspector/index.html
View file

@ -2768,6 +2768,94 @@
gap: 20px;
}
.desktop-panel {
display: flex;
flex-direction: column;
gap: 16px;
}
.desktop-state-grid {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 12px;
margin-bottom: 12px;
}
.desktop-start-controls {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 10px;
}
.desktop-input-group {
display: flex;
flex-direction: column;
gap: 4px;
}
.desktop-chip-list {
display: flex;
flex-wrap: wrap;
gap: 8px;
}
.desktop-command {
margin-top: 6px;
padding: 8px 10px;
border-radius: var(--radius);
border: 1px solid var(--border);
background: var(--surface);
overflow-x: auto;
}
.desktop-diagnostic-block + .desktop-diagnostic-block {
margin-top: 14px;
}
.desktop-process-list {
display: flex;
flex-direction: column;
gap: 10px;
margin-top: 8px;
}
.desktop-process-item {
padding: 10px;
border-radius: var(--radius);
border: 1px solid var(--border);
background: var(--surface);
display: flex;
flex-direction: column;
gap: 4px;
}
.desktop-screenshot-empty {
padding: 18px;
border: 1px dashed var(--border);
border-radius: var(--radius);
color: var(--muted);
background: var(--surface);
text-align: center;
}
.desktop-screenshot-frame {
border-radius: calc(var(--radius) + 2px);
overflow: hidden;
border: 1px solid var(--border);
background:
linear-gradient(135deg, rgba(15, 23, 42, 0.9), rgba(30, 41, 59, 0.92)),
radial-gradient(circle at top right, rgba(56, 189, 248, 0.12), transparent 40%);
padding: 10px;
}
.desktop-screenshot-image {
display: block;
width: 100%;
height: auto;
border-radius: var(--radius);
background: rgba(0, 0, 0, 0.24);
}
.processes-section {
display: flex;
flex-direction: column;
@ -3430,6 +3518,11 @@
grid-template-columns: 1fr;
}
.desktop-state-grid,
.desktop-start-controls {
grid-template-columns: 1fr;
}
.session-sidebar {
display: none;
}

1
frontend/packages/inspector/package.json
View file

@ -18,6 +18,7 @@
"@types/react-dom": "^18.3.0",
"@vitejs/plugin-react": "^4.3.1",
"fake-indexeddb": "^6.2.4",
"jsdom": "^26.1.0",
"typescript": "^5.7.3",
"vite": "^5.4.7",
"vitest": "^3.0.0"

13
frontend/packages/inspector/src/components/debug/DebugPanel.tsx
View file

@ -1,4 +1,4 @@
import { ChevronLeft, ChevronRight, Cloud, Play, PlayCircle, Server, Terminal, Wrench } from "lucide-react";
import { ChevronLeft, ChevronRight, Cloud, Monitor, Play, PlayCircle, Server, Terminal, Wrench } from "lucide-react";
import type { AgentInfo, SandboxAgent, SessionEvent } from "sandbox-agent";
type AgentModeInfo = { id: string; name: string; description: string };
@ -9,9 +9,10 @@ import ProcessesTab from "./ProcessesTab";
import ProcessRunTab from "./ProcessRunTab";
import SkillsTab from "./SkillsTab";
import RequestLogTab from "./RequestLogTab";
import DesktopTab from "./DesktopTab";
import type { RequestLog } from "../../types/requestLog";
export type DebugTab = "log" | "events" | "agents" | "mcp" | "skills" | "processes" | "run-process";
export type DebugTab = "log" | "events" | "agents" | "desktop" | "mcp" | "skills" | "processes" | "run-process";
const DebugPanel = ({
debugTab,
@ -79,6 +80,10 @@ const DebugPanel = ({
<Cloud className="button-icon" style={{ marginRight: 4, width: 12, height: 12 }} />
Agents
</button>
<button className={`debug-tab ${debugTab === "desktop" ? "active" : ""}`} onClick={() => onDebugTabChange("desktop")}>
<Monitor className="button-icon" style={{ marginRight: 4, width: 12, height: 12 }} />
Desktop
</button>
<button className={`debug-tab ${debugTab === "mcp" ? "active" : ""}`} onClick={() => onDebugTabChange("mcp")}>
<Server className="button-icon" style={{ marginRight: 4, width: 12, height: 12 }} />
MCP
@ -128,6 +133,10 @@ const DebugPanel = ({
/>
)}
{debugTab === "desktop" && (
<DesktopTab getClient={getClient} />
)}
{debugTab === "mcp" && (
<McpTab getClient={getClient} />
)}

150
frontend/packages/inspector/src/components/debug/DesktopTab.test.tsx Normal file
View file

@ -0,0 +1,150 @@
// @vitest-environment jsdom
import { act } from "react";
import { createRoot, type Root } from "react-dom/client";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { SandboxAgent } from "sandbox-agent";
import DesktopTab from "./DesktopTab";
type MockDesktopClient = Pick<
SandboxAgent,
"getDesktopStatus" | "startDesktop" | "stopDesktop" | "takeDesktopScreenshot"
>;
describe("DesktopTab", () => {
let container: HTMLDivElement;
let root: Root;
let createObjectUrl: ReturnType<typeof vi.fn>;
let revokeObjectUrl: ReturnType<typeof vi.fn>;
beforeEach(() => {
vi.useFakeTimers();
vi.stubGlobal("IS_REACT_ACT_ENVIRONMENT", true);
container = document.createElement("div");
document.body.appendChild(container);
root = createRoot(container);
createObjectUrl = vi.fn(() => "blob:test-screenshot");
revokeObjectUrl = vi.fn();
vi.stubGlobal(
"URL",
Object.assign(URL, {
createObjectURL: createObjectUrl,
revokeObjectURL: revokeObjectUrl,
}),
);
});
afterEach(async () => {
await act(async () => {
root.unmount();
});
container.remove();
vi.runOnlyPendingTimers();
vi.useRealTimers();
vi.unstubAllGlobals();
});
it("renders install remediation when desktop deps are missing", async () => {
const client = {
getDesktopStatus: vi.fn().mockResolvedValue({
state: "install_required",
display: null,
resolution: null,
startedAt: null,
lastError: {
code: "desktop_dependencies_missing",
message: "Desktop dependencies are not installed",
},
missingDependencies: ["Xvfb", "openbox"],
installCommand: "sandbox-agent install desktop --yes",
processes: [],
runtimeLogPath: "/tmp/runtime.log",
}),
startDesktop: vi.fn(),
stopDesktop: vi.fn(),
takeDesktopScreenshot: vi.fn(),
} as unknown as MockDesktopClient;
await act(async () => {
root.render(<DesktopTab getClient={() => client as unknown as SandboxAgent} />);
});
expect(container.textContent).toContain("install_required");
expect(container.textContent).toContain("sandbox-agent install desktop --yes");
expect(container.textContent).toContain("Xvfb");
expect(client.getDesktopStatus).toHaveBeenCalledTimes(1);
});
it("starts desktop, refreshes screenshot, and stops desktop", async () => {
const client = {
getDesktopStatus: vi.fn().mockResolvedValue({
state: "inactive",
display: null,
resolution: null,
startedAt: null,
lastError: null,
missingDependencies: [],
installCommand: null,
processes: [],
runtimeLogPath: null,
}),
startDesktop: vi.fn().mockResolvedValue({
state: "active",
display: ":99",
resolution: { width: 1440, height: 900, dpi: 96 },
startedAt: "2026-03-07T00:00:00Z",
lastError: null,
missingDependencies: [],
installCommand: null,
processes: [],
runtimeLogPath: "/tmp/runtime.log",
}),
stopDesktop: vi.fn().mockResolvedValue({
state: "inactive",
display: null,
resolution: null,
startedAt: null,
lastError: null,
missingDependencies: [],
installCommand: null,
processes: [],
runtimeLogPath: "/tmp/runtime.log",
}),
takeDesktopScreenshot: vi.fn().mockResolvedValue(
new Uint8Array([137, 80, 78, 71, 13, 10, 26, 10]),
),
} as unknown as MockDesktopClient;
await act(async () => {
root.render(<DesktopTab getClient={() => client as unknown as SandboxAgent} />);
});
const startButton = Array.from(container.querySelectorAll("button")).find((button) =>
button.textContent?.includes("Start Desktop"),
);
expect(startButton).toBeTruthy();
await act(async () => {
startButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
await vi.runAllTimersAsync();
});
expect(client.startDesktop).toHaveBeenCalledTimes(1);
expect(client.takeDesktopScreenshot).toHaveBeenCalled();
const screenshot = container.querySelector("img[alt='Desktop screenshot']") as HTMLImageElement | null;
expect(screenshot?.src).toContain("blob:test-screenshot");
const stopButton = Array.from(container.querySelectorAll("button")).find((button) =>
button.textContent?.includes("Stop Desktop"),
);
expect(stopButton).toBeTruthy();
await act(async () => {
stopButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
await vi.runAllTimersAsync();
});
expect(client.stopDesktop).toHaveBeenCalledTimes(1);
expect(container.textContent).toContain("inactive");
});
});

373
frontend/packages/inspector/src/components/debug/DesktopTab.tsx Normal file
View file

@ -0,0 +1,373 @@
import { Loader2, Monitor, Play, RefreshCw, Square, Camera } from "lucide-react";
import { useCallback, useEffect, useMemo, useState } from "react";
import { SandboxAgentError } from "sandbox-agent";
import type {
DesktopStatusResponse,
SandboxAgent,
} from "sandbox-agent";
const MIN_SPIN_MS = 350;
const extractErrorMessage = (error: unknown, fallback: string): string => {
if (error instanceof SandboxAgentError && error.problem?.detail) return error.problem.detail;
if (error instanceof Error) return error.message;
return fallback;
};
const formatStartedAt = (value: string | null | undefined): string => {
if (!value) {
return "Not started";
}
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? value : parsed.toLocaleString();
};
const DesktopTab = ({
getClient,
}: {
getClient: () => SandboxAgent;
}) => {
const [status, setStatus] = useState<DesktopStatusResponse | null>(null);
const [loading, setLoading] = useState(false);
const [refreshing, setRefreshing] = useState(false);
const [acting, setActing] = useState<"start" | "stop" | null>(null);
const [error, setError] = useState<string | null>(null);
const [width, setWidth] = useState("1440");
const [height, setHeight] = useState("900");
const [dpi, setDpi] = useState("96");
const [screenshotUrl, setScreenshotUrl] = useState<string | null>(null);
const [screenshotLoading, setScreenshotLoading] = useState(false);
const [screenshotError, setScreenshotError] = useState<string | null>(null);
const revokeScreenshotUrl = useCallback(() => {
setScreenshotUrl((current) => {
if (current) {
URL.revokeObjectURL(current);
}
return null;
});
}, []);
const loadStatus = useCallback(async (mode: "initial" | "refresh" = "initial") => {
if (mode === "initial") {
setLoading(true);
} else {
setRefreshing(true);
}
setError(null);
try {
const next = await getClient().getDesktopStatus();
setStatus(next);
return next;
} catch (loadError) {
setError(extractErrorMessage(loadError, "Unable to load desktop status."));
return null;
} finally {
setLoading(false);
setRefreshing(false);
}
}, [getClient]);
const refreshScreenshot = useCallback(async () => {
setScreenshotLoading(true);
setScreenshotError(null);
try {
const bytes = await getClient().takeDesktopScreenshot();
revokeScreenshotUrl();
const payload = new Uint8Array(bytes.byteLength);
payload.set(bytes);
const blob = new Blob([payload.buffer], { type: "image/png" });
setScreenshotUrl(URL.createObjectURL(blob));
} catch (captureError) {
revokeScreenshotUrl();
setScreenshotError(extractErrorMessage(captureError, "Unable to capture desktop screenshot."));
} finally {
setScreenshotLoading(false);
}
}, [getClient, revokeScreenshotUrl]);
useEffect(() => {
void loadStatus();
}, [loadStatus]);
useEffect(() => {
if (status?.state === "active") {
void refreshScreenshot();
} else {
revokeScreenshotUrl();
}
}, [refreshScreenshot, revokeScreenshotUrl, status?.state]);
useEffect(() => {
return () => {
revokeScreenshotUrl();
};
}, [revokeScreenshotUrl]);
const handleStart = async () => {
const parsedWidth = Number.parseInt(width, 10);
const parsedHeight = Number.parseInt(height, 10);
const parsedDpi = Number.parseInt(dpi, 10);
setActing("start");
setError(null);
const startedAt = Date.now();
try {
const next = await getClient().startDesktop({
width: Number.isFinite(parsedWidth) ? parsedWidth : undefined,
height: Number.isFinite(parsedHeight) ? parsedHeight : undefined,
dpi: Number.isFinite(parsedDpi) ? parsedDpi : undefined,
});
setStatus(next);
if (next.state === "active") {
await refreshScreenshot();
}
} catch (startError) {
setError(extractErrorMessage(startError, "Unable to start desktop runtime."));
await loadStatus("refresh");
} finally {
const elapsedMs = Date.now() - startedAt;
if (elapsedMs < MIN_SPIN_MS) {
await new Promise((resolve) => window.setTimeout(resolve, MIN_SPIN_MS - elapsedMs));
}
setActing(null);
}
};
const handleStop = async () => {
setActing("stop");
setError(null);
const startedAt = Date.now();
try {
const next = await getClient().stopDesktop();
setStatus(next);
revokeScreenshotUrl();
} catch (stopError) {
setError(extractErrorMessage(stopError, "Unable to stop desktop runtime."));
await loadStatus("refresh");
} finally {
const elapsedMs = Date.now() - startedAt;
if (elapsedMs < MIN_SPIN_MS) {
await new Promise((resolve) => window.setTimeout(resolve, MIN_SPIN_MS - elapsedMs));
}
setActing(null);
}
};
const canRefreshScreenshot = status?.state === "active";
const resolutionLabel = useMemo(() => {
const resolution = status?.resolution;
if (!resolution) return "Unknown";
const dpiLabel = resolution.dpi ? ` @ ${resolution.dpi} DPI` : "";
return `${resolution.width} x ${resolution.height}${dpiLabel}`;
}, [status?.resolution]);
return (
<div className="desktop-panel">
<div className="inline-row" style={{ marginBottom: 16 }}>
<button
className="button secondary small"
onClick={() => void loadStatus("refresh")}
disabled={loading || refreshing}
>
<RefreshCw className={`button-icon ${loading || refreshing ? "spinner-icon" : ""}`} />
Refresh Status
</button>
<button
className="button secondary small"
onClick={() => void refreshScreenshot()}
disabled={!canRefreshScreenshot || screenshotLoading}
>
{screenshotLoading ? (
<Loader2 className="button-icon spinner-icon" />
) : (
<Camera className="button-icon" />
)}
Refresh Screenshot
</button>
</div>
{error && <div className="banner error">{error}</div>}
{screenshotError && <div className="banner error">{screenshotError}</div>}
<div className="card">
<div className="card-header">
<span className="card-title">
<Monitor size={14} style={{ marginRight: 6 }} />
Desktop Runtime
</span>
<span className={`pill ${
status?.state === "active"
? "success"
: status?.state === "install_required"
? "warning"
: status?.state === "failed"
? "danger"
: ""
}`}>
{status?.state ?? "unknown"}
</span>
</div>
<div className="desktop-state-grid">
<div>
<div className="card-meta">Display</div>
<div className="mono">{status?.display ?? "Not assigned"}</div>
</div>
<div>
<div className="card-meta">Resolution</div>
<div className="mono">{resolutionLabel}</div>
</div>
<div>
<div className="card-meta">Started</div>
<div>{formatStartedAt(status?.startedAt)}</div>
</div>
</div>
<div className="desktop-start-controls">
<div className="desktop-input-group">
<label className="label">Width</label>
<input
className="setup-input mono"
value={width}
onChange={(event) => setWidth(event.target.value)}
inputMode="numeric"
/>
</div>
<div className="desktop-input-group">
<label className="label">Height</label>
<input
className="setup-input mono"
value={height}
onChange={(event) => setHeight(event.target.value)}
inputMode="numeric"
/>
</div>
<div className="desktop-input-group">
<label className="label">DPI</label>
<input
className="setup-input mono"
value={dpi}
onChange={(event) => setDpi(event.target.value)}
inputMode="numeric"
/>
</div>
</div>
<div className="card-actions">
<button
className="button success small"
onClick={() => void handleStart()}
disabled={acting === "start"}
>
{acting === "start" ? (
<Loader2 className="button-icon spinner-icon" />
) : (
<Play className="button-icon" />
)}
Start Desktop
</button>
<button
className="button danger small"
onClick={() => void handleStop()}
disabled={acting === "stop"}
>
{acting === "stop" ? (
<Loader2 className="button-icon spinner-icon" />
) : (
<Square className="button-icon" />
)}
Stop Desktop
</button>
</div>
</div>
{status?.missingDependencies && status.missingDependencies.length > 0 && (
<div className="card">
<div className="card-header">
<span className="card-title">Missing Dependencies</span>
</div>
<div className="desktop-chip-list">
{status.missingDependencies.map((dependency) => (
<span key={dependency} className="pill warning">{dependency}</span>
))}
</div>
{status.installCommand && (
<>
<div className="card-meta" style={{ marginTop: 12 }}>Install command</div>
<div className="mono desktop-command">{status.installCommand}</div>
</>
)}
</div>
)}
{(status?.lastError || status?.runtimeLogPath || (status?.processes?.length ?? 0) > 0) && (
<div className="card">
<div className="card-header">
<span className="card-title">Diagnostics</span>
</div>
{status?.lastError && (
<div className="desktop-diagnostic-block">
<div className="card-meta">Last error</div>
<div className="mono">{status.lastError.code}</div>
<div>{status.lastError.message}</div>
</div>
)}
{status?.runtimeLogPath && (
<div className="desktop-diagnostic-block">
<div className="card-meta">Runtime log</div>
<div className="mono">{status.runtimeLogPath}</div>
</div>
)}
{status?.processes && status.processes.length > 0 && (
<div className="desktop-diagnostic-block">
<div className="card-meta">Processes</div>
<div className="desktop-process-list">
{status.processes.map((process) => (
<div key={`${process.name}-${process.pid ?? "none"}`} className="desktop-process-item">
<div>
<strong>{process.name}</strong>
<span className={`pill ${process.running ? "success" : "danger"}`} style={{ marginLeft: 8 }}>
{process.running ? "running" : "stopped"}
</span>
</div>
<div className="mono">
{process.pid ? `pid ${process.pid}` : "no pid"}
</div>
{process.logPath && <div className="mono">{process.logPath}</div>}
</div>
))}
</div>
</div>
)}
</div>
)}
<div className="card">
<div className="card-header">
<span className="card-title">Latest Screenshot</span>
{status?.state === "active" ? (
<span className="card-meta">Manual refresh only</span>
) : null}
</div>
{loading ? <div className="card-meta">Loading...</div> : null}
{!loading && !screenshotUrl && (
<div className="desktop-screenshot-empty">
{status?.state === "active"
? "No screenshot loaded yet."
: "Start the desktop runtime to capture a screenshot."}
</div>
)}
{screenshotUrl && (
<div className="desktop-screenshot-frame">
<img src={screenshotUrl} alt="Desktop screenshot" className="desktop-screenshot-image" />
</div>
)}
</div>
</div>
);
};
export default DesktopTab;

313
pnpm-lock.yaml generated
View file

@ -17,7 +17,7 @@ importers:
version: 2.7.6
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
examples/boxlite:
dependencies:
@ -82,7 +82,7 @@ importers:
version: 6.4.1(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
wrangler:
specifier: latest
version: 4.71.0(@cloudflare/workers-types@4.20260305.1)
@ -110,7 +110,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
examples/daytona:
dependencies:
@ -160,7 +160,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
examples/e2b:
dependencies:
@ -185,7 +185,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
examples/file-system:
dependencies:
@ -417,7 +417,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
frontend/packages/inspector:
dependencies:
@ -449,6 +449,9 @@ importers:
fake-indexeddb:
specifier: ^6.2.4
version: 6.2.5
jsdom:
specifier: ^26.1.0
version: 26.1.0
sandbox-agent:
specifier: workspace:*
version: link:../../../sdks/typescript
@ -460,7 +463,7 @@ importers:
version: 5.4.21(@types/node@25.3.5)
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
frontend/packages/website:
dependencies:
@ -591,7 +594,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/cli:
dependencies:
@ -617,7 +620,7 @@ importers:
devDependencies:
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/cli-shared:
devDependencies:
@ -665,7 +668,7 @@ importers:
devDependencies:
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/gigacode/platforms/darwin-arm64: {}
@ -697,7 +700,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/persist-postgres:
dependencies:
@ -722,7 +725,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/persist-rivet:
dependencies:
@ -744,7 +747,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/persist-sqlite:
dependencies:
@ -769,7 +772,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
sdks/react:
dependencies:
@ -823,7 +826,7 @@ importers:
version: 5.9.3
vitest:
specifier: ^3.0.0
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
version: 3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2)
ws:
specifier: ^8.19.0
version: 8.19.0
@ -839,6 +842,9 @@ packages:
resolution: {integrity: sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw==}
engines: {node: '>=10'}
'@asamuzakjp/css-color@3.2.0':
resolution: {integrity: sha512-K1A6z8tS3XsmCMM86xoWdn7Fkdn9m6RSVtocUrJYIwZnFVkng/PvkEoWtOWmP+Scc6saYWHWZYbndEEXxl24jw==}
'@asteasolutions/zod-to-openapi@8.4.0':
resolution: {integrity: sha512-Ckp971tmTw4pnv+o7iK85ldBHBKk6gxMaoNyLn3c2Th/fKoTG8G3jdYuOanpdGqwlDB0z01FOjry2d32lfTqrA==}
peerDependencies:
@ -1293,6 +1299,34 @@ packages:
resolution: {integrity: sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==}
engines: {node: '>=12'}
'@csstools/color-helpers@5.1.0':
resolution: {integrity: sha512-S11EXWJyy0Mz5SYvRmY8nJYTFFd1LCNV+7cXyAgQtOOuzb4EsgfqDufL+9esx72/eLhsRdGZwaldu/h+E4t4BA==}
engines: {node: '>=18'}
'@csstools/css-calc@2.1.4':
resolution: {integrity: sha512-3N8oaj+0juUw/1H3YwmDDJXCgTB1gKU6Hc/bB502u9zR0q2vd786XJH9QfrKIEgFlZmhZiq6epXl4rHqhzsIgQ==}
engines: {node: '>=18'}
peerDependencies:
'@csstools/css-parser-algorithms': ^3.0.5
'@csstools/css-tokenizer': ^3.0.4
'@csstools/css-color-parser@3.1.0':
resolution: {integrity: sha512-nbtKwh3a6xNVIp/VRuXV64yTKnb1IjTAEEh3irzS+HkKjAOYLTGNb9pmVNntZ8iVBHcWDA2Dof0QtPgFI1BaTA==}
engines: {node: '>=18'}
peerDependencies:
'@csstools/css-parser-algorithms': ^3.0.5
'@csstools/css-tokenizer': ^3.0.4
'@csstools/css-parser-algorithms@3.0.5':
resolution: {integrity: sha512-DaDeUkXZKjdGhgYaHNJTV9pV7Y9B3b644jCLs9Upc3VeNGg6LWARAT6O+Q+/COo+2gg/bM5rhpMAtf70WqfBdQ==}
engines: {node: '>=18'}
peerDependencies:
'@csstools/css-tokenizer': ^3.0.4
'@csstools/css-tokenizer@3.0.4':
resolution: {integrity: sha512-Vd/9EVDiu6PPJt9yAh6roZP6El1xHrdvIVGjyBsHR0RYwNHgL7FJPyIIW4fANJNG6FtyZfvlRPpFI4ZM/lubvw==}
engines: {node: '>=18'}
'@daytonaio/api-client@0.149.0':
resolution: {integrity: sha512-tlqVFnJll4JUAY3Ictwl7kGI3jo6HP+AcHl8FsZg/lSG7t/SdlZVO9iPPt6kjxmY3WN8BYRI1NYtIFFh8SJolw==}
@ -2954,6 +2988,10 @@ packages:
engines: {node: '>=0.4.0'}
hasBin: true
agent-base@7.1.4:
resolution: {integrity: sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==}
engines: {node: '>= 14'}
ajv-formats@3.0.1:
resolution: {integrity: sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==}
peerDependencies:
@ -3351,9 +3389,17 @@ packages:
resolution: {integrity: sha512-0LrrStPOdJj+SPCCrGhzryycLjwcgUSHBtxNA8aIDxf0GLsRh1cKYhB00Gd1lDOS4yGH69+SNn13+TWbVHETFQ==}
engines: {node: ^10 || ^12.20.0 || ^14.13.0 || >=15.0.0, npm: '>=7.0.0'}
cssstyle@4.6.0:
resolution: {integrity: sha512-2z+rWdzbbSZv6/rhtvzvqeZQHrBaqgogqt85sqFNbabZOuFbCVFb8kPeEtZjiKkbrm395irpNKiYeFeLiQnFPg==}
engines: {node: '>=18'}
csstype@3.2.3:
resolution: {integrity: sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==}
data-urls@5.0.0:
resolution: {integrity: sha512-ZYP5VBHshaDAiVZxjbRVcFJpc+4xGgT0bK3vzy1HLN8jTO975HEbuYzZJcHoQEY5K1a0z8YayJkyVETa08eNTg==}
engines: {node: '>=18'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
@ -3363,6 +3409,9 @@ packages:
supports-color:
optional: true
decimal.js@10.6.0:
resolution: {integrity: sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg==}
decode-named-character-reference@1.3.0:
resolution: {integrity: sha512-GtpQYB283KrPp6nRw50q3U9/VfOutZOe103qlN7BPP6Ad27xYnOIWv4lPzo8HCAL+mMZofJ9KEy30fq6MfaK6Q==}
@ -3837,6 +3886,10 @@ packages:
resolution: {integrity: sha512-gJnaDHXKDayjt8ue0n8Gs0A007yKXj4Xzb8+cNjZeYsSzzwKc0Lr+OZgYwVfB0pHfUs17EPoLvrOsEaJ9mj+Tg==}
engines: {node: '>=16.9.0'}
html-encoding-sniffer@4.0.0:
resolution: {integrity: sha512-Y22oTqIU4uuPgEemfz7NDJz6OeKf12Lsu+QC+s3BVpda64lTiMYCyGwg5ki4vFxkMwQdeZDl2adZoqUgdFuTgQ==}
engines: {node: '>=18'}
html-escaper@3.0.3:
resolution: {integrity: sha512-RuMffC89BOWQoY0WKGpIhn5gX3iI54O6nRA0yC124NYVtzjmFWBIiFd8M0x+ZdX0P9R4lADg1mgP8C7PxGOWuQ==}
@ -3850,10 +3903,22 @@ packages:
resolution: {integrity: sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==}
engines: {node: '>= 0.8'}
http-proxy-agent@7.0.2:
resolution: {integrity: sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==}
engines: {node: '>= 14'}
https-proxy-agent@7.0.6:
resolution: {integrity: sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==}
engines: {node: '>= 14'}
human-signals@5.0.0:
resolution: {integrity: sha512-AXcZb6vzzrFAUE61HnN4mpLqd/cSIwNQjtNWR0euPm6y0iqx3G4gOXaIDdtdDwZmhwe82LA6+zinmW4UBWVePQ==}
engines: {node: '>=16.17.0'}
iconv-lite@0.6.3:
resolution: {integrity: sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==}
engines: {node: '>=0.10.0'}
iconv-lite@0.7.2:
resolution: {integrity: sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==}
engines: {node: '>=0.10.0'}
@ -3929,6 +3994,9 @@ packages:
resolution: {integrity: sha512-+Pgi+vMuUNkJyExiMBt5IlFoMyKnr5zhJ4Uspz58WOhBF5QoIZkFyNHIbBAtHwzVAgk5RtndVNsDRN61/mmDqg==}
engines: {node: '>=12'}
is-potential-custom-element-name@1.0.1:
resolution: {integrity: sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==}
is-promise@4.0.0:
resolution: {integrity: sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==}
@ -3976,6 +4044,15 @@ packages:
resolution: {integrity: sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==}
hasBin: true
jsdom@26.1.0:
resolution: {integrity: sha512-Cvc9WUhxSMEo4McES3P7oK3QaXldCfNWp7pl2NNeiIFlCoLr3kfq9kb1fxftiwk1FLV7CvpvDfonxtzUDeSOPg==}
engines: {node: '>=18'}
peerDependencies:
canvas: ^3.0.0
peerDependenciesMeta:
canvas:
optional: true
jsesc@3.1.0:
resolution: {integrity: sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA==}
engines: {node: '>=6'}
@ -4333,6 +4410,9 @@ packages:
nth-check@2.1.1:
resolution: {integrity: sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w==}
nwsapi@2.2.23:
resolution: {integrity: sha512-7wfH4sLbt4M0gCDzGE6vzQBo0bfTKjU7Sfpqy/7gs1qBfYz2vEJH6vXcBKpO3+6Yu1telwd0t9HpyOoLEQQbIQ==}
object-assign@4.1.1:
resolution: {integrity: sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==}
engines: {node: '>=0.10.0'}
@ -4662,6 +4742,10 @@ packages:
pump@3.0.3:
resolution: {integrity: sha512-todwxLMY7/heScKmntwQG8CXVkWUOdYxIvY2s0VWAAMh/nd8SoYiRaKjlr7+iCs984f2P8zvrfWcDDYVb73NfA==}
punycode@2.3.1:
resolution: {integrity: sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==}
engines: {node: '>=6'}
qs@6.14.1:
resolution: {integrity: sha512-4EK3+xJl8Ts67nLYNwqw/dsFVnCf+qR7RgXSK9jEEm9unao3njwMDdmsdvoKBKHzxd7tCYz5e5M+SnMjdtXGQQ==}
engines: {node: '>=0.6'}
@ -4840,6 +4924,9 @@ packages:
resolution: {integrity: sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==}
engines: {node: '>= 18'}
rrweb-cssom@0.8.0:
resolution: {integrity: sha512-guoltQEx+9aMf2gDZ0s62EcV8lsXR+0w8915TC3ITdn2YueuNjdAYh/levpU9nFaoChh9RUS5ZdQMrKfVEN9tw==}
run-parallel@1.2.0:
resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==}
@ -4857,6 +4944,10 @@ packages:
resolution: {integrity: sha512-1n3r/tGXO6b6VXMdFT54SHzT9ytu9yr7TaELowdYpMqY/Ao7EnlQGmAQ1+RatX7Tkkdm6hONI2owqNx2aZj5Sw==}
engines: {node: '>=11.0.0'}
saxes@6.0.0:
resolution: {integrity: sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA==}
engines: {node: '>=v12.22.7'}
scheduler@0.23.2:
resolution: {integrity: sha512-UOShsPwz7NrMUqhR6t0hWjFduvOzbtv7toDH1/hIrfRNIDBnnBWd0CwJTGvTpngVlmwGCdP9/Zl/tVrDqcuYzQ==}
@ -5053,6 +5144,9 @@ packages:
engines: {node: '>=16'}
hasBin: true
symbol-tree@3.2.4:
resolution: {integrity: sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==}
tailwindcss@3.4.19:
resolution: {integrity: sha512-3ofp+LL8E+pK/JuPLPggVAIaEuhvIz4qNcf3nA1Xn2o/7fb7s/TYpHhwGDv1ZU3PkBluUVaF8PyCHcm48cKLWQ==}
engines: {node: '>=14.0.0'}
@ -5120,6 +5214,13 @@ packages:
resolution: {integrity: sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==}
engines: {node: '>=14.0.0'}
tldts-core@6.1.86:
resolution: {integrity: sha512-Je6p7pkk+KMzMv2XXKmAE3McmolOQFdxkKw0R8EYNr7sELW46JqnNeTX8ybPiQgvg1ymCoF8LXs5fzFaZvJPTA==}
tldts@6.1.86:
resolution: {integrity: sha512-WMi/OQ2axVTf/ykqCQgXiIct+mSQDFdH2fkwhPwgEwvJ1kSzZRiinb0zF2Xb8u4+OqPChmyI6MEu4EezNJz+FQ==}
hasBin: true
to-regex-range@5.0.1:
resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==}
engines: {node: '>=8.0'}
@ -5128,6 +5229,14 @@ packages:
resolution: {integrity: sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==}
engines: {node: '>=0.6'}
tough-cookie@5.1.2:
resolution: {integrity: sha512-FVDYdxtnj0G6Qm/DhNPSb8Ju59ULcup3tuJxkFb5K8Bv2pUXILbf0xZWU8PX8Ov19OXljbUyveOFwRMwkXzO+A==}
engines: {node: '>=16'}
tr46@5.1.1:
resolution: {integrity: sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw==}
engines: {node: '>=18'}
tree-kill@1.2.2:
resolution: {integrity: sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==}
hasBin: true
@ -5523,9 +5632,30 @@ packages:
vscode-languageserver-types@3.17.5:
resolution: {integrity: sha512-Ld1VelNuX9pdF39h2Hgaeb5hEZM2Z3jUrrMgWQAu82jMtZp7p3vJT3BzToKtZI7NgQssZje5o0zryOrhQvzQAg==}
w3c-xmlserializer@5.0.0:
resolution: {integrity: sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA==}
engines: {node: '>=18'}
web-namespaces@2.0.1:
resolution: {integrity: sha512-bKr1DkiNa2krS7qxNtdrtHAmzuYGFQLiQ13TsorsdT6ULTkPLKuu5+GsFpDlg6JFjUTwX2DyhMPG2be8uPrqsQ==}
webidl-conversions@7.0.0:
resolution: {integrity: sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g==}
engines: {node: '>=12'}
whatwg-encoding@3.1.1:
resolution: {integrity: sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ==}
engines: {node: '>=18'}
deprecated: Use @exodus/bytes instead for a more spec-conformant and faster implementation
whatwg-mimetype@4.0.0:
resolution: {integrity: sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg==}
engines: {node: '>=18'}
whatwg-url@14.2.0:
resolution: {integrity: sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw==}
engines: {node: '>=18'}
which-pm-runs@1.1.0:
resolution: {integrity: sha512-n1brCuqClxfFfq/Rb0ICg9giSZqCS+pLtccdag6C2HyufBrh3fBOiy9nb6ggRMvWOVH5GrdJskj5iGTZNxd7SA==}
engines: {node: '>=4'}
@ -5606,6 +5736,13 @@ packages:
resolution: {integrity: sha512-sqMMuL1rc0FmMBOzCpd0yuy9trqF2yTTVe+E9ogwCSWQCdDEtQUwrZPT6AxqtsFGRNxycgncbP/xmOOSPw5ZUw==}
engines: {node: '>= 6.0'}
xml-name-validator@5.0.0:
resolution: {integrity: sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==}
engines: {node: '>=18'}
xmlchars@2.2.0:
resolution: {integrity: sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==}
xtend@4.0.2:
resolution: {integrity: sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==}
engines: {node: '>=0.4'}
@ -5686,6 +5823,14 @@ snapshots:
'@alloc/quick-lru@5.2.0': {}
'@asamuzakjp/css-color@3.2.0':
dependencies:
'@csstools/css-calc': 2.1.4(@csstools/css-parser-algorithms@3.0.5(@csstools/css-tokenizer@3.0.4))(@csstools/css-tokenizer@3.0.4)
'@csstools/css-color-parser': 3.1.0(@csstools/css-parser-algorithms@3.0.5(@csstools/css-tokenizer@3.0.4))(@csstools/css-tokenizer@3.0.4)
'@csstools/css-parser-algorithms': 3.0.5(@csstools/css-tokenizer@3.0.4)
'@csstools/css-tokenizer': 3.0.4
lru-cache: 10.4.3
'@asteasolutions/zod-to-openapi@8.4.0(zod@4.3.6)':
dependencies:
openapi3-ts: 4.5.0
@ -6516,6 +6661,26 @@ snapshots:
dependencies:
'@jridgewell/trace-mapping': 0.3.9
'@csstools/color-helpers@5.1.0': {}
'@csstools/css-calc@2.1.4(@csstools/css-parser-algorithms@3.0.5(@csstools/css-tokenizer@3.0.4))(@csstools/css-tokenizer@3.0.4)':
dependencies:
'@csstools/css-parser-algorithms': 3.0.5(@csstools/css-tokenizer@3.0.4)
'@csstools/css-tokenizer': 3.0.4
'@csstools/css-color-parser@3.1.0(@csstools/css-parser-algorithms@3.0.5(@csstools/css-tokenizer@3.0.4))(@csstools/css-tokenizer@3.0.4)':
dependencies:
'@csstools/color-helpers': 5.1.0
'@csstools/css-calc': 2.1.4(@csstools/css-parser-algorithms@3.0.5(@csstools/css-tokenizer@3.0.4))(@csstools/css-tokenizer@3.0.4)
'@csstools/css-parser-algorithms': 3.0.5(@csstools/css-tokenizer@3.0.4)
'@csstools/css-tokenizer': 3.0.4
'@csstools/css-parser-algorithms@3.0.5(@csstools/css-tokenizer@3.0.4)':
dependencies:
'@csstools/css-tokenizer': 3.0.4
'@csstools/css-tokenizer@3.0.4': {}
'@daytonaio/api-client@0.149.0':
dependencies:
axios: 1.13.5
@ -8093,6 +8258,8 @@ snapshots:
acorn@8.15.0: {}
agent-base@7.1.4: {}
ajv-formats@3.0.1(ajv@8.17.1):
optionalDependencies:
ajv: 8.17.1
@ -8563,12 +8730,24 @@ snapshots:
dependencies:
css-tree: 2.2.1
cssstyle@4.6.0:
dependencies:
'@asamuzakjp/css-color': 3.2.0
rrweb-cssom: 0.8.0
csstype@3.2.3: {}
data-urls@5.0.0:
dependencies:
whatwg-mimetype: 4.0.0
whatwg-url: 14.2.0
debug@4.4.3:
dependencies:
ms: 2.1.3
decimal.js@10.6.0: {}
decode-named-character-reference@1.3.0:
dependencies:
character-entities: 2.0.2
@ -9203,6 +9382,10 @@ snapshots:
hono@4.12.2: {}
html-encoding-sniffer@4.0.0:
dependencies:
whatwg-encoding: 3.1.1
html-escaper@3.0.3: {}
html-void-elements@3.0.0: {}
@ -9217,8 +9400,26 @@ snapshots:
statuses: 2.0.2
toidentifier: 1.0.1
http-proxy-agent@7.0.2:
dependencies:
agent-base: 7.1.4
debug: 4.4.3
transitivePeerDependencies:
- supports-color
https-proxy-agent@7.0.6:
dependencies:
agent-base: 7.1.4
debug: 4.4.3
transitivePeerDependencies:
- supports-color
human-signals@5.0.0: {}
iconv-lite@0.6.3:
dependencies:
safer-buffer: 2.1.2
iconv-lite@0.7.2:
dependencies:
safer-buffer: 2.1.2
@ -9276,6 +9477,8 @@ snapshots:
is-plain-obj@4.1.0: {}
is-potential-custom-element-name@1.0.1: {}
is-promise@4.0.0: {}
is-stream@3.0.0: {}
@ -9314,6 +9517,33 @@ snapshots:
dependencies:
argparse: 2.0.1
jsdom@26.1.0:
dependencies:
cssstyle: 4.6.0
data-urls: 5.0.0
decimal.js: 10.6.0
html-encoding-sniffer: 4.0.0
http-proxy-agent: 7.0.2
https-proxy-agent: 7.0.6
is-potential-custom-element-name: 1.0.1
nwsapi: 2.2.23
parse5: 7.3.0
rrweb-cssom: 0.8.0
saxes: 6.0.0
symbol-tree: 3.2.4
tough-cookie: 5.1.2
w3c-xmlserializer: 5.0.0
webidl-conversions: 7.0.0
whatwg-encoding: 3.1.1
whatwg-mimetype: 4.0.0
whatwg-url: 14.2.0
ws: 8.19.0
xml-name-validator: 5.0.0
transitivePeerDependencies:
- bufferutil
- supports-color
- utf-8-validate
jsesc@3.1.0: {}
json-schema-traverse@1.0.0: {}
@ -9817,6 +10047,8 @@ snapshots:
dependencies:
boolbase: 1.0.0
nwsapi@2.2.23: {}
object-assign@4.1.1: {}
object-hash@3.0.0: {}
@ -10140,6 +10372,8 @@ snapshots:
end-of-stream: 1.4.5
once: 1.4.0
punycode@2.3.1: {}
qs@6.14.1:
dependencies:
side-channel: 1.1.0
@ -10399,6 +10633,8 @@ snapshots:
transitivePeerDependencies:
- supports-color
rrweb-cssom@0.8.0: {}
run-parallel@1.2.0:
dependencies:
queue-microtask: 1.2.3
@ -10411,6 +10647,10 @@ snapshots:
sax@1.4.4: {}
saxes@6.0.0:
dependencies:
xmlchars: 2.2.0
scheduler@0.23.2:
dependencies:
loose-envify: 1.4.0
@ -10666,6 +10906,8 @@ snapshots:
picocolors: 1.1.1
sax: 1.4.4
symbol-tree@3.2.4: {}
tailwindcss@3.4.19(tsx@4.21.0)(yaml@2.8.2):
dependencies:
'@alloc/quick-lru': 5.2.0
@ -10771,12 +11013,26 @@ snapshots:
tinyspy@4.0.4: {}
tldts-core@6.1.86: {}
tldts@6.1.86:
dependencies:
tldts-core: 6.1.86
to-regex-range@5.0.1:
dependencies:
is-number: 7.0.0
toidentifier@1.0.1: {}
tough-cookie@5.1.2:
dependencies:
tldts: 6.1.86
tr46@5.1.1:
dependencies:
punycode: 2.3.1
tree-kill@1.2.2: {}
trim-lines@3.0.1: {}
@ -11103,7 +11359,7 @@ snapshots:
optionalDependencies:
vite: 6.4.1(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2)
vitest@3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2):
vitest@3.2.4(@types/debug@4.1.12)(@types/node@22.19.7)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2):
dependencies:
'@types/chai': 5.2.3
'@vitest/expect': 3.2.4
@ -11131,6 +11387,7 @@ snapshots:
optionalDependencies:
'@types/debug': 4.1.12
'@types/node': 22.19.7
jsdom: 26.1.0
transitivePeerDependencies:
- jiti
- less
@ -11145,7 +11402,7 @@ snapshots:
- tsx
- yaml
vitest@3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(tsx@4.21.0)(yaml@2.8.2):
vitest@3.2.4(@types/debug@4.1.12)(@types/node@25.3.5)(jiti@1.21.7)(jsdom@26.1.0)(tsx@4.21.0)(yaml@2.8.2):
dependencies:
'@types/chai': 5.2.3
'@vitest/expect': 3.2.4
@ -11173,6 +11430,7 @@ snapshots:
optionalDependencies:
'@types/debug': 4.1.12
'@types/node': 25.3.5
jsdom: 26.1.0
transitivePeerDependencies:
- jiti
- less
@ -11191,8 +11449,25 @@ snapshots:
vscode-languageserver-types@3.17.5: {}
w3c-xmlserializer@5.0.0:
dependencies:
xml-name-validator: 5.0.0
web-namespaces@2.0.1: {}
webidl-conversions@7.0.0: {}
whatwg-encoding@3.1.1:
dependencies:
iconv-lite: 0.6.3
whatwg-mimetype@4.0.0: {}
whatwg-url@14.2.0:
dependencies:
tr46: 5.1.1
webidl-conversions: 7.0.0
which-pm-runs@1.1.0: {}
which@2.0.2:
@ -11265,6 +11540,10 @@ snapshots:
dependencies:
os-paths: 4.4.0
xml-name-validator@5.0.0: {}
xmlchars@2.2.0: {}
xtend@4.0.2: {}
xxhash-wasm@1.1.0: {}

10
research/acp/friction.md
View file

@ -247,3 +247,13 @@ Update this file continuously during the migration.
- Owner: Unassigned.
- Status: in_progress
- Links: `research/acp/simplify-server.md`, `docs/mcp-config.mdx`, `docs/skills-config.mdx`
- Date: 2026-03-07
- Area: Desktop host/runtime API boundary
- Issue: Desktop automation needed screenshot/input/file-transfer-like host capabilities, but routing it through ACP would have mixed agent protocol semantics with host-owned runtime control and binary payloads.
- Impact: A desktop feature built as ACP methods would blur the division between agent/session behavior and Sandbox Agent host/runtime APIs, and would complicate binary screenshot transport.
- Proposed direction: Ship desktop as first-party HTTP endpoints under `/v1/desktop/*`, keep health/install/remediation in the server runtime, and expose the feature through the SDK and inspector without ACP extension methods.
- Decision: Accepted and implemented for phase one.
- Owner: Unassigned.
- Status: resolved
- Links: `server/packages/sandbox-agent/src/router.rs`, `server/packages/sandbox-agent/src/desktop_runtime.rs`, `sdks/typescript/src/client.ts`, `frontend/packages/inspector/src/components/debug/DesktopTab.tsx`

89
sdks/typescript/src/client.ts
View file

@ -25,6 +25,19 @@ import {
type AgentInstallRequest,
type AgentInstallResponse,
type AgentListResponse,
type DesktopActionResponse,
type DesktopDisplayInfoResponse,
type DesktopKeyboardPressRequest,
type DesktopKeyboardTypeRequest,
type DesktopMouseClickRequest,
type DesktopMouseDragRequest,
type DesktopMouseMoveRequest,
type DesktopMousePositionResponse,
type DesktopMouseScrollRequest,
type DesktopRegionScreenshotQuery,
type DesktopScreenshotQuery,
type DesktopStartRequest,
type DesktopStatusResponse,
type FsActionResponse,
type FsDeleteQuery,
type FsEntriesQuery,
@ -1294,6 +1307,82 @@ export class SandboxAgent {
return this.requestHealth();
}
async startDesktop(request: DesktopStartRequest = {}): Promise<DesktopStatusResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/start`, {
body: request,
});
}
async stopDesktop(): Promise<DesktopStatusResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/stop`);
}
async getDesktopStatus(): Promise<DesktopStatusResponse> {
return this.requestJson("GET", `${API_PREFIX}/desktop/status`);
}
async getDesktopDisplayInfo(): Promise<DesktopDisplayInfoResponse> {
return this.requestJson("GET", `${API_PREFIX}/desktop/display/info`);
}
async takeDesktopScreenshot(query: DesktopScreenshotQuery = {}): Promise<Uint8Array> {
const response = await this.requestRaw("GET", `${API_PREFIX}/desktop/screenshot`, {
query,
accept: "image/png",
});
const buffer = await response.arrayBuffer();
return new Uint8Array(buffer);
}
async takeDesktopRegionScreenshot(query: DesktopRegionScreenshotQuery): Promise<Uint8Array> {
const response = await this.requestRaw("GET", `${API_PREFIX}/desktop/screenshot/region`, {
query,
accept: "image/png",
});
const buffer = await response.arrayBuffer();
return new Uint8Array(buffer);
}
async getDesktopMousePosition(): Promise<DesktopMousePositionResponse> {
return this.requestJson("GET", `${API_PREFIX}/desktop/mouse/position`);
}
async moveDesktopMouse(request: DesktopMouseMoveRequest): Promise<DesktopMousePositionResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/mouse/move`, {
body: request,
});
}
async clickDesktop(request: DesktopMouseClickRequest): Promise<DesktopMousePositionResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/mouse/click`, {
body: request,
});
}
async dragDesktopMouse(request: DesktopMouseDragRequest): Promise<DesktopMousePositionResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/mouse/drag`, {
body: request,
});
}
async scrollDesktop(request: DesktopMouseScrollRequest): Promise<DesktopMousePositionResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/mouse/scroll`, {
body: request,
});
}
async typeDesktopText(request: DesktopKeyboardTypeRequest): Promise<DesktopActionResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/keyboard/type`, {
body: request,
});
}
async pressDesktopKey(request: DesktopKeyboardPressRequest): Promise<DesktopActionResponse> {
return this.requestJson("POST", `${API_PREFIX}/desktop/keyboard/press`, {
body: request,
});
}
async listAgents(options?: AgentQueryOptions): Promise<AgentListResponse> {
return this.requestJson("GET", `${API_PREFIX}/agents`, {
query: toAgentQuery(options),

651
sdks/typescript/src/generated/openapi.ts
View file

@ -32,6 +32,109 @@ export interface paths {
put: operations["put_v1_config_skills"];
delete: operations["delete_v1_config_skills"];
};
"/v1/desktop/display/info": {
/**
* Get desktop display information.
* @description Performs a health-gated display query against the managed desktop and
* returns the current display identifier and resolution.
*/
get: operations["get_v1_desktop_display_info"];
};
"/v1/desktop/keyboard/press": {
/**
* Press a desktop keyboard shortcut.
* @description Performs a health-gated `xdotool key` operation against the managed
* desktop.
*/
post: operations["post_v1_desktop_keyboard_press"];
};
"/v1/desktop/keyboard/type": {
/**
* Type desktop keyboard text.
* @description Performs a health-gated `xdotool type` operation against the managed
* desktop.
*/
post: operations["post_v1_desktop_keyboard_type"];
};
"/v1/desktop/mouse/click": {
/**
* Click on the desktop.
* @description Performs a health-gated pointer move and click against the managed desktop
* and returns the resulting mouse position.
*/
post: operations["post_v1_desktop_mouse_click"];
};
"/v1/desktop/mouse/drag": {
/**
* Drag the desktop mouse.
* @description Performs a health-gated drag gesture against the managed desktop and
* returns the resulting mouse position.
*/
post: operations["post_v1_desktop_mouse_drag"];
};
"/v1/desktop/mouse/move": {
/**
* Move the desktop mouse.
* @description Performs a health-gated absolute pointer move on the managed desktop and
* returns the resulting mouse position.
*/
post: operations["post_v1_desktop_mouse_move"];
};
"/v1/desktop/mouse/position": {
/**
* Get the current desktop mouse position.
* @description Performs a health-gated mouse position query against the managed desktop.
*/
get: operations["get_v1_desktop_mouse_position"];
};
"/v1/desktop/mouse/scroll": {
/**
* Scroll the desktop mouse wheel.
* @description Performs a health-gated scroll gesture at the requested coordinates and
* returns the resulting mouse position.
*/
post: operations["post_v1_desktop_mouse_scroll"];
};
"/v1/desktop/screenshot": {
/**
* Capture a full desktop screenshot.
* @description Performs a health-gated full-frame screenshot of the managed desktop and
* returns PNG bytes.
*/
get: operations["get_v1_desktop_screenshot"];
};
"/v1/desktop/screenshot/region": {
/**
* Capture a desktop screenshot region.
* @description Performs a health-gated screenshot crop against the managed desktop and
* returns the requested PNG region bytes.
*/
get: operations["get_v1_desktop_screenshot_region"];
};
"/v1/desktop/start": {
/**
* Start the private desktop runtime.
* @description Lazily launches the managed Xvfb/openbox stack, validates display health,
* and returns the resulting desktop status snapshot.
*/
post: operations["post_v1_desktop_start"];
};
"/v1/desktop/status": {
/**
* Get desktop runtime status.
* @description Returns the current desktop runtime state, dependency status, active
* display metadata, and supervised process information.
*/
get: operations["get_v1_desktop_status"];
};
"/v1/desktop/stop": {
/**
* Stop the private desktop runtime.
* @description Terminates the managed openbox/Xvfb/dbus processes owned by the desktop
* runtime and returns the resulting status snapshot.
*/
post: operations["post_v1_desktop_stop"];
};
"/v1/fs/entries": {
get: operations["get_v1_fs_entries"];
};
@ -234,6 +337,119 @@ export interface components {
AgentListResponse: {
agents: components["schemas"]["AgentInfo"][];
};
DesktopActionResponse: {
ok: boolean;
};
DesktopDisplayInfoResponse: {
display: string;
resolution: components["schemas"]["DesktopResolution"];
};
DesktopErrorInfo: {
code: string;
message: string;
};
DesktopKeyboardPressRequest: {
key: string;
};
DesktopKeyboardTypeRequest: {
/** Format: int32 */
delayMs?: number | null;
text: string;
};
/** @enum {string} */
DesktopMouseButton: "left" | "middle" | "right";
DesktopMouseClickRequest: {
button?: components["schemas"]["DesktopMouseButton"] | null;
/** Format: int32 */
clickCount?: number | null;
/** Format: int32 */
x: number;
/** Format: int32 */
y: number;
};
DesktopMouseDragRequest: {
button?: components["schemas"]["DesktopMouseButton"] | null;
/** Format: int32 */
endX: number;
/** Format: int32 */
endY: number;
/** Format: int32 */
startX: number;
/** Format: int32 */
startY: number;
};
DesktopMouseMoveRequest: {
/** Format: int32 */
x: number;
/** Format: int32 */
y: number;
};
DesktopMousePositionResponse: {
/** Format: int32 */
screen?: number | null;
window?: string | null;
/** Format: int32 */
x: number;
/** Format: int32 */
y: number;
};
DesktopMouseScrollRequest: {
/** Format: int32 */
deltaX?: number | null;
/** Format: int32 */
deltaY?: number | null;
/** Format: int32 */
x: number;
/** Format: int32 */
y: number;
};
DesktopProcessInfo: {
logPath?: string | null;
name: string;
/** Format: int32 */
pid?: number | null;
running: boolean;
};
DesktopRegionScreenshotQuery: {
/** Format: int32 */
height: number;
/** Format: int32 */
width: number;
/** Format: int32 */
x: number;
/** Format: int32 */
y: number;
};
DesktopResolution: {
/** Format: int32 */
dpi?: number | null;
/** Format: int32 */
height: number;
/** Format: int32 */
width: number;
};
DesktopScreenshotQuery: Record<string, never>;
DesktopStartRequest: {
/** Format: int32 */
dpi?: number | null;
/** Format: int32 */
height?: number | null;
/** Format: int32 */
width?: number | null;
};
/** @enum {string} */
DesktopState: "inactive" | "install_required" | "starting" | "active" | "stopping" | "failed";
DesktopStatusResponse: {
display?: string | null;
installCommand?: string | null;
lastError?: components["schemas"]["DesktopErrorInfo"] | null;
missingDependencies?: string[];
processes?: components["schemas"]["DesktopProcessInfo"][];
resolution?: components["schemas"]["DesktopResolution"] | null;
runtimeLogPath?: string | null;
startedAt?: string | null;
state: components["schemas"]["DesktopState"];
};
/** @enum {string} */
ErrorType: "invalid_request" | "conflict" | "unsupported_agent" | "agent_not_installed" | "install_failed" | "agent_process_exited" | "token_invalid" | "permission_denied" | "not_acceptable" | "unsupported_media_type" | "not_found" | "session_not_found" | "session_already_exists" | "mode_not_supported" | "stream_error" | "timeout";
FsActionResponse: {
@ -811,6 +1027,441 @@ export interface operations {
};
};
};
/**
* Get desktop display information.
* @description Performs a health-gated display query against the managed desktop and
* returns the current display identifier and resolution.
*/
get_v1_desktop_display_info: {
responses: {
/** @description Desktop display information */
200: {
content: {
"application/json": components["schemas"]["DesktopDisplayInfoResponse"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or display query failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Press a desktop keyboard shortcut.
* @description Performs a health-gated `xdotool key` operation against the managed
* desktop.
*/
post_v1_desktop_keyboard_press: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopKeyboardPressRequest"];
};
};
responses: {
/** @description Desktop keyboard action result */
200: {
content: {
"application/json": components["schemas"]["DesktopActionResponse"];
};
};
/** @description Invalid keyboard press request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Type desktop keyboard text.
* @description Performs a health-gated `xdotool type` operation against the managed
* desktop.
*/
post_v1_desktop_keyboard_type: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopKeyboardTypeRequest"];
};
};
responses: {
/** @description Desktop keyboard action result */
200: {
content: {
"application/json": components["schemas"]["DesktopActionResponse"];
};
};
/** @description Invalid keyboard type request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Click on the desktop.
* @description Performs a health-gated pointer move and click against the managed desktop
* and returns the resulting mouse position.
*/
post_v1_desktop_mouse_click: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopMouseClickRequest"];
};
};
responses: {
/** @description Desktop mouse position after click */
200: {
content: {
"application/json": components["schemas"]["DesktopMousePositionResponse"];
};
};
/** @description Invalid mouse click request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Drag the desktop mouse.
* @description Performs a health-gated drag gesture against the managed desktop and
* returns the resulting mouse position.
*/
post_v1_desktop_mouse_drag: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopMouseDragRequest"];
};
};
responses: {
/** @description Desktop mouse position after drag */
200: {
content: {
"application/json": components["schemas"]["DesktopMousePositionResponse"];
};
};
/** @description Invalid mouse drag request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Move the desktop mouse.
* @description Performs a health-gated absolute pointer move on the managed desktop and
* returns the resulting mouse position.
*/
post_v1_desktop_mouse_move: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopMouseMoveRequest"];
};
};
responses: {
/** @description Desktop mouse position after move */
200: {
content: {
"application/json": components["schemas"]["DesktopMousePositionResponse"];
};
};
/** @description Invalid mouse move request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Get the current desktop mouse position.
* @description Performs a health-gated mouse position query against the managed desktop.
*/
get_v1_desktop_mouse_position: {
responses: {
/** @description Desktop mouse position */
200: {
content: {
"application/json": components["schemas"]["DesktopMousePositionResponse"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input check failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Scroll the desktop mouse wheel.
* @description Performs a health-gated scroll gesture at the requested coordinates and
* returns the resulting mouse position.
*/
post_v1_desktop_mouse_scroll: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopMouseScrollRequest"];
};
};
responses: {
/** @description Desktop mouse position after scroll */
200: {
content: {
"application/json": components["schemas"]["DesktopMousePositionResponse"];
};
};
/** @description Invalid mouse scroll request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or input failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Capture a full desktop screenshot.
* @description Performs a health-gated full-frame screenshot of the managed desktop and
* returns PNG bytes.
*/
get_v1_desktop_screenshot: {
responses: {
/** @description Desktop screenshot as PNG bytes */
200: {
content: never;
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or screenshot capture failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Capture a desktop screenshot region.
* @description Performs a health-gated screenshot crop against the managed desktop and
* returns the requested PNG region bytes.
*/
get_v1_desktop_screenshot_region: {
parameters: {
query: {
/** @description Region x coordinate */
x: number;
/** @description Region y coordinate */
y: number;
/** @description Region width */
width: number;
/** @description Region height */
height: number;
};
};
responses: {
/** @description Desktop screenshot region as PNG bytes */
200: {
content: never;
};
/** @description Invalid screenshot region */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is not ready */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime health or screenshot capture failed */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Start the private desktop runtime.
* @description Lazily launches the managed Xvfb/openbox stack, validates display health,
* and returns the resulting desktop status snapshot.
*/
post_v1_desktop_start: {
requestBody: {
content: {
"application/json": components["schemas"]["DesktopStartRequest"];
};
};
responses: {
/** @description Desktop runtime status after start */
200: {
content: {
"application/json": components["schemas"]["DesktopStatusResponse"];
};
};
/** @description Invalid desktop start request */
400: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime is already transitioning */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop API unsupported on this platform */
501: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
/** @description Desktop runtime could not be started */
503: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Get desktop runtime status.
* @description Returns the current desktop runtime state, dependency status, active
* display metadata, and supervised process information.
*/
get_v1_desktop_status: {
responses: {
/** @description Desktop runtime status */
200: {
content: {
"application/json": components["schemas"]["DesktopStatusResponse"];
};
};
/** @description Authentication required */
401: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
/**
* Stop the private desktop runtime.
* @description Terminates the managed openbox/Xvfb/dbus processes owned by the desktop
* runtime and returns the resulting status snapshot.
*/
post_v1_desktop_stop: {
responses: {
/** @description Desktop runtime status after stop */
200: {
content: {
"application/json": components["schemas"]["DesktopStatusResponse"];
};
};
/** @description Desktop runtime is already transitioning */
409: {
content: {
"application/json": components["schemas"]["ProblemDetails"];
};
};
};
};
get_v1_fs_entries: {
parameters: {
query?: {

18
sdks/typescript/src/index.ts
View file

@ -45,6 +45,24 @@ export type {
AgentInstallRequest,
AgentInstallResponse,
AgentListResponse,
DesktopActionResponse,
DesktopDisplayInfoResponse,
DesktopErrorInfo,
DesktopKeyboardPressRequest,
DesktopKeyboardTypeRequest,
DesktopMouseButton,
DesktopMouseClickRequest,
DesktopMouseDragRequest,
DesktopMouseMoveRequest,
DesktopMousePositionResponse,
DesktopMouseScrollRequest,
DesktopProcessInfo,
DesktopRegionScreenshotQuery,
DesktopResolution,
DesktopScreenshotQuery,
DesktopStartRequest,
DesktopState,
DesktopStatusResponse,
FsActionResponse,
FsDeleteQuery,
FsEntriesQuery,

21
sdks/typescript/src/types.ts
View file

@ -9,6 +9,27 @@ import type { components, operations } from "./generated/openapi.ts";
export type ProblemDetails = components["schemas"]["ProblemDetails"];
export type HealthResponse = JsonResponse<operations["get_v1_health"], 200>;
export type DesktopState = components["schemas"]["DesktopState"];
export type DesktopResolution = components["schemas"]["DesktopResolution"];
export type DesktopErrorInfo = components["schemas"]["DesktopErrorInfo"];
export type DesktopProcessInfo = components["schemas"]["DesktopProcessInfo"];
export type DesktopStatusResponse = JsonResponse<operations["get_v1_desktop_status"], 200>;
export type DesktopStartRequest = JsonRequestBody<operations["post_v1_desktop_start"]>;
export type DesktopScreenshotQuery =
QueryParams<operations["get_v1_desktop_screenshot"]> extends never
? Record<string, never>
: QueryParams<operations["get_v1_desktop_screenshot"]>;
export type DesktopRegionScreenshotQuery = QueryParams<operations["get_v1_desktop_screenshot_region"]>;
export type DesktopMousePositionResponse = JsonResponse<operations["get_v1_desktop_mouse_position"], 200>;
export type DesktopMouseButton = components["schemas"]["DesktopMouseButton"];
export type DesktopMouseMoveRequest = JsonRequestBody<operations["post_v1_desktop_mouse_move"]>;
export type DesktopMouseClickRequest = JsonRequestBody<operations["post_v1_desktop_mouse_click"]>;
export type DesktopMouseDragRequest = JsonRequestBody<operations["post_v1_desktop_mouse_drag"]>;
export type DesktopMouseScrollRequest = JsonRequestBody<operations["post_v1_desktop_mouse_scroll"]>;
export type DesktopKeyboardTypeRequest = JsonRequestBody<operations["post_v1_desktop_keyboard_type"]>;
export type DesktopKeyboardPressRequest = JsonRequestBody<operations["post_v1_desktop_keyboard_press"]>;
export type DesktopActionResponse = JsonResponse<operations["post_v1_desktop_keyboard_type"], 200>;
export type DesktopDisplayInfoResponse = JsonResponse<operations["get_v1_desktop_display_info"], 200>;
export type AgentListResponse = JsonResponse<operations["get_v1_agents"], 200>;
export type AgentInfo = components["schemas"]["AgentInfo"];
export type AgentQuery = QueryParams<operations["get_v1_agents"]>;

237
sdks/typescript/tests/integration.test.ts
View file

@ -1,6 +1,6 @@
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { existsSync } from "node:fs";
import { mkdtempSync, rmSync } from "node:fs";
import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
@ -150,21 +150,166 @@ function nodeCommand(source: string): { command: string; args: string[] } {
};
}
function writeExecutable(path: string, source: string): void {
writeFileSync(path, source, "utf8");
chmodSync(path, 0o755);
}
function prepareFakeDesktopEnv(root: string): Record<string, string> {
const binDir = join(root, "bin");
const xdgStateHome = join(root, "xdg-state");
const fakeStateDir = join(root, "fake-state");
mkdirSync(binDir, { recursive: true });
mkdirSync(xdgStateHome, { recursive: true });
mkdirSync(fakeStateDir, { recursive: true });
writeExecutable(
join(binDir, "Xvfb"),
`#!/usr/bin/env sh
set -eu
display="\${1:-:191}"
number="\${display#:}"
socket="/tmp/.X11-unix/X\${number}"
mkdir -p /tmp/.X11-unix
touch "$socket"
cleanup() {
rm -f "$socket"
exit 0
}
trap cleanup INT TERM EXIT
while :; do
sleep 1
done
`,
);
writeExecutable(
join(binDir, "openbox"),
`#!/usr/bin/env sh
set -eu
trap 'exit 0' INT TERM
while :; do
sleep 1
done
`,
);
writeExecutable(
join(binDir, "dbus-launch"),
`#!/usr/bin/env sh
set -eu
echo "DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/sandbox-agent-test-bus"
echo "DBUS_SESSION_BUS_PID=$$"
`,
);
writeExecutable(
join(binDir, "xrandr"),
`#!/usr/bin/env sh
set -eu
cat <<'EOF'
Screen 0: minimum 1 x 1, current 1440 x 900, maximum 32767 x 32767
EOF
`,
);
writeExecutable(
join(binDir, "import"),
`#!/usr/bin/env sh
set -eu
printf '\\211PNG\\r\\n\\032\\n\\000\\000\\000\\rIHDR\\000\\000\\000\\001\\000\\000\\000\\001\\010\\006\\000\\000\\000\\037\\025\\304\\211\\000\\000\\000\\013IDATx\\234c\\000\\001\\000\\000\\005\\000\\001\\r\\n-\\264\\000\\000\\000\\000IEND\\256B\`\\202'
`,
);
writeExecutable(
join(binDir, "xdotool"),
`#!/usr/bin/env sh
set -eu
state_dir="\${SANDBOX_AGENT_DESKTOP_FAKE_STATE_DIR:?missing fake state dir}"
state_file="\${state_dir}/mouse"
mkdir -p "$state_dir"
if [ ! -f "$state_file" ]; then
printf '0 0\\n' > "$state_file"
fi
read_state() {
read -r x y < "$state_file"
}
write_state() {
printf '%s %s\\n' "$1" "$2" > "$state_file"
}
command="\${1:-}"
case "$command" in
getmouselocation)
read_state
printf 'X=%s\\nY=%s\\nSCREEN=0\\nWINDOW=0\\n' "$x" "$y"
;;
mousemove)
shift
x="\${1:-0}"
y="\${2:-0}"
shift 2 || true
while [ "$#" -gt 0 ]; do
token="$1"
shift
case "$token" in
mousemove)
x="\${1:-0}"
y="\${2:-0}"
shift 2 || true
;;
mousedown|mouseup)
shift 1 || true
;;
click)
if [ "\${1:-}" = "--repeat" ]; then
shift 2 || true
fi
shift 1 || true
;;
esac
done
write_state "$x" "$y"
;;
type|key)
exit 0
;;
*)
exit 0
;;
esac
`,
);
return {
SANDBOX_AGENT_DESKTOP_TEST_ASSUME_LINUX: "1",
SANDBOX_AGENT_DESKTOP_DISPLAY_NUM: "191",
SANDBOX_AGENT_DESKTOP_FAKE_STATE_DIR: fakeStateDir,
XDG_STATE_HOME: xdgStateHome,
PATH: `${binDir}:${process.env.PATH ?? ""}`,
};
}
describe("Integration: TypeScript SDK flat session API", () => {
let handle: SandboxAgentSpawnHandle;
let baseUrl: string;
let token: string;
let dataHome: string;
let desktopHome: string;
beforeAll(async () => {
dataHome = mkdtempSync(join(tmpdir(), "sdk-integration-"));
desktopHome = mkdtempSync(join(tmpdir(), "sdk-desktop-"));
const agentEnv = prepareMockAgentDataHome(dataHome);
const desktopEnv = prepareFakeDesktopEnv(desktopHome);
handle = await spawnSandboxAgent({
enabled: true,
log: "silent",
timeoutMs: 30000,
env: agentEnv,
env: { ...agentEnv, ...desktopEnv },
});
baseUrl = handle.baseUrl;
token = handle.token;
@ -173,6 +318,7 @@ describe("Integration: TypeScript SDK flat session API", () => {
afterAll(async () => {
await handle.dispose();
rmSync(dataHome, { recursive: true, force: true });
rmSync(desktopHome, { recursive: true, force: true });
});
it("detects Node.js runtime", () => {
@ -882,4 +1028,91 @@ describe("Integration: TypeScript SDK flat session API", () => {
await sdk.dispose();
}
});
it("covers desktop status, screenshot, display, mouse, and keyboard helpers", async () => {
const sdk = await SandboxAgent.connect({
baseUrl,
token,
});
try {
const initialStatus = await sdk.getDesktopStatus();
expect(initialStatus.state).toBe("inactive");
const started = await sdk.startDesktop({
width: 1440,
height: 900,
dpi: 96,
});
expect(started.state).toBe("active");
expect(started.display?.startsWith(":")).toBe(true);
expect(started.missingDependencies).toEqual([]);
const displayInfo = await sdk.getDesktopDisplayInfo();
expect(displayInfo.display).toBe(started.display);
expect(displayInfo.resolution.width).toBe(1440);
expect(displayInfo.resolution.height).toBe(900);
const screenshot = await sdk.takeDesktopScreenshot();
expect(Buffer.from(screenshot.subarray(0, 8)).equals(Buffer.from("\x89PNG\r\n\x1a\n", "binary"))).toBe(true);
const region = await sdk.takeDesktopRegionScreenshot({
x: 10,
y: 20,
width: 40,
height: 50,
});
expect(Buffer.from(region.subarray(0, 8)).equals(Buffer.from("\x89PNG\r\n\x1a\n", "binary"))).toBe(true);
const moved = await sdk.moveDesktopMouse({ x: 40, y: 50 });
expect(moved.x).toBe(40);
expect(moved.y).toBe(50);
const dragged = await sdk.dragDesktopMouse({
startX: 40,
startY: 50,
endX: 80,
endY: 90,
button: "left",
});
expect(dragged.x).toBe(80);
expect(dragged.y).toBe(90);
const clicked = await sdk.clickDesktop({
x: 80,
y: 90,
button: "left",
clickCount: 1,
});
expect(clicked.x).toBe(80);
expect(clicked.y).toBe(90);
const scrolled = await sdk.scrollDesktop({
x: 80,
y: 90,
deltaY: -2,
});
expect(scrolled.x).toBe(80);
expect(scrolled.y).toBe(90);
const position = await sdk.getDesktopMousePosition();
expect(position.x).toBe(80);
expect(position.y).toBe(90);
const typed = await sdk.typeDesktopText({
text: "hello desktop",
delayMs: 5,
});
expect(typed.ok).toBe(true);
const pressed = await sdk.pressDesktopKey({ key: "ctrl+l" });
expect(pressed.ok).toBe(true);
const stopped = await sdk.stopDesktop();
expect(stopped.state).toBe("inactive");
} finally {
await sdk.stopDesktop().catch(() => {});
await sdk.dispose();
}
});
});

47
server/packages/sandbox-agent/src/cli.rs
View file

@ -11,6 +11,9 @@ mod build_version {
include!(concat!(env!("OUT_DIR"), "/version.rs"));
}
use crate::desktop_install::{
install_desktop, DesktopInstallRequest, DesktopPackageManager,
};
use crate::router::{
build_router_with_state, shutdown_servers, AppState, AuthConfig, BrandingMode,
};
@ -75,6 +78,8 @@ pub enum Command {
Server(ServerArgs),
/// Call the HTTP API without writing client code.
Api(ApiArgs),
/// Install first-party runtime dependencies.
Install(InstallArgs),
/// EXPERIMENTAL: OpenCode compatibility layer (disabled until ACP Phase 7).
Opencode(OpencodeArgs),
/// Manage the sandbox-agent background daemon.
@ -115,6 +120,12 @@ pub struct ApiArgs {
command: ApiCommand,
}
#[derive(Args, Debug)]
pub struct InstallArgs {
#[command(subcommand)]
command: InstallCommand,
}
#[derive(Args, Debug)]
pub struct OpencodeArgs {
#[arg(long, short = 'H', default_value = DEFAULT_HOST)]
@ -153,6 +164,12 @@ pub struct DaemonArgs {
command: DaemonCommand,
}
#[derive(Subcommand, Debug)]
pub enum InstallCommand {
/// Install desktop runtime dependencies.
Desktop(InstallDesktopArgs),
}
#[derive(Subcommand, Debug)]
pub enum DaemonCommand {
/// Start the daemon in the background.
@ -304,6 +321,18 @@ pub struct InstallAgentArgs {
agent_process_version: Option<String>,
}
#[derive(Args, Debug)]
pub struct InstallDesktopArgs {
#[arg(long, default_value_t = false)]
yes: bool,
#[arg(long, default_value_t = false)]
print_only: bool,
#[arg(long, value_enum)]
package_manager: Option<DesktopPackageManager>,
#[arg(long, default_value_t = false)]
no_fonts: bool,
}
#[derive(Args, Debug)]
pub struct CredentialsExtractArgs {
#[arg(long, short = 'a', value_enum)]
@ -399,6 +428,7 @@ pub fn run_command(command: &Command, cli: &CliConfig) -> Result<(), CliError> {
match command {
Command::Server(args) => run_server(cli, args),
Command::Api(subcommand) => run_api(&subcommand.command, cli),
Command::Install(subcommand) => run_install(&subcommand.command),
Command::Opencode(args) => run_opencode(cli, args),
Command::Daemon(subcommand) => run_daemon(&subcommand.command, cli),
Command::InstallAgent(args) => install_agent_local(args),
@ -406,6 +436,12 @@ pub fn run_command(command: &Command, cli: &CliConfig) -> Result<(), CliError> {
}
}
fn run_install(command: &InstallCommand) -> Result<(), CliError> {
match command {
InstallCommand::Desktop(args) => install_desktop_local(args),
}
}
fn run_server(cli: &CliConfig, server: &ServerArgs) -> Result<(), CliError> {
let auth = if let Some(token) = cli.token.clone() {
AuthConfig::with_token(token)
@ -470,6 +506,17 @@ fn run_api(command: &ApiCommand, cli: &CliConfig) -> Result<(), CliError> {
}
}
fn install_desktop_local(args: &InstallDesktopArgs) -> Result<(), CliError> {
install_desktop(DesktopInstallRequest {
yes: args.yes,
print_only: args.print_only,
package_manager: args.package_manager,
no_fonts: args.no_fonts,
})
.map(|_| ())
.map_err(CliError::Server)
}
fn run_agents(command: &AgentsCommand, cli: &CliConfig) -> Result<(), CliError> {
match command {
AgentsCommand::List(args) => {

158
server/packages/sandbox-agent/src/desktop_errors.rs Normal file
View file

@ -0,0 +1,158 @@
use sandbox_agent_error::ProblemDetails;
use serde_json::{json, Map, Value};
use crate::desktop_types::{DesktopErrorInfo, DesktopProcessInfo};
#[derive(Debug, Clone)]
pub struct DesktopProblem {
status: u16,
title: &'static str,
code: &'static str,
message: String,
missing_dependencies: Vec<String>,
install_command: Option<String>,
processes: Vec<DesktopProcessInfo>,
}
impl DesktopProblem {
pub fn unsupported_platform(message: impl Into<String>) -> Self {
Self::new(501, "Desktop Unsupported", "desktop_unsupported_platform", message)
}
pub fn dependencies_missing(
missing_dependencies: Vec<String>,
install_command: Option<String>,
processes: Vec<DesktopProcessInfo>,
) -> Self {
let message = if missing_dependencies.is_empty() {
"Desktop dependencies are not installed".to_string()
} else {
format!(
"Desktop dependencies are not installed: {}",
missing_dependencies.join(", ")
)
};
Self::new(
503,
"Desktop Dependencies Missing",
"desktop_dependencies_missing",
message,
)
.with_missing_dependencies(missing_dependencies)
.with_install_command(install_command)
.with_processes(processes)
}
pub fn runtime_inactive(message: impl Into<String>) -> Self {
Self::new(409, "Desktop Runtime Inactive", "desktop_runtime_inactive", message)
}
pub fn runtime_starting(message: impl Into<String>) -> Self {
Self::new(409, "Desktop Runtime Starting", "desktop_runtime_starting", message)
}
pub fn runtime_failed(
message: impl Into<String>,
install_command: Option<String>,
processes: Vec<DesktopProcessInfo>,
) -> Self {
Self::new(503, "Desktop Runtime Failed", "desktop_runtime_failed", message)
.with_install_command(install_command)
.with_processes(processes)
}
pub fn invalid_action(message: impl Into<String>) -> Self {
Self::new(400, "Desktop Invalid Action", "desktop_invalid_action", message)
}
pub fn screenshot_failed(message: impl Into<String>, processes: Vec<DesktopProcessInfo>) -> Self {
Self::new(502, "Desktop Screenshot Failed", "desktop_screenshot_failed", message)
.with_processes(processes)
}
pub fn input_failed(message: impl Into<String>, processes: Vec<DesktopProcessInfo>) -> Self {
Self::new(502, "Desktop Input Failed", "desktop_input_failed", message)
.with_processes(processes)
}
pub fn to_problem_details(&self) -> ProblemDetails {
let mut extensions = Map::new();
extensions.insert("code".to_string(), Value::String(self.code.to_string()));
if !self.missing_dependencies.is_empty() {
extensions.insert(
"missingDependencies".to_string(),
Value::Array(
self.missing_dependencies
.iter()
.cloned()
.map(Value::String)
.collect(),
),
);
}
if let Some(install_command) = self.install_command.as_ref() {
extensions.insert(
"installCommand".to_string(),
Value::String(install_command.clone()),
);
}
if !self.processes.is_empty() {
extensions.insert(
"processes".to_string(),
json!(self.processes),
);
}
ProblemDetails {
type_: format!("urn:sandbox-agent:error:{}", self.code),
title: self.title.to_string(),
status: self.status,
detail: Some(self.message.clone()),
instance: None,
extensions,
}
}
pub fn to_error_info(&self) -> DesktopErrorInfo {
DesktopErrorInfo {
code: self.code.to_string(),
message: self.message.clone(),
}
}
pub fn code(&self) -> &'static str {
self.code
}
fn new(
status: u16,
title: &'static str,
code: &'static str,
message: impl Into<String>,
) -> Self {
Self {
status,
title,
code,
message: message.into(),
missing_dependencies: Vec::new(),
install_command: None,
processes: Vec::new(),
}
}
fn with_missing_dependencies(mut self, missing_dependencies: Vec<String>) -> Self {
self.missing_dependencies = missing_dependencies;
self
}
fn with_install_command(mut self, install_command: Option<String>) -> Self {
self.install_command = install_command;
self
}
fn with_processes(mut self, processes: Vec<DesktopProcessInfo>) -> Self {
self.processes = processes;
self
}
}

282
server/packages/sandbox-agent/src/desktop_install.rs Normal file
View file

@ -0,0 +1,282 @@
use std::fmt;
use std::io::{self, Write};
use std::path::PathBuf;
use std::process::Command as ProcessCommand;
use clap::ValueEnum;
#[derive(Debug, Clone, Copy, PartialEq, Eq, ValueEnum)]
pub enum DesktopPackageManager {
Apt,
Dnf,
Apk,
}
#[derive(Debug, Clone)]
pub struct DesktopInstallRequest {
pub yes: bool,
pub print_only: bool,
pub package_manager: Option<DesktopPackageManager>,
pub no_fonts: bool,
}
pub fn install_desktop(request: DesktopInstallRequest) -> Result<(), String> {
if std::env::consts::OS != "linux" {
return Err("desktop installation is only supported on Linux hosts and sandboxes".to_string());
}
let package_manager = match request.package_manager {
Some(value) => value,
None => detect_package_manager().ok_or_else(|| {
"could not detect a supported package manager (expected apt, dnf, or apk)".to_string()
})?,
};
let packages = desktop_packages(package_manager, request.no_fonts);
let used_sudo = !running_as_root() && find_binary("sudo").is_some();
if !running_as_root() && !used_sudo {
return Err(
"desktop installation requires root or sudo access; rerun as root or install dependencies manually"
.to_string(),
);
}
println!("Desktop package manager: {}", package_manager);
println!("Desktop packages:");
for package in &packages {
println!(" - {package}");
}
println!("Install command:");
println!(" {}", render_install_command(package_manager, used_sudo, &packages));
if request.print_only {
return Ok(());
}
if !request.yes && !prompt_yes_no("Proceed with desktop dependency installation? [y/N] ")? {
return Err("installation cancelled".to_string());
}
run_install_commands(package_manager, used_sudo, &packages)?;
println!("Desktop dependencies installed.");
Ok(())
}
fn detect_package_manager() -> Option<DesktopPackageManager> {
if find_binary("apt-get").is_some() {
return Some(DesktopPackageManager::Apt);
}
if find_binary("dnf").is_some() {
return Some(DesktopPackageManager::Dnf);
}
if find_binary("apk").is_some() {
return Some(DesktopPackageManager::Apk);
}
None
}
fn desktop_packages(
package_manager: DesktopPackageManager,
no_fonts: bool,
) -> Vec<String> {
let mut packages = match package_manager {
DesktopPackageManager::Apt => vec![
"xvfb",
"openbox",
"xdotool",
"imagemagick",
"x11-xserver-utils",
"dbus-x11",
"xauth",
"fonts-dejavu-core",
],
DesktopPackageManager::Dnf => vec![
"xorg-x11-server-Xvfb",
"openbox",
"xdotool",
"ImageMagick",
"xrandr",
"dbus-x11",
"xauth",
"dejavu-sans-fonts",
],
DesktopPackageManager::Apk => vec![
"xvfb",
"openbox",
"xdotool",
"imagemagick",
"xrandr",
"dbus",
"xauth",
"ttf-dejavu",
],
}
.into_iter()
.map(str::to_string)
.collect::<Vec<_>>();
if no_fonts {
packages.retain(|package| {
package != "fonts-dejavu-core"
&& package != "dejavu-sans-fonts"
&& package != "ttf-dejavu"
});
}
packages
}
fn render_install_command(
package_manager: DesktopPackageManager,
used_sudo: bool,
packages: &[String],
) -> String {
let sudo = if used_sudo { "sudo " } else { "" };
match package_manager {
DesktopPackageManager::Apt => format!(
"{sudo}apt-get update && {sudo}env DEBIAN_FRONTEND=noninteractive apt-get install -y {}",
packages.join(" ")
),
DesktopPackageManager::Dnf => {
format!("{sudo}dnf install -y {}", packages.join(" "))
}
DesktopPackageManager::Apk => {
format!("{sudo}apk add --no-cache {}", packages.join(" "))
}
}
}
fn run_install_commands(
package_manager: DesktopPackageManager,
used_sudo: bool,
packages: &[String],
) -> Result<(), String> {
match package_manager {
DesktopPackageManager::Apt => {
run_command(command_with_privilege(
used_sudo,
"apt-get",
vec!["update".to_string()],
))?;
let mut args = vec![
"DEBIAN_FRONTEND=noninteractive".to_string(),
"apt-get".to_string(),
"install".to_string(),
"-y".to_string(),
];
args.extend(packages.iter().cloned());
run_command(command_with_privilege(used_sudo, "env", args))?;
}
DesktopPackageManager::Dnf => {
let mut args = vec!["install".to_string(), "-y".to_string()];
args.extend(packages.iter().cloned());
run_command(command_with_privilege(used_sudo, "dnf", args))?;
}
DesktopPackageManager::Apk => {
let mut args = vec!["add".to_string(), "--no-cache".to_string()];
args.extend(packages.iter().cloned());
run_command(command_with_privilege(used_sudo, "apk", args))?;
}
}
Ok(())
}
fn command_with_privilege(
used_sudo: bool,
program: &str,
args: Vec<String>,
) -> (String, Vec<String>) {
if used_sudo {
let mut sudo_args = vec![program.to_string()];
sudo_args.extend(args);
("sudo".to_string(), sudo_args)
} else {
(program.to_string(), args)
}
}
fn run_command((program, args): (String, Vec<String>)) -> Result<(), String> {
let status = ProcessCommand::new(&program)
.args(&args)
.status()
.map_err(|err| format!("failed to run `{program}`: {err}"))?;
if !status.success() {
return Err(format!(
"command `{}` exited with status {}",
format_command(&program, &args),
status
));
}
Ok(())
}
fn prompt_yes_no(prompt: &str) -> Result<bool, String> {
print!("{prompt}");
io::stdout()
.flush()
.map_err(|err| format!("failed to flush prompt: {err}"))?;
let mut input = String::new();
io::stdin()
.read_line(&mut input)
.map_err(|err| format!("failed to read confirmation: {err}"))?;
let normalized = input.trim().to_ascii_lowercase();
Ok(matches!(normalized.as_str(), "y" | "yes"))
}
fn running_as_root() -> bool {
#[cfg(unix)]
unsafe {
return libc::geteuid() == 0;
}
#[cfg(not(unix))]
{
false
}
}
fn find_binary(name: &str) -> Option<PathBuf> {
let path_env = std::env::var_os("PATH")?;
for path in std::env::split_paths(&path_env) {
let candidate = path.join(name);
if candidate.is_file() {
return Some(candidate);
}
}
None
}
fn format_command(program: &str, args: &[String]) -> String {
let mut parts = vec![program.to_string()];
parts.extend(args.iter().cloned());
parts.join(" ")
}
impl fmt::Display for DesktopPackageManager {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
DesktopPackageManager::Apt => write!(f, "apt"),
DesktopPackageManager::Dnf => write!(f, "dnf"),
DesktopPackageManager::Apk => write!(f, "apk"),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn desktop_packages_support_no_fonts() {
let packages = desktop_packages(DesktopPackageManager::Apt, true);
assert!(!packages.iter().any(|value| value == "fonts-dejavu-core"));
assert!(packages.iter().any(|value| value == "xvfb"));
}
#[test]
fn render_install_command_matches_package_manager() {
let packages = vec!["xvfb".to_string(), "openbox".to_string()];
let command = render_install_command(DesktopPackageManager::Apk, false, &packages);
assert_eq!(command, "apk add --no-cache xvfb openbox");
}
}

1313
server/packages/sandbox-agent/src/desktop_runtime.rs Normal file
View file

File diff suppressed because it is too large Load diff

173
server/packages/sandbox-agent/src/desktop_types.rs Normal file
View file

@ -0,0 +1,173 @@
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use utoipa::ToSchema;
#[derive(Debug, Clone, Copy, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum DesktopState {
Inactive,
InstallRequired,
Starting,
Active,
Stopping,
Failed,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopResolution {
pub width: u32,
pub height: u32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub dpi: Option<u32>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopErrorInfo {
pub code: String,
pub message: String,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopProcessInfo {
pub name: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub pid: Option<u32>,
pub running: bool,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub log_path: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopStatusResponse {
pub state: DesktopState,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub display: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub resolution: Option<DesktopResolution>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub started_at: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub last_error: Option<DesktopErrorInfo>,
#[serde(default)]
pub missing_dependencies: Vec<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub install_command: Option<String>,
#[serde(default)]
pub processes: Vec<DesktopProcessInfo>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub runtime_log_path: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, Default)]
#[serde(rename_all = "camelCase")]
pub struct DesktopStartRequest {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub width: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub height: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub dpi: Option<u32>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, Default)]
#[serde(rename_all = "camelCase")]
pub struct DesktopScreenshotQuery {}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopRegionScreenshotQuery {
pub x: i32,
pub y: i32,
pub width: u32,
pub height: u32,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopMousePositionResponse {
pub x: i32,
pub y: i32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub screen: Option<i32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub window: Option<String>,
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum DesktopMouseButton {
Left,
Middle,
Right,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopMouseMoveRequest {
pub x: i32,
pub y: i32,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopMouseClickRequest {
pub x: i32,
pub y: i32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub button: Option<DesktopMouseButton>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub click_count: Option<u32>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopMouseDragRequest {
pub start_x: i32,
pub start_y: i32,
pub end_x: i32,
pub end_y: i32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub button: Option<DesktopMouseButton>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopMouseScrollRequest {
pub x: i32,
pub y: i32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delta_x: Option<i32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delta_y: Option<i32>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopKeyboardTypeRequest {
pub text: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delay_ms: Option<u32>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct DesktopKeyboardPressRequest {
pub key: String,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopActionResponse {
pub ok: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, ToSchema, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct DesktopDisplayInfoResponse {
pub display: String,
pub resolution: DesktopResolution,
}

4
server/packages/sandbox-agent/src/lib.rs
View file

@ -1,6 +1,10 @@
//! Sandbox agent core utilities.
mod acp_proxy_runtime;
mod desktop_install;
mod desktop_errors;
mod desktop_runtime;
pub mod desktop_types;
pub mod cli;
pub mod daemon;
mod process_runtime;

363
server/packages/sandbox-agent/src/router.rs
View file

@ -37,6 +37,9 @@ use tracing::Span;
use utoipa::{Modify, OpenApi, ToSchema};
use crate::acp_proxy_runtime::{AcpProxyRuntime, ProxyPostOutcome};
use crate::desktop_errors::DesktopProblem;
use crate::desktop_runtime::DesktopRuntime;
use crate::desktop_types::*;
use crate::process_runtime::{
decode_input_bytes, ProcessLogFilter, ProcessLogFilterStream, ProcessRuntime,
ProcessRuntimeConfig, ProcessSnapshot, ProcessStartSpec, ProcessStatus, ProcessStream, RunSpec,
@ -87,6 +90,7 @@ pub struct AppState {
acp_proxy: Arc<AcpProxyRuntime>,
opencode_server_manager: Arc<OpenCodeServerManager>,
process_runtime: Arc<ProcessRuntime>,
desktop_runtime: Arc<DesktopRuntime>,
pub(crate) branding: BrandingMode,
version_cache: Mutex<HashMap<AgentId, CachedAgentVersion>>,
}
@ -111,12 +115,14 @@ impl AppState {
},
));
let process_runtime = Arc::new(ProcessRuntime::new());
let desktop_runtime = Arc::new(DesktopRuntime::new());
Self {
auth,
agent_manager,
acp_proxy,
opencode_server_manager,
process_runtime,
desktop_runtime,
branding,
version_cache: Mutex::new(HashMap::new()),
}
@ -138,6 +144,10 @@ impl AppState {
self.process_runtime.clone()
}
pub(crate) fn desktop_runtime(&self) -> Arc<DesktopRuntime> {
self.desktop_runtime.clone()
}
pub(crate) fn purge_version_cache(&self, agent: AgentId) {
self.version_cache.lock().unwrap().remove(&agent);
}
@ -172,6 +182,22 @@ pub fn build_router(state: AppState) -> Router {
pub fn build_router_with_state(shared: Arc<AppState>) -> (Router, Arc<AppState>) {
let mut v1_router = Router::new()
.route("/health", get(get_v1_health))
.route("/desktop/status", get(get_v1_desktop_status))
.route("/desktop/start", post(post_v1_desktop_start))
.route("/desktop/stop", post(post_v1_desktop_stop))
.route("/desktop/screenshot", get(get_v1_desktop_screenshot))
.route(
"/desktop/screenshot/region",
get(get_v1_desktop_screenshot_region),
)
.route("/desktop/mouse/position", get(get_v1_desktop_mouse_position))
.route("/desktop/mouse/move", post(post_v1_desktop_mouse_move))
.route("/desktop/mouse/click", post(post_v1_desktop_mouse_click))
.route("/desktop/mouse/drag", post(post_v1_desktop_mouse_drag))
.route("/desktop/mouse/scroll", post(post_v1_desktop_mouse_scroll))
.route("/desktop/keyboard/type", post(post_v1_desktop_keyboard_type))
.route("/desktop/keyboard/press", post(post_v1_desktop_keyboard_press))
.route("/desktop/display/info", get(get_v1_desktop_display_info))
.route("/agents", get(get_v1_agents))
.route("/agents/:agent", get(get_v1_agent))
.route("/agents/:agent/install", post(post_v1_agent_install))
@ -316,12 +342,26 @@ async fn opencode_unavailable() -> Response {
pub async fn shutdown_servers(state: &Arc<AppState>) {
state.acp_proxy().shutdown_all().await;
state.opencode_server_manager().shutdown().await;
state.desktop_runtime().shutdown().await;
}
#[derive(OpenApi)]
#[openapi(
paths(
get_v1_health,
get_v1_desktop_status,
post_v1_desktop_start,
post_v1_desktop_stop,
get_v1_desktop_screenshot,
get_v1_desktop_screenshot_region,
get_v1_desktop_mouse_position,
post_v1_desktop_mouse_move,
post_v1_desktop_mouse_click,
post_v1_desktop_mouse_drag,
post_v1_desktop_mouse_scroll,
post_v1_desktop_keyboard_type,
post_v1_desktop_keyboard_press,
get_v1_desktop_display_info,
get_v1_agents,
get_v1_agent,
post_v1_agent_install,
@ -360,6 +400,24 @@ pub async fn shutdown_servers(state: &Arc<AppState>) {
components(
schemas(
HealthResponse,
DesktopState,
DesktopResolution,
DesktopErrorInfo,
DesktopProcessInfo,
DesktopStatusResponse,
DesktopStartRequest,
DesktopScreenshotQuery,
DesktopRegionScreenshotQuery,
DesktopMousePositionResponse,
DesktopMouseButton,
DesktopMouseMoveRequest,
DesktopMouseClickRequest,
DesktopMouseDragRequest,
DesktopMouseScrollRequest,
DesktopKeyboardTypeRequest,
DesktopKeyboardPressRequest,
DesktopActionResponse,
DesktopDisplayInfoResponse,
ServerStatus,
ServerStatusInfo,
AgentCapabilities,
@ -438,6 +496,12 @@ impl From<ProblemDetails> for ApiError {
}
}
impl From<DesktopProblem> for ApiError {
fn from(value: DesktopProblem) -> Self {
Self::Problem(value.to_problem_details())
}
}
impl IntoResponse for ApiError {
fn into_response(self) -> Response {
let problem = match &self {
@ -476,6 +540,305 @@ async fn get_v1_health() -> Json<HealthResponse> {
})
}
/// Get desktop runtime status.
///
/// Returns the current desktop runtime state, dependency status, active
/// display metadata, and supervised process information.
#[utoipa::path(
get,
path = "/v1/desktop/status",
tag = "v1",
responses(
(status = 200, description = "Desktop runtime status", body = DesktopStatusResponse),
(status = 401, description = "Authentication required", body = ProblemDetails)
)
)]
async fn get_v1_desktop_status(
State(state): State<Arc<AppState>>,
) -> Result<Json<DesktopStatusResponse>, ApiError> {
Ok(Json(state.desktop_runtime().status().await))
}
/// Start the private desktop runtime.
///
/// Lazily launches the managed Xvfb/openbox stack, validates display health,
/// and returns the resulting desktop status snapshot.
#[utoipa::path(
post,
path = "/v1/desktop/start",
tag = "v1",
request_body = DesktopStartRequest,
responses(
(status = 200, description = "Desktop runtime status after start", body = DesktopStatusResponse),
(status = 400, description = "Invalid desktop start request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is already transitioning", body = ProblemDetails),
(status = 501, description = "Desktop API unsupported on this platform", body = ProblemDetails),
(status = 503, description = "Desktop runtime could not be started", body = ProblemDetails)
)
)]
async fn post_v1_desktop_start(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopStartRequest>,
) -> Result<Json<DesktopStatusResponse>, ApiError> {
let status = state.desktop_runtime().start(body).await?;
Ok(Json(status))
}
/// Stop the private desktop runtime.
///
/// Terminates the managed openbox/Xvfb/dbus processes owned by the desktop
/// runtime and returns the resulting status snapshot.
#[utoipa::path(
post,
path = "/v1/desktop/stop",
tag = "v1",
responses(
(status = 200, description = "Desktop runtime status after stop", body = DesktopStatusResponse),
(status = 409, description = "Desktop runtime is already transitioning", body = ProblemDetails)
)
)]
async fn post_v1_desktop_stop(
State(state): State<Arc<AppState>>,
) -> Result<Json<DesktopStatusResponse>, ApiError> {
let status = state.desktop_runtime().stop().await?;
Ok(Json(status))
}
/// Capture a full desktop screenshot.
///
/// Performs a health-gated full-frame screenshot of the managed desktop and
/// returns PNG bytes.
#[utoipa::path(
get,
path = "/v1/desktop/screenshot",
tag = "v1",
responses(
(status = 200, description = "Desktop screenshot as PNG bytes"),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or screenshot capture failed", body = ProblemDetails)
)
)]
async fn get_v1_desktop_screenshot(
State(state): State<Arc<AppState>>,
) -> Result<Response, ApiError> {
let bytes = state.desktop_runtime().screenshot().await?;
Ok(([(header::CONTENT_TYPE, "image/png")], Bytes::from(bytes)).into_response())
}
/// Capture a desktop screenshot region.
///
/// Performs a health-gated screenshot crop against the managed desktop and
/// returns the requested PNG region bytes.
#[utoipa::path(
get,
path = "/v1/desktop/screenshot/region",
tag = "v1",
params(
("x" = i32, Query, description = "Region x coordinate"),
("y" = i32, Query, description = "Region y coordinate"),
("width" = u32, Query, description = "Region width"),
("height" = u32, Query, description = "Region height")
),
responses(
(status = 200, description = "Desktop screenshot region as PNG bytes"),
(status = 400, description = "Invalid screenshot region", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or screenshot capture failed", body = ProblemDetails)
)
)]
async fn get_v1_desktop_screenshot_region(
State(state): State<Arc<AppState>>,
Query(query): Query<DesktopRegionScreenshotQuery>,
) -> Result<Response, ApiError> {
let bytes = state.desktop_runtime().screenshot_region(query).await?;
Ok(([(header::CONTENT_TYPE, "image/png")], Bytes::from(bytes)).into_response())
}
/// Get the current desktop mouse position.
///
/// Performs a health-gated mouse position query against the managed desktop.
#[utoipa::path(
get,
path = "/v1/desktop/mouse/position",
tag = "v1",
responses(
(status = 200, description = "Desktop mouse position", body = DesktopMousePositionResponse),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input check failed", body = ProblemDetails)
)
)]
async fn get_v1_desktop_mouse_position(
State(state): State<Arc<AppState>>,
) -> Result<Json<DesktopMousePositionResponse>, ApiError> {
let position = state.desktop_runtime().mouse_position().await?;
Ok(Json(position))
}
/// Move the desktop mouse.
///
/// Performs a health-gated absolute pointer move on the managed desktop and
/// returns the resulting mouse position.
#[utoipa::path(
post,
path = "/v1/desktop/mouse/move",
tag = "v1",
request_body = DesktopMouseMoveRequest,
responses(
(status = 200, description = "Desktop mouse position after move", body = DesktopMousePositionResponse),
(status = 400, description = "Invalid mouse move request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input failed", body = ProblemDetails)
)
)]
async fn post_v1_desktop_mouse_move(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopMouseMoveRequest>,
) -> Result<Json<DesktopMousePositionResponse>, ApiError> {
let position = state.desktop_runtime().move_mouse(body).await?;
Ok(Json(position))
}
/// Click on the desktop.
///
/// Performs a health-gated pointer move and click against the managed desktop
/// and returns the resulting mouse position.
#[utoipa::path(
post,
path = "/v1/desktop/mouse/click",
tag = "v1",
request_body = DesktopMouseClickRequest,
responses(
(status = 200, description = "Desktop mouse position after click", body = DesktopMousePositionResponse),
(status = 400, description = "Invalid mouse click request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input failed", body = ProblemDetails)
)
)]
async fn post_v1_desktop_mouse_click(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopMouseClickRequest>,
) -> Result<Json<DesktopMousePositionResponse>, ApiError> {
let position = state.desktop_runtime().click_mouse(body).await?;
Ok(Json(position))
}
/// Drag the desktop mouse.
///
/// Performs a health-gated drag gesture against the managed desktop and
/// returns the resulting mouse position.
#[utoipa::path(
post,
path = "/v1/desktop/mouse/drag",
tag = "v1",
request_body = DesktopMouseDragRequest,
responses(
(status = 200, description = "Desktop mouse position after drag", body = DesktopMousePositionResponse),
(status = 400, description = "Invalid mouse drag request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input failed", body = ProblemDetails)
)
)]
async fn post_v1_desktop_mouse_drag(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopMouseDragRequest>,
) -> Result<Json<DesktopMousePositionResponse>, ApiError> {
let position = state.desktop_runtime().drag_mouse(body).await?;
Ok(Json(position))
}
/// Scroll the desktop mouse wheel.
///
/// Performs a health-gated scroll gesture at the requested coordinates and
/// returns the resulting mouse position.
#[utoipa::path(
post,
path = "/v1/desktop/mouse/scroll",
tag = "v1",
request_body = DesktopMouseScrollRequest,
responses(
(status = 200, description = "Desktop mouse position after scroll", body = DesktopMousePositionResponse),
(status = 400, description = "Invalid mouse scroll request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input failed", body = ProblemDetails)
)
)]
async fn post_v1_desktop_mouse_scroll(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopMouseScrollRequest>,
) -> Result<Json<DesktopMousePositionResponse>, ApiError> {
let position = state.desktop_runtime().scroll_mouse(body).await?;
Ok(Json(position))
}
/// Type desktop keyboard text.
///
/// Performs a health-gated `xdotool type` operation against the managed
/// desktop.
#[utoipa::path(
post,
path = "/v1/desktop/keyboard/type",
tag = "v1",
request_body = DesktopKeyboardTypeRequest,
responses(
(status = 200, description = "Desktop keyboard action result", body = DesktopActionResponse),
(status = 400, description = "Invalid keyboard type request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input failed", body = ProblemDetails)
)
)]
async fn post_v1_desktop_keyboard_type(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopKeyboardTypeRequest>,
) -> Result<Json<DesktopActionResponse>, ApiError> {
let response = state.desktop_runtime().type_text(body).await?;
Ok(Json(response))
}
/// Press a desktop keyboard shortcut.
///
/// Performs a health-gated `xdotool key` operation against the managed
/// desktop.
#[utoipa::path(
post,
path = "/v1/desktop/keyboard/press",
tag = "v1",
request_body = DesktopKeyboardPressRequest,
responses(
(status = 200, description = "Desktop keyboard action result", body = DesktopActionResponse),
(status = 400, description = "Invalid keyboard press request", body = ProblemDetails),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or input failed", body = ProblemDetails)
)
)]
async fn post_v1_desktop_keyboard_press(
State(state): State<Arc<AppState>>,
Json(body): Json<DesktopKeyboardPressRequest>,
) -> Result<Json<DesktopActionResponse>, ApiError> {
let response = state.desktop_runtime().press_key(body).await?;
Ok(Json(response))
}
/// Get desktop display information.
///
/// Performs a health-gated display query against the managed desktop and
/// returns the current display identifier and resolution.
#[utoipa::path(
get,
path = "/v1/desktop/display/info",
tag = "v1",
responses(
(status = 200, description = "Desktop display information", body = DesktopDisplayInfoResponse),
(status = 409, description = "Desktop runtime is not ready", body = ProblemDetails),
(status = 503, description = "Desktop runtime health or display query failed", body = ProblemDetails)
)
)]
async fn get_v1_desktop_display_info(
State(state): State<Arc<AppState>>,
) -> Result<Json<DesktopDisplayInfoResponse>, ApiError> {
let info = state.desktop_runtime().display_info().await?;
Ok(Json(info))
}
#[utoipa::path(
get,
path = "/v1/agents",

158
server/packages/sandbox-agent/tests/v1_api.rs
View file

@ -50,6 +50,15 @@ struct EnvVarGuard {
previous: Option<std::ffi::OsString>,
}
struct FakeDesktopEnv {
_temp: TempDir,
_path: EnvVarGuard,
_xdg_state_home: EnvVarGuard,
_assume_linux: EnvVarGuard,
_display_num: EnvVarGuard,
_fake_state_dir: EnvVarGuard,
}
struct LiveServer {
address: SocketAddr,
shutdown_tx: Option<oneshot::Sender<()>>,
@ -167,6 +176,153 @@ exit 0
);
}
fn setup_fake_desktop_env() -> FakeDesktopEnv {
let temp = tempfile::tempdir().expect("create fake desktop tempdir");
let bin_dir = temp.path().join("bin");
let xdg_state_home = temp.path().join("state");
let fake_state_dir = temp.path().join("desktop-state");
fs::create_dir_all(&bin_dir).expect("create fake desktop bin dir");
fs::create_dir_all(&xdg_state_home).expect("create xdg state home");
fs::create_dir_all(&fake_state_dir).expect("create fake state dir");
write_executable(
&bin_dir.join("Xvfb"),
r#"#!/usr/bin/env sh
set -eu
display="${1:-:99}"
number="${display#:}"
socket="/tmp/.X11-unix/X${number}"
mkdir -p /tmp/.X11-unix
touch "$socket"
cleanup() {
rm -f "$socket"
exit 0
}
trap cleanup INT TERM EXIT
while :; do
sleep 1
done
"#,
);
write_executable(
&bin_dir.join("openbox"),
r#"#!/usr/bin/env sh
set -eu
trap 'exit 0' INT TERM
while :; do
sleep 1
done
"#,
);
write_executable(
&bin_dir.join("dbus-launch"),
r#"#!/usr/bin/env sh
set -eu
echo "DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/sandbox-agent-test-bus"
echo "DBUS_SESSION_BUS_PID=$$"
"#,
);
write_executable(
&bin_dir.join("xrandr"),
r#"#!/usr/bin/env sh
set -eu
cat <<'EOF'
Screen 0: minimum 1 x 1, current 1440 x 900, maximum 32767 x 32767
EOF
"#,
);
write_executable(
&bin_dir.join("import"),
r#"#!/usr/bin/env sh
set -eu
printf '\211PNG\r\n\032\n\000\000\000\rIHDR\000\000\000\001\000\000\000\001\010\006\000\000\000\037\025\304\211\000\000\000\013IDATx\234c\000\001\000\000\005\000\001\r\n-\264\000\000\000\000IEND\256B`\202'
"#,
);
write_executable(
&bin_dir.join("xdotool"),
r#"#!/usr/bin/env sh
set -eu
state_dir="${SANDBOX_AGENT_DESKTOP_FAKE_STATE_DIR:?missing fake state dir}"
state_file="${state_dir}/mouse"
mkdir -p "$state_dir"
if [ ! -f "$state_file" ]; then
printf '0 0\n' > "$state_file"
fi
read_state() {
read -r x y < "$state_file"
}
write_state() {
printf '%s %s\n' "$1" "$2" > "$state_file"
}
command="${1:-}"
case "$command" in
getmouselocation)
read_state
printf 'X=%s\nY=%s\nSCREEN=0\nWINDOW=0\n' "$x" "$y"
;;
mousemove)
shift
x="${1:-0}"
y="${2:-0}"
shift 2 || true
while [ "$#" -gt 0 ]; do
token="$1"
shift
case "$token" in
mousemove)
x="${1:-0}"
y="${2:-0}"
shift 2 || true
;;
mousedown|mouseup)
shift 1 || true
;;
click)
if [ "${1:-}" = "--repeat" ]; then
shift 2 || true
fi
shift 1 || true
;;
esac
done
write_state "$x" "$y"
;;
type|key)
exit 0
;;
*)
exit 0
;;
esac
"#,
);
let original_path = std::env::var_os("PATH").unwrap_or_default();
let mut paths = vec![bin_dir];
paths.extend(std::env::split_paths(&original_path));
let merged_path = std::env::join_paths(paths).expect("join PATH");
FakeDesktopEnv {
_temp: temp,
_path: EnvVarGuard::set_os("PATH", merged_path.as_os_str()),
_xdg_state_home: EnvVarGuard::set_os("XDG_STATE_HOME", xdg_state_home.as_os_str()),
_assume_linux: EnvVarGuard::set("SANDBOX_AGENT_DESKTOP_TEST_ASSUME_LINUX", "1"),
_display_num: EnvVarGuard::set("SANDBOX_AGENT_DESKTOP_DISPLAY_NUM", "190"),
_fake_state_dir: EnvVarGuard::set_os(
"SANDBOX_AGENT_DESKTOP_FAKE_STATE_DIR",
fake_state_dir.as_os_str(),
),
}
}
fn serve_registry_once(document: Value) -> String {
let listener = TcpListener::bind("127.0.0.1:0").expect("bind registry server");
let address = listener.local_addr().expect("registry address");
@ -375,5 +531,7 @@ mod acp_transport;
mod config_endpoints;
#[path = "v1_api/control_plane.rs"]
mod control_plane;
#[path = "v1_api/desktop.rs"]
mod desktop;
#[path = "v1_api/processes.rs"]
mod processes;

222
server/packages/sandbox-agent/tests/v1_api/desktop.rs Normal file
View file

@ -0,0 +1,222 @@
use super::*;
use serial_test::serial;
#[tokio::test]
#[serial]
async fn v1_desktop_status_reports_install_required_when_dependencies_are_missing() {
let temp = tempfile::tempdir().expect("create empty path tempdir");
let _path = EnvVarGuard::set_os("PATH", temp.path().as_os_str());
let _assume_linux = EnvVarGuard::set("SANDBOX_AGENT_DESKTOP_TEST_ASSUME_LINUX", "1");
let test_app = TestApp::new(AuthConfig::disabled());
let (status, _, body) = send_request(
&test_app.app,
Method::GET,
"/v1/desktop/status",
None,
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let parsed = parse_json(&body);
assert_eq!(parsed["state"], "install_required");
assert!(parsed["missingDependencies"]
.as_array()
.expect("missingDependencies array")
.iter()
.any(|value| value == "Xvfb"));
assert_eq!(
parsed["installCommand"],
"sandbox-agent install desktop --yes"
);
}
#[tokio::test]
#[serial]
async fn v1_desktop_lifecycle_and_actions_work_with_fake_runtime() {
let _fake = setup_fake_desktop_env();
let test_app = TestApp::new(AuthConfig::disabled());
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/start",
Some(json!({
"width": 1440,
"height": 900,
"dpi": 96
})),
&[],
)
.await;
assert_eq!(
status,
StatusCode::OK,
"unexpected start response: {}",
String::from_utf8_lossy(&body)
);
let parsed = parse_json(&body);
assert_eq!(parsed["state"], "active");
let display = parsed["display"].as_str().expect("desktop display").to_string();
assert!(display.starts_with(':'));
assert_eq!(parsed["resolution"]["width"], 1440);
assert_eq!(parsed["resolution"]["height"], 900);
let (status, headers, body) = send_request_raw(
&test_app.app,
Method::GET,
"/v1/desktop/screenshot",
None,
&[],
None,
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(
headers
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok()),
Some("image/png")
);
assert!(body.starts_with(b"\x89PNG\r\n\x1a\n"));
let (status, _, body) = send_request_raw(
&test_app.app,
Method::GET,
"/v1/desktop/screenshot/region?x=10&y=20&width=30&height=40",
None,
&[],
None,
)
.await;
assert_eq!(status, StatusCode::OK);
assert!(body.starts_with(b"\x89PNG\r\n\x1a\n"));
let (status, _, body) = send_request(
&test_app.app,
Method::GET,
"/v1/desktop/display/info",
None,
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let display_info = parse_json(&body);
assert_eq!(display_info["display"], display);
assert_eq!(display_info["resolution"]["width"], 1440);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/mouse/move",
Some(json!({ "x": 400, "y": 300 })),
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let mouse = parse_json(&body);
assert_eq!(mouse["x"], 400);
assert_eq!(mouse["y"], 300);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/mouse/drag",
Some(json!({
"startX": 100,
"startY": 110,
"endX": 220,
"endY": 230,
"button": "left"
})),
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let dragged = parse_json(&body);
assert_eq!(dragged["x"], 220);
assert_eq!(dragged["y"], 230);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/mouse/click",
Some(json!({
"x": 220,
"y": 230,
"button": "left",
"clickCount": 1
})),
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let clicked = parse_json(&body);
assert_eq!(clicked["x"], 220);
assert_eq!(clicked["y"], 230);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/mouse/scroll",
Some(json!({
"x": 220,
"y": 230,
"deltaY": -3
})),
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let scrolled = parse_json(&body);
assert_eq!(scrolled["x"], 220);
assert_eq!(scrolled["y"], 230);
let (status, _, body) = send_request(
&test_app.app,
Method::GET,
"/v1/desktop/mouse/position",
None,
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
let position = parse_json(&body);
assert_eq!(position["x"], 220);
assert_eq!(position["y"], 230);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/keyboard/type",
Some(json!({ "text": "hello world", "delayMs": 5 })),
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(parse_json(&body)["ok"], true);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/keyboard/press",
Some(json!({ "key": "ctrl+l" })),
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(parse_json(&body)["ok"], true);
let (status, _, body) = send_request(
&test_app.app,
Method::POST,
"/v1/desktop/stop",
None,
&[],
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(parse_json(&body)["state"], "inactive");
}