feat: replace neko with native GStreamer WebRTC for desktop streaming

Replace the neko binary dependency with a native GStreamer pipeline
(ximagesrc -> vp8enc -> webrtcbin) for desktop video streaming. This
removes the external neko process and integrates screen capture directly
via gstreamer-rs crate bindings behind a `desktop-gstreamer` feature flag.

Key changes:
- Add desktop_gstreamer.rs with GStreamer WebRTC pipeline management
- Rewrite signaling protocol (ready/offer/answer/candidate over WS)
- Add leaky queues and videorate for low-latency streaming
- Rewrite ICE candidates to 127.0.0.1 for Docker connectivity
- Constrain UDP port range (30000-30100) via libnice agent
- Update TypeScript SDK desktop-stream.ts for new signaling
- Update inspector DesktopTab with WebRTC Live View
- Update Dockerfiles to install GStreamer dev packages

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nathan Flurry 2026-03-16 17:54:39 -07:00
parent e638148345
commit 944ad1ba55
22 changed files with 1848 additions and 1170 deletions

40
server/compose.dev.yaml Normal file
View file

@ -0,0 +1,40 @@
name: sandbox-agent-dev
services:
backend:
build:
context: ..
dockerfile: docker/test-agent/Dockerfile
image: sandbox-agent-dev
command: ["server", "--host", "0.0.0.0", "--port", "3000", "--no-token"]
environment:
RUST_LOG: "${RUST_LOG:-info}"
ports:
- "2468:3000"
frontend:
build:
context: ..
dockerfile: server/docker/frontend.dev.Dockerfile
working_dir: /app
depends_on:
- backend
environment:
SANDBOX_AGENT_URL: "http://backend:3000"
ports:
- "5173:5173"
volumes:
- "..:/app"
# Keep Linux-native node_modules inside the container.
- "sa_root_node_modules:/app/node_modules"
- "sa_inspector_node_modules:/app/frontend/packages/inspector/node_modules"
- "sa_react_node_modules:/app/sdks/react/node_modules"
- "sa_typescript_node_modules:/app/sdks/typescript/node_modules"
- "sa_pnpm_store:/root/.local/share/pnpm/store"
volumes:
sa_root_node_modules: {}
sa_inspector_node_modules: {}
sa_react_node_modules: {}
sa_typescript_node_modules: {}
sa_pnpm_store: {}