feat: replace neko with native GStreamer WebRTC for desktop streaming

Replace the neko binary dependency with a native GStreamer pipeline
(ximagesrc -> vp8enc -> webrtcbin) for desktop video streaming. This
removes the external neko process and integrates screen capture directly
via gstreamer-rs crate bindings behind a `desktop-gstreamer` feature flag.

Key changes:
- Add desktop_gstreamer.rs with GStreamer WebRTC pipeline management
- Rewrite signaling protocol (ready/offer/answer/candidate over WS)
- Add leaky queues and videorate for low-latency streaming
- Rewrite ICE candidates to 127.0.0.1 for Docker connectivity
- Constrain UDP port range (30000-30100) via libnice agent
- Update TypeScript SDK desktop-stream.ts for new signaling
- Update inspector DesktopTab with WebRTC Live View
- Update Dockerfiles to install GStreamer dev packages

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nathan Flurry 2026-03-16 17:54:39 -07:00
parent e638148345
commit 944ad1ba55
22 changed files with 1848 additions and 1170 deletions

View file

@ -76,6 +76,26 @@ run-gigacode *ARGS:
dev-docs:
cd docs && pnpm dlx mintlify dev --host 0.0.0.0
# Start the desktop dev stack (sandbox-agent backend in Docker + inspector frontend)
[group('server')]
server-dev:
docker compose -f server/compose.dev.yaml up --build --force-recreate -d
# Stop the desktop dev stack
[group('server')]
server-dev-down:
docker compose -f server/compose.dev.yaml down
# Tail desktop dev stack logs
[group('server')]
server-dev-logs *ARGS:
docker compose -f server/compose.dev.yaml logs -f --tail=200 {{ ARGS }}
# Rebuild and restart only the backend container
[group('server')]
server-dev-restart-backend:
docker compose -f server/compose.dev.yaml up --build --force-recreate -d backend
install:
pnpm install
pnpm build --filter @sandbox-agent/inspector...