harivansh-afk/sandbox-agent
1
0
Fork 0
mirror of https://github.com/harivansh-afk/sandbox-agent.git synced 2026-04-15 05:02:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: desktop computer-use APIs with neko-based streaming

Browse source 
Add desktop runtime management (Xvfb, openbox, dbus), screen capture,
mouse/keyboard input, and video streaming via neko binary extracted
from the m1k1o/neko container. Includes Docker test rig, TypeScript SDK
desktop support, and inspector Desktop tab.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nathan Flurry 2026-03-16 17:56:39 -07:00
parent 3895e34bdb
commit 33821d8660
66 changed files with 13190 additions and 1135 deletions
Download patch file Download diff file Expand all files Collapse all files

56
CLAUDE.md
View file

@ -22,19 +22,6 @@
- `server/packages/sandbox-agent/src/cli.rs`
- Keep docs aligned to implemented endpoints/commands only (for example ACP under `/v1/acp`, not legacy `/v1/sessions` APIs).
## E2E Agent Testing
- When asked to test agents e2e and you do not have the API tokens/credentials required, always stop and ask the user where to find the tokens before proceeding.
## ACP Adapter Audit
- `scripts/audit-acp-deps/adapters.json` is the single source of truth for ACP adapter npm packages, pinned versions, and the `@agentclientprotocol/sdk` pin.
- The Rust fallback install path in `server/packages/agent-management/src/agents.rs` reads adapter entries from `adapters.json` at compile time via `include_str!`.
- Run `cd scripts/audit-acp-deps && npx tsx audit.ts` to compare our pinned versions against the ACP registry and npm latest.
- When bumping an adapter version, update `adapters.json` only — the Rust code picks it up automatically.
- When adding a new agent, add an entry to `adapters.json` (the `_` fallback arm in `install_agent_process_fallback` handles it).
- When updating the `@agentclientprotocol/sdk` pin, update both `adapters.json` (sdkDeps) and `sdks/acp-http-client/package.json`.
## Change Tracking
- If the user asks to "push" changes, treat that as permission to commit and push all current workspace changes, not a hand-picked subset, unless the user explicitly scopes the push.
@ -43,41 +30,13 @@
- Regenerate `docs/openapi.json` when HTTP contracts change.
- Keep `docs/inspector.mdx` and `docs/sdks/typescript.mdx` aligned with implementation.
- Append blockers/decisions to `research/acp/friction.md` during ACP work.
- Each agent has its own doc page at `docs/agents/<name>.mdx` listing models, modes, and thought levels. Update the relevant page when changing `fallback_config_options`. To regenerate capability data, run `cd scripts/agent-configs && npx tsx dump.ts`. Source data: `scripts/agent-configs/resources/*.json` and hardcoded entries in `server/packages/sandbox-agent/src/router/support.rs` (`fallback_config_options`).
- `docs/agent-capabilities.mdx` lists models/modes/thought levels per agent. Update it when adding a new agent or changing `fallback_config_options`. If its "Last updated" date is >2 weeks old, re-run `cd scripts/agent-configs && npx tsx dump.ts` and update the doc to match. Source data: `scripts/agent-configs/resources/*.json` and hardcoded entries in `server/packages/sandbox-agent/src/router/support.rs` (`fallback_config_options`).
- Some agent models are gated by subscription (e.g. Claude `opus`). The live report only shows models available to the current credentials. The static doc and JSON resource files should list all known models regardless of subscription tier.
## Adding Providers
## Docker Test Image
When adding a new sandbox provider, update all of the following:
- `sdks/typescript/src/providers/<name>.ts` — provider implementation
- `sdks/typescript/package.json` — add `./<name>` export, peerDependencies, peerDependenciesMeta, devDependencies
- `sdks/typescript/tsup.config.ts` — add entry point and external
- `sdks/typescript/tests/providers.test.ts` — add test entry
- `examples/<name>/` — create example with `src/index.ts` and `tests/<name>.test.ts`
- `docs/deploy/<name>.mdx` — create deploy guide
- `docs/docs.json` — add to Deploy pages navigation
- `docs/quickstart.mdx` — add tab in "Start the sandbox" step, add credentials entry in "Passing LLM credentials" accordion
## Adding Agents
When adding a new agent, update all of the following:
- `docs/agents/<name>.mdx` — create agent page with usage snippet and capabilities table
- `docs/docs.json` — add to the Agents group under Agent
- `docs/quickstart.mdx` — add tab in the "Create a session and send a prompt" CodeGroup
## Persist Packages (Deprecated)
- The `@sandbox-agent/persist-*` npm packages (`persist-sqlite`, `persist-postgres`, `persist-indexeddb`, `persist-rivet`) are deprecated stubs. They still publish to npm but throw a deprecation error at import time.
- Driver implementations now live inline in examples and consuming packages:
- SQLite: `examples/persist-sqlite/src/persist.ts`
- Postgres: `examples/persist-postgres/src/persist.ts`
- IndexedDB: `frontend/packages/inspector/src/persist-indexeddb.ts`
- Rivet: inlined in `docs/multiplayer.mdx`
- In-memory: built into the main `sandbox-agent` SDK (`InMemorySessionPersistDriver`)
- Docs (`docs/session-persistence.mdx`) link to the example implementations on GitHub instead of referencing the packages.
- Do not re-add `@sandbox-agent/persist-*` as dependencies anywhere. New persist drivers should be copied into the consuming project directly.
- Docker-backed Rust and TypeScript tests build `docker/test-agent/Dockerfile` directly in-process and cache the image tag only in memory (`OnceLock` in Rust, module-level variable in TypeScript).
- Do not add cross-process image-build scripts unless there is a concrete need for them.
## Install Version References
@ -93,27 +52,20 @@ When adding a new agent, update all of the following:
- `docs/sdk-overview.mdx`
- `docs/react-components.mdx`
- `docs/session-persistence.mdx`
- `docs/architecture.mdx`
- `docs/deploy/local.mdx`
- `docs/deploy/cloudflare.mdx`
- `docs/deploy/vercel.mdx`
- `docs/deploy/daytona.mdx`
- `docs/deploy/e2b.mdx`
- `docs/deploy/docker.mdx`
- `docs/deploy/boxlite.mdx`
- `docs/deploy/modal.mdx`
- `docs/deploy/computesdk.mdx`
- `frontend/packages/website/src/components/GetStarted.tsx`
- `.claude/commands/post-release-testing.md`
- `examples/cloudflare/Dockerfile`
- `examples/boxlite/Dockerfile`
- `examples/boxlite-python/Dockerfile`
- `examples/daytona/src/index.ts`
- `examples/shared/src/docker.ts`
- `examples/docker/src/index.ts`
- `examples/e2b/src/index.ts`
- `examples/vercel/src/index.ts`
- `sdks/typescript/src/providers/shared.ts`
- `scripts/release/main.ts`
- `scripts/release/promote-artifacts.ts`
- `scripts/release/sdk.ts`