Poolside wants to ship an AI software engineer, not just autocomplete. In a new Latent Space interview, the team outlines a playbook for execution-first coding agents—and what it takes to make them useful in the real world.
Why it matters: the center of gravity is shifting from code suggestions to agents that read, modify, run, and validate code across real repos. The winners will close the loop—plan, edit, execute, and verify—safely and repeatably.
The Poolside playbook in 5 moves
- Close the loop: Treat coding as a plan-edit-run-test cycle, not a chat. Give the agent tools to execute code, run tests, and check results automatically.
- Ground in real repo context: Index codebases (symbols, tests, dependency graphs) so the agent navigates like a dev—file-aware, test-aware, and build-aware.
- Build ruthless evals: Use issue-style tasks with deterministic harnesses (fixtures, timeouts, flaky-test filters) to measure end-to-end task completion, not token-level accuracy.
- Data flywheel from execution: Log every attempt, diff, test result, and traceback. Turn those traces into training/eval data to steadily harden the agent.
- Safety and controls first: Sandboxes, package allowlists, secrets isolation, and human-in-the-loop reviews keep agents productive without introducing supply-chain risk.
What you can implement now
- Spin up ephemeral, sandboxed dev environments (e.g., containers with restricted network and mounted test data only).
- Expose essential tools: run tests, build, search code, edit files, create branches/PRs, and revert safely.
- Start with scoped tasks: bug fixes with failing tests, small refactors, docstring generation validated by linters.
- Instrument everything: capture diffs, logs, exit codes, and wall-clock time; store traces for replay and regression.
- Add review guardrails: require passing tests and a lightweight human sign-off before merges.
Metrics that matter
- Task success rate (end-to-end pass in harness) and time-to-success.
- Revert rate after merge and escaped defects.
- Coverage of test harness over changed lines and files.
- Failure modes taxonomy (environment, dependency, compile, logic, flaky test) to target fixes.
Risks and safeguards
- Dependency and supply-chain risk: enforce pinned versions, allowlists, SBOMs, and signature checks.
- Secrets exposure: isolate credentials; never grant write access to production data or CI tokens by default.
- License contamination: restrict third-party snippets and verify licenses on generated diffs.
- Cost and drift: cap execution budgets; run daily regressions to catch breaking changes in tools or models.
Starter stack ideas
- Models: pair a strong general LLM with a code-specialized model; prefer function/tool-use reliability over raw tokens-per-second.
- Sandbox: containerized runners with timeouts, cgroups, and network egress policies; snapshot test fixtures for determinism.
- Context: repo index (symbols, tests, call graphs) and retrieval tuned for tasks (e.g., failing-test centric retrieval).
- Evals: adopt issue-style tasks and a deterministic harness. For inspiration, see SWE-bench to design end-to-end measurements.
- Workflow: draft PRs with agent-produced diffs, run CI, and require human review before merge.
Sources
Primary: Latent Space — Poolside. Benchmark design reference: SWE-bench.
Takeaway
Execution-first beats chat-first. If you want useful coding agents, invest in harnesses, sandboxes, and repo context—then let data from real executions harden the system.
Like this? Get one practical AI nugget in your inbox each week. Subscribe to our newsletter.
