How a science sandbox works
Science sandboxes are controlled scientific environments in which AI agents investigate unfamiliar systems through experimentation. By studying what agents discover and how they reason, we can identify today’s limits and build increasingly ambitious challenges that push toward genuine scientific discovery.
Three parts
Every science sandbox has specimens, assays and an oracle. Because the source of feedback is hidden, the agent must infer the rules of the system from the evidence alone.
Definitions from the preprint (Rao et al., arXiv 2608.30165).
Specimens
- What they are
- Anything that can be submitted for testing: a DNA sequence, protein, chemical, cell, designed construct or abstract string.
- In the drug example
- One compound at one dose.
Assays
- What they are
- Methods for measuring properties of those specimens.
- In the drug example
- The share of cells still alive after treatment.
Oracle
- What it is
- A hidden mechanism that applies the assay and reports back. The report can be every result or a summary as small as one number.
- In the drug example
- A lab, a model or an invented rule that returns one well’s result.
Three small white trays. Wet: a plate of cells and a cluster of cells. Damp: a model drawn as blue nodes on a stand beside a small plate. Dry: blue and white blocks placed by a rule on a grid.
Where the results come from
Unlike software or mathematics, where candidate solutions can be checked instantly through unit tests or formal proofs, empirical sciences like biology generally lack scalable, automated verifiers. We think of experimental feedback as a spectrum.
Step 1. Wet: a real laboratory.
Results come from actual physical experiments: the most trustworthy evidence there is, but often expensive, time-consuming and noisy. In BroadBox: MelanomaBox, on automated wet-lab infrastructure.
Step 2. Damp: a model trained on experimental data.
Results come from computational models that approximate experimental results. Repeated experimentation becomes scalable, and the sandbox inherits the model’s data biases and assumptions. In BroadBox: MPRAbox.
A sandbox may also have a known physical meaning, a concealed one (for example DNA letters replaced by arbitrary symbols, so the agent cannot lean on familiar biology), or none at all. The distinction lets a sandbox ask whether an agent can infer unfamiliar rules from evidence, not only exploit known science.
The notebook, not only the score
The central object of evaluation is the agent’s reasoning: its lab notebook. Notebooks let people, or independent AI judges, see whether an agent merely found higher-scoring specimens through local search or formed, tested and revised hypotheses that explain the system.
GRNBox example evaluation
Participant-approved example. Agent: Claude Opus 5, team opus-5.
| World | r |
|---|---|
| World 1 | 0.2899 |
| World 2 | 0.2539 |
| World 3 | 0.3621 |
| Mean | 0.3020 |
- Question
- “Which directed, signed edges make up the hidden 64-node network, given only 28 credits and six experiment calls—and how can rows for nodes that can never be clamped be recovered at all?”
- Observation
- “Pinning D moved AO 0.29 while pinning AO moved D 0.36, so the network contains cycles.”
- What changed
- “Two prior beliefs were falsified. Strict feed-forwardness is wrong—D and AO are reciprocal.”
The preprint
Science sandboxes measure the scientific capability of AI agents
Rao, Castro, Gosai, Hsu, Ektefaie, Singh, Bhatia, Reilly, Tewhey, Lander and Sabeti. arXiv 2608.30165, 2026.
The preprint introduces science sandboxes and instantiates them in two biological settings: MPRAbox, a damp sandbox for regulatory DNA library design, and CodonBox, a dry sandbox of invented genetic worlds.
Its central finding is a gap between optimising a metric and understanding a system. Agents optimised well and could match or exceed human-designed strategies when they could lean on familiar biology, but their ability to infer rules collapsed when the rules required positional, mathematical or combinatorial logic outside standard biological intuition.
The notebooks showed a recurring weakness in experimental strategy: agents often exhausted their budgets through brute-force search, while stronger trajectories used structured exploration followed by targeted tests. The authors propose automating notebook review with independent AI judges scored against the oracle’s ground truth.