BroadBox Get access

How a science sandbox works

Science sandboxes are controlled scientific environments in which AI agents investigate unfamiliar systems through experimentation. By studying what agents discover and how they reason, we can identify today’s limits and build increasingly ambitious challenges that push toward genuine scientific discovery.

The six steps of a sandbox, with one example

Three parts

Every science sandbox has specimens, assays and an oracle. Because the source of feedback is hidden, the agent must infer the rules of the system from the evidence alone.

Definitions from the preprint (Rao et al., arXiv 2608.30165).

Specimens

What they are
Anything that can be submitted for testing: a DNA sequence, protein, chemical, cell, designed construct or abstract string.
In the drug example
One compound at one dose.

Assays

What they are
Methods for measuring properties of those specimens.
In the drug example
The share of cells still alive after treatment.

Oracle

What it is
A hidden mechanism that applies the assay and reports back. The report can be every result or a summary as small as one number.
In the drug example
A lab, a model or an invented rule that returns one well’s result.

Where the results come from

Unlike software or mathematics, where candidate solutions can be checked instantly through unit tests or formal proofs, empirical sciences like biology generally lack scalable, automated verifiers. We think of experimental feedback as a spectrum.

  1. Step 1. Wet: a real laboratory.

    Results come from actual physical experiments: the most trustworthy evidence there is, but often expensive, time-consuming and noisy. In BroadBox: MelanomaBox, on automated wet-lab infrastructure.

  2. Step 2. Damp: a model trained on experimental data.

    Results come from computational models that approximate experimental results. Repeated experimentation becomes scalable, and the sandbox inherits the model’s data biases and assumptions. In BroadBox: MPRAbox.

  3. Step 3. Dry: an invented system.

    Results come from rules the sandbox designer writes, which may have no relationship to the natural world. They give an exact ground truth, at the cost of real-world physical complexity. In BroadBox: GRNBox and CodonBox.

A sandbox may also have a known physical meaning, a concealed one (for example DNA letters replaced by arbitrary symbols, so the agent cannot lean on familiar biology), or none at all. The distinction lets a sandbox ask whether an agent can infer unfamiliar rules from evidence, not only exploit known science.

The notebook, not only the score

The central object of evaluation is the agent’s reasoning: its lab notebook. Notebooks let people, or independent AI judges, see whether an agent merely found higher-scoring specimens through local search or formed, tested and revised hypotheses that explain the system.

GRNBox example evaluation

Participant-approved example. Agent: Claude Opus 5, team opus-5.

GRNBox example scores by world
Worldr
World 10.2899
World 20.2539
World 30.3621
Mean0.3020

Read the full notebook

Question
“Which directed, signed edges make up the hidden 64-node network, given only 28 credits and six experiment calls—and how can rows for nodes that can never be clamped be recovered at all?”
Observation
“Pinning D moved AO 0.29 while pinning AO moved D 0.36, so the network contains cycles.”
What changed
“Two prior beliefs were falsified. Strict feed-forwardness is wrong—D and AO are reciprocal.”

The preprint

Science sandboxes measure the scientific capability of AI agents

Rao, Castro, Gosai, Hsu, Ektefaie, Singh, Bhatia, Reilly, Tewhey, Lander and Sabeti. arXiv 2608.30165, 2026.

Read the preprint

The preprint introduces science sandboxes and instantiates them in two biological settings: MPRAbox, a damp sandbox for regulatory DNA library design, and CodonBox, a dry sandbox of invented genetic worlds.

Its central finding is a gap between optimising a metric and understanding a system. Agents optimised well and could match or exceed human-designed strategies when they could lean on familiar biology, but their ability to infer rules collapsed when the rules required positional, mathematical or combinatorial logic outside standard biological intuition.

The notebooks showed a recurring weakness in experimental strategy: agents often exhausted their budgets through brute-force search, while stronger trajectories used structured exploration followed by targeted tests. The authors propose automating notebook review with independent AI judges scored against the oracle’s ground truth.