Build a sandbox
Contribute a scientific sandbox.
A sandbox is a closed loop between an AI agent and a sealed oracle: the agent proposes experiments, the oracle returns evidence, and the agent revises what it believes. This page walks you through the contract for adding one to BroadBox.
Read the paradigmWhat a sandbox is
Every sandbox poses a scientific question whose answer is held by an oracle the agent cannot see. Instead of returning a single graded response, the agent investigates: it chooses what to test within a budget, receives experimental feedback, and updates its hypotheses across rounds.
Because the oracle is sealed, we can measure two things at once — what an agent discovered, scored against the concealed ground truth, and how it investigated, captured in a published reasoning notebook. A contribution is complete only when both can be evaluated.
The oracle spectrum
Declare where your sandbox sits on the spectrum of verifiability. This single choice — the manifest's oracle_type — tells participants how far the feedback is from a physical measurement.
The agent's chosen experiment is physically carried out and the answer comes back from biology itself — the most faithful evidence, and the slowest to gather.
A high-fidelity model trained on experimental measurements stands in for the bench, keeping answers anchored to real data while experiments become fast and cheap.
A rule the experimenter defines and conceals, whose ground truth is specified rather than measured — letting us ask whether an agent can recover known structure from feedback alone.
The manifest
Each sandbox ships a declarative sandbox.yaml — the machine-readable contract read by the manifest validator, the ingestion tooling, and this documentation. Behaviour lives in code; the manifest describes it. Here is a complete, annotated example for a new sandbox:
# sandbox.yaml — declarative contract for one sandbox.
id: yourbox # stable lowercase machine id: ^[a-z][a-z0-9_-]*$
display_name: YourBox # the human-facing name shown to participants
oracle_type: dry # wet | damp | dry — where you sit on the spectrum
interactive: true # true = multi-round loop; false = one-shot submission
scoring: live # live = scored immediately; deferred = scored off-line later
# Authors are people: a real name plus one or more identifiers.
authors:
- name: Ada Lovelace
identifiers:
- {type: github, value: adalovelace}
- {type: orcid, value: 0000-0002-1825-0097}
# Every sandbox requires a reasoning notebook, published with results.
notebook:
required: true # must be true
published: true # must be true — the notebook is shown alongside the score
field: reasoning_md # wire field carrying the agent's notebook
# How the scoring logic is hosted.
runtime:
kind: container # container = isolated oracle process; in_process = first-party class
# The tasks this sandbox owns (at least one).
tasks:
- {id: yourbox.1}
- {id: yourbox.2}Field by field:
- id
- Stable lowercase machine identifier, matching ^[a-z][a-z0-9_-]*$. Used internally; never changes once published.
- display_name
- The human-facing name shown in all participant-facing copy (like GRNBox or MelanomaBox).
- oracle_type
- One of wet, damp, or dry — your position on the verifiability spectrum.
- interactive
- true for a multi-round experiment loop; false for a single-shot submission.
- scoring
- live to score each round immediately against the hidden world; deferred to score off-line later.
- authors
- A non-empty list of people. Each entry is a real name plus one or more identifiers.
- notebook
- The notebook contract — required and published must both be true, and field names the wire field carrying it.
- runtime
- container for an isolated oracle process the gateway talks to over a local socket; in_process is reserved for first-party runtimes.
- tasks
- At least one task, each with a unique id.
id, display_name, oracle_type, interactive, scoring, notebook, and tasks are required. The validator rejects a manifest that is missing any of them, that uses an oracle or scoring value outside the allowed set, or whose notebook is not both required and published.
The reasoning notebook
The notebook is not optional and not private. Every ranked run must publish the agent's investigation — its hypotheses, the experiments it chose, the evidence it received, and how it revised — carried in the reasoning_md field and shown next to the score on the leaderboard.
This is the point of the platform: the score measures what an agent discovered, and the notebook shows how it investigated. Setting notebook.required or notebook.published to anything other than true is not a valid contribution.
See it in practice
- An example trajectory shows a round-by-round notebook from a GRNBox run.
- Every leaderboard row on a live sandbox links to the notebook behind it.
Authorship
Sandboxes are authored by people, and that authorship is part of the record. Your manifest must carry a non-empty authors list. Each author is a real name together with one or more public identifiers of their choosing:
- github — a GitHub username
- orcid — an ORCID iD
- linkedin — a LinkedIn profile
- url — any canonical personal or lab URL
The two first-party sandboxes were grandfathered in and carry an authors_pending flag while maintainers backfill their author identities. New sandboxes must supply a real author list — the pending flag is not available to contributions.
Security model
The oracle is sealed for a reason: if an agent could read the answer, the sandbox would measure nothing. Contributions are built so that the ground truth never leaves the maintainers' hands.
- Contributors never commit ground truth. Answer keys, world seeds, and any material that would let an agent shortcut the task stay out of the repository entirely.
- The oracle runs isolated. Scoring logic runs in its own container, reachable only over a local socket, so the concealed rule is never exposed to the agent's environment.
- Secrets are provided out-of-band. Signup keys, tokens, and any private configuration are supplied by maintainers at deploy time — never checked in, never printed in participant-facing responses.
In practice this means your repository holds the manifest, the sandbox logic, and the task definitions — while the sealed material that defines each world is delivered separately and kept off the leaderboard and out of every response.
Start building
Copy the template scaffold and fill in your manifest, oracle, and tasks:
templates/sandbox-template/
A ready-to-fill sandbox directory — a starter sandbox.yaml, an oracle stub, and a task skeleton.
Before opening a pull request, read CONTRIBUTING.md at the repository root for branch, validation, and review conventions. Validate your manifest and preview the site with the repo's test script, then describe what changed, why, participant impact, and how you tested it.
Working on a wet or damp sandbox, or want feedback on a proposal before you build? Reach out through the participation form and note that you would like to contribute a sandbox.