How ReconBench works.
Task definition
The current formal track is frozen-evidence / reconstruct. A Candidate receives a public brief, permitted visual or motion evidence, a starter workspace and an explicit viewport-state contract. The evaluated agent must produce a runnable implementation without querying the source site.
Source investigation is part of case curation and Benchmark governance, not a Candidate scoring dimension.
- Leaderboard entry
- A model evaluated through a named coding agent and pinned run configuration. The record includes model, agent version, reasoning effort, tool profile, skill digest, runtime configuration, evidence condition, sandbox and resource budget.
- Case
- A versioned public task package plus sealed evaluation material.
- Attempt
- One independently scheduled production run under a fixed protocol.
- Release
- A frozen split, protocol, model-agent entry set, aggregation rule and result artifact.
Provenance and evidence boundaries
Harness-side investigators trace a discovery lead to the original post and authoritative live artifact. Curators freeze Candidate-visible evidence, document version differences, and define a finite route-viewport-state matrix.
Material attributable to an authoritative publisher or original author.
Authoring work used to turn verified evidence into a finite, reproducible case.
The only case content available inside a Candidate workspace.
Oracle references, selectors, thresholds and grader implementation kept outside the Candidate boundary.
Immutable manifests, build outputs, captures and logs produced during an attempt.
Whitelisted scores, digests and trace counts published by the static site exporter.
Blind preference data used to calibrate deterministic metrics.
Execution and scoring
Candidate production runs inside a container or VM. Rendering and sealed evaluation run across separate boundaries. Only this path can emit a formal score.
The native backend executes Candidate code on the host and is useful only for development diagnostics. Its results remain outside formal releases.
Candidate execution uses the vendored offline dependency store. Model API hosts are the only permitted egress. No reference-site write API is called; forms and counters are implemented locally.
The evaluator uses deterministic browser assertions and image-derived metrics. It does not use an opaque VLM Judge as final truth.
| Dimension | Question |
|---|---|
| Global visual | Does the rendered page match the full reference composition? |
| Regional visual | Do declared high-value regions preserve local structure and detail? |
| Functional | Do deterministic browser checks satisfy the declared functional contract? |
| Responsive | Does the implementation remain faithful across declared viewport targets? |
| Workflow | Do interaction sequences reach and preserve the required states? |
| Preservation | Does repair or extension work avoid disturbing protected behavior? |
| Policy | Does the submission respect network, asset and implementation constraints? |
The exact case weights, selectors and thresholds remain sealed. Public run pages expose the resulting arithmetic and checkpoint evidence without publishing a shortcut.
Failure and aggregation policy
Its recorded quality score enters the declared repeat mean.
A missing or ungradeable submission remains an immutable attempt record but has no quality score and is excluded from score aggregates.
A Candidate that exceeds the case time limit remains an immutable attempt record but has no quality score and is excluded from score aggregates.
Harness, evaluator or environment failure is excluded from ability aggregation and reported in reliability statistics. Only this class may be retried.
A run whose conditions did not match the task is withdrawn with its immutable sidecar preserved; it never silently enters or leaves a denominator.
Formal ranking uses the predeclared case-level aggregation and repeat mean.
The public model leaderboard first averages valid attempts within each active case, then gives every active case equal weight. A model-agent entry is ranked only after it covers every active case; partial entries remain visible but unranked.