reconbench

Reading this record

reconbench measures whether an agent can trace a design-discovery link back to its real source and then rebuild the responsive, interactive experience it points at. This site is a static view of recorded runs. Nothing here is recomputed in the browser, and nothing is published that a candidate could use to shortcut the task.

01

Trust

trusted

Candidate code ran inside an isolated boundary — a container or VM — with the sealed evaluator outside it. Only these results are eligible to be reported as formal scores.

dev / untrusted

The run used the development-only native backend. The agent CLI and the candidate's install and build scripts executed directly on the host machine, in the same trust boundary as the evaluator. The harness requires an explicit--allow-untrusted-native acknowledgement for these runs. Treat the numbers as useful development signal, not as a publishable benchmark result.

02

Run statuses

completed

The agent produced a submission, the build gate passed, and the evaluator scored it. Only these runs carry a score.

candidate failure

The agent did not produce a gradeable submission — no build, missing required artifacts, or an abandoned attempt. This is benchmark evidence and is kept as recorded. It is never silently retried into a success, and no score is shown or implied.

infra failure

The harness itself failed: evaluator, tooling, or environment. Infrastructure failures are the only class of failure eligible for a retry, and they are never converted into model failures.

timeout

The run hit the case time limit. No score is computed.

Failure runs are set muted in every ledger and never display a score number, including in "best overall" aggregates.

03

How a run is scored

Install and build are gates, not weighted components: a build failure scores zero. Once the gate passes, the score is a weighted sum of normalized visual fidelity across every declared viewport target and interaction state, plus deterministic functional checks. Each case declares its own weights and visual floor; the run page shows the exact arithmetic used.

normalized_visual = max(0, (visual - visual_floor) / (1 - visual_floor))
overall           = w_visual * normalized_visual + w_functional * functional

The visual grader reports dimension, per-pixel, coarse-layout thumbnail, foreground-overlap and edge-overlap similarities. It deliberately avoids an opaque vision-model judge: every number on a run page is reproducible from the published digests. Missing checkpoints stay in the denominator, so skipping a viewport is not a way to raise a score.

During the harness-side live-web authoring pipeline, admission checks compare the curator's public source report against sealed provenance: discovery URL, original post, chosen ground truth, evidence-chain coverage, classification and redistribution boundary. Candidates then reconstruct from the frozen bundle; source tracing is not candidate scoring.

04

What is and is not published

Published

  • The public task brief and rendered candidate prompts.
  • Oracle reference screenshots, because comparison views need them.
  • Per-checkpoint similarity metrics and per-check pass/fail with error text.
  • Agent selector, variant dimensions, timing, token usage and cost.
  • Content digests for task, submission, oracle, grader and evaluator environment.
  • A derived trace summary: event, tool-call, command and edit counts.

Withheld

  • Grader code, thresholds and check selectors.
  • Provenance forensics and any sealed private assets.
  • Raw events.jsonl and stderr.log.
  • Absolute host paths, environment variable names and values, argv, platform detail.

The export step builds this bundle by explicit field whitelist — never by copying a manifest or a grade wholesale.

05

Redistribution

Every case declares a redistribution policy. A case marked research-only derives its references from third-party media held under a research boundary: the captures are shown so results can be checked, and must not be redistributed or reused as source assets. Cases marked public use original, licensed or explicitly redistributable evidence.

Cases marked dev-smoke were used while the harness itself was being built. They exercise the whole protocol but are excluded from leaderboard aggregates.