The Measurement Gap
OverviewPart IPart IIPart IIIPart IVAppendicesPDF

Appendices

Appendices

A worked example of the metric, and every source used. About 4 min.

Appendices

Appendix A: Worked example of recovery efficiency

The numbers in this appendix are illustrative. They were chosen to show how the metric behaves, not measured.

Consider a single DRIFT environment: a 12 by 12 grid, an agent, three colored keys, three matching doors, and a goal cell behind the last door. Core knowledge priors only: objects, containment, matching by color.

Pre-mutation phase. A human participant explores, discovers that each key opens its matching door, and reaches the goal in a median of 41 actions. A frontier model under the Standard harness reaches it in 28. So \(a_{0,H} = 41\) and \(a_{0,M} = 28\). The model is more action-efficient on the base game, consistent with ARC’s Astra findings.

Class P mutation. At \(\tau_1\), without announcement, every door now requires two keys of its color, and a second key of each color has appeared elsewhere on the grid. A human notices the first door does not open, looks for another key, and reaches the goal in a median of 22 further actions. The model, under the Standard harness, spends 35 actions re-verifying the door mechanics it had already recorded before finding the second key.

$$ \rho_H = \tfrac{22}{41} = 0.54, \qquad \rho_M = \tfrac{35}{28} = 1.25, \qquad \mathrm{RE}_P = \min(1, 0.54/1.25) = 0.43 $$

The model was faster than the human at learning the game and slower at learning that the game had changed. Its recovery ratio above 1 says the old model got in the way: it trusted its notes over the screen. That is the number a leaderboard should show, and it is invisible in a completion score, because both the human and the model completed the level.

Class S mutation, second run. Same environment, but now at \(\tau_1\) the red key, when picked up, closes the blue door instead of opening the red one. This is a rewiring, not a coefficient. Human median \(a_{1,H} = 38\); the humans nearly re-learn the game. Model \(a_{1,M} = 61\).

$$ \rho_H = \tfrac{38}{41} = 0.93, \qquad \rho_M = \tfrac{61}{28} = 2.18, \qquad \mathrm{RE}_S = \min(1, 0.93/2.18) = 0.43 $$

Same RE by coincidence, different story. Here the human also struggled; structural surprise is hard for everyone. The model’s ratio of 2.18 means it needed more than twice its original learning to recover, which is what you would expect from a system that built a policy rather than a rule set.

Aggregating. With one run each of Class P and S and the weights from section 3.3, restricted to those two classes and renormalized:

$$ \mathrm{DRIFT}_{\text{partial}} = \frac{0.20 \cdot 0.43 + 0.35 \cdot 0.43}{0.55} = 0.43 $$

Under the Provider harness the same model might show \(a_{1,M} = 19\) on the Class P mutation, because persistent state made re-verification unnecessary:

$$ \rho_M = \tfrac{19}{28} = 0.68, \qquad \mathrm{RE}_P = \min(1, 0.54/0.68) = 0.79, \qquad H_2^{(P)} = \tfrac{0.79}{0.43} = 1.84 $$

An \(H_2\) of 1.84 on adaptation, in this made-up example, would be the same finding ARC reported on ARC-AGI-3 in a different form: most of the gap between the model and the human was a memory gap, not a modeling gap. Which is the kind of thing a benchmark should be able to say.

Appendix B: Sources

Primary sources referenced in the text, checked September 7, 2026.

  • Aschenbrenner, L. (2024). Situational Awareness: The Decade Ahead. situational-awareness.ai
  • ARC Prize (2026). “OpenAI’s GPT-6 Astra on ARC-AGI-3.” arcprize.org/blog/astra, 3 Sept 2026.
  • ARC Prize (2026). ARC-AGI-3 benchmark page and 2026 competition rules. arcprize.org/arc-agi/3
  • ARC Prize Foundation (2026). “ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence.” arXiv:2603.24621, 24 March 2026.
  • ARC Prize (2026). ARC-AGI-2 benchmark page (human calibration study, 400+ participants). arcprize.org/arc-agi/2
  • Chollet, F. (2026). Posts on X, 12 Feb and 26 Mar 2026, on ARC-AGI-4 timing and the yearly release cadence.
  • ARC Prize (2026). “Announcing: ARC Prize Foundation + General Intuition” (donation toward ARC-AGI-4). arcprize.org newsletter.
  • Imbue (2026). “Beating ARC-AGI-2 with Code Evolution.” imbue.com, 27 Feb 2026.
  • Ho, A. et al. (2024). “Algorithmic progress in language models.” Epoch AI.
  • Anderson, P. et al. (2018). “On Evaluation of Embodied Navigation Agents.” (source of the SPL metric that RHAE adapts)
  • ARC Prize (2025). ARC Prize 2025 Technical Report. arXiv:2601.10904.
  • Chollet, F. et al. (2025). “ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems.” arXiv:2505.11831.
  • OpenAI (2026). “How enabling two settings tripled our scores on the ARC-AGI-3 benchmark.” openai.com, 29 July 2026.
  • OpenAI (2026). GPT-6 Astra System Card. deploymentsafety.openai.com/gpt-6-astra.
  • OpenAI (2026). “Path to Astra: critical capabilities and frontier safeguards.” openai.com/index/path-to-astra.
  • Anthropic (2026). “Introducing Claude Fable 5.1 and Claude Mythos 5.1.” anthropic.com/claude-fable-and-mythos-5-1.
  • “Explore Before You Solve: The Speed-Depth Trade-off in Epistemic Agents for ARC-AGI-3.” arXiv:2605.25931.
  • “Situational Awareness: A Two-Year Scorecard.” EA Forum, June 2026.
  • “Situational Awareness: A One-Year Retrospective.” LessWrong, June 2025.
  • “Aschenbrenner’s Receipts.” philippdubach.com, May 2026.
  • Axios (2026). “OpenAI releases new model GPT-6 Astra, says it may represent AGI.” 3 Sept 2026.
  • Wikipedia. “GPT-6 Astra” and “Claude Mythos.” Accessed 7 Sept 2026.

Background reading that shaped the benchmark design but is not cited directly:

  • Chollet, F. (2019). “On the Measure of Intelligence.” arXiv:1911.01547.
  • Stanley, K. and Lehman, J. (2015). Why Greatness Cannot Be Planned.
  • Oudeyer, P.-Y. and Kaplan, F. (2007). “What is intrinsic motivation? A typology of computational approaches.”
  • Merrill, W. and Sabharwal, A. (2023). “The Parallelism Tradeoff: Limitations of Log-Precision Transformers.”