Part II: What ARC-AGI-3 measured

2.1 The benchmark

ARC-AGI-3 launched in March 2026 as the first interactive entry in a series that began in 2019. Where ARC-AGI-1 and 2 were static grid puzzles, ARC-AGI-3 is a set of turn-based game environments with no instructions. The agent sees a screen, has a small set of actions, and must figure out what the game is, what winning means, and how to get there. The environments are built only on what Chollet calls core knowledge priors: objectness, basic geometry, counting, agency, the things a child has before school. Every environment was validated as solvable by untrained humans: ARC’s technical report (arXiv 2603.24621, March 2026) states that humans solve 100 percent of the environments and that frontier systems at launch scored below 1 percent, and its September post cites roughly 500 members of the public in the baseline study, none selected for puzzle-solving ability.

ARC lists four capabilities the benchmark targets: exploration, modeling, goal-setting, and planning and execution. Note what is not on the list. Continual learning across environments. Handling rules that change. Operating under noise. Inventing anything. These were not design failures; they were scope decisions, and ARC has been explicit that the environments are deterministic and closed-ended.

The scoring metric matters. ARC-AGI-3 does not score task completion. It scores Relative Human Action Efficiency, which compares the number of actions an agent takes to solve a level against the median number of actions among humans who solved it. An agent that solves a level in three times as many actions as a human is not credited as having solved it in the same sense. The metric is adapted from Success weighted by Path Length, the standard efficiency measure in embodied navigation, and a subsequent formal analysis (arXiv 2605.25931) argues it penalizes inefficiency quadratically. This is the right instinct. Completion tells you the agent got there; action count tells you how much experience it needed to get there, which is closer to what “learning” means.

2.2 The Astra numbers

Here is the full table ARC published on September 3 for the Semi-Private set, reproduced because the structure of the numbers is the argument.

The Standard harness gives the model a visible notes field and nothing else; it chooses what to carry forward. The Provider Adapter harness lets OpenAI’s own context management run: opaque reasoning state persists between requests and long conversations are compacted. ARC reports that across the 167 game-and-effort pairs both harnesses solved, the Provider Adapter runs were about 3.66 times faster and used 49 percent fewer tokens.

Read the table column by column.

The Standard column behaves like a capability curve. Score rises with reasoning effort, from 17.5 to 62.7 percent, a spread of 45 points. More thinking, better results. This is what everyone expects a benchmark to show.

The Provider column is flat. From 96.7 at no reasoning to 99.9 at high: a spread of 3.2 points. Reasoning effort barely matters. The model at zero reasoning effort with good state management beats the model at maximum reasoning effort without it, by 34 points.

There is also a small anomaly worth flagging: under the Standard harness, “none” (35.2 percent) outperforms “low” (17.5 percent). I do not have an explanation, and ARC’s September post does not offer one. It may be that at low effort the model spends its limited reasoning budget on note-taking rather than play, and at zero effort it just plays. It may be noise. Either way, a benchmark where less reasoning beats more reasoning at some point on the curve is telling you that something other than reasoning is being measured at that point.

2.3 What the flatness means

Here is the interpretation I think the data forces.

Under the Standard harness, the model has to reconstruct its understanding of the game from its own notes on every turn. Reasoning effort helps because more reasoning means better reconstruction. The 45-point spread is not measuring how well Astra reasons about the game. It is measuring how well Astra reasons its way back to where it was before the context reset. It is a memory-management benchmark wearing a reasoning benchmark’s clothes.

Under the Provider harness, the state persists. There is nothing to reconstruct. The residual task, the actual game, turns out to be nearly saturated at every reasoning level, which is why the column is flat.

So the “capability curve” in the Standard column is mostly a curve of harness compensation. The true capability signal, the part that responds to the games rather than to the interface, is the flat line at the top of the Provider column.

I want to be careful here. It is possible to read this the other way: that state management is a core capability, that a system which cannot keep its own notes straight is less intelligent, and that the Standard harness is the honest test. ARC itself says a future AGI should be able to pass under Standard conditions. I do not disagree. But the two readings have opposite implications for what the score means, and the leaderboard cannot tell them apart. That is the problem.

2.4 The harness ratio

The fix is a number. Define, for a model \(M\) on benchmark \(B\), the harness ratio:

$$ H(M,B) = \frac{S_{\text{provider}}(M,B)}{S_{\text{standard}}(M,B)} $$

the score under the most permissive harness the evaluator will run, divided by the score under the standard one. It is a dimensionless measure of how much of a model’s apparent capability lives in the scaffold rather than the weights.

For Astra on ARC-AGI-3, from the table:

For GPT-5.6 Sol on the public set in July, from OpenAI’s own numbers: 38.3 / 13.3 = 2.88.

An \(H\) of 1.0 means the harness does not matter: the model manages its own state well enough that giving it help changes nothing. That is, I would argue, part of what AGI should mean. An \(H\) of 5.6 means that at that operating point, 82 percent of the reported score is scaffold.

Three properties of \(H\) that make it useful:

It is monotone in reasoning effort, for the right reason. As reasoning effort rises, \(H\) falls toward 1. More thinking lets the model compensate for the missing state. This is exactly what you would expect if the Standard harness penalty is a reconstruction cost, and it lets you read off, roughly, how much reasoning it takes to substitute for memory.

It is cheap. Both numbers are already being computed. ARC has committed to reporting both harnesses going forward. All \(H\) adds is the division.

It is comparable across benchmarks. A model with \(H\) near 1 on three different benchmarks has demonstrated something about itself, not about the benchmarks. A model with \(H\) of 3 everywhere has demonstrated something about its provider’s context management.

The obvious objection: providers will optimize \(H\) directly, by making the Standard harness score look good. Fine. That is the point. A model that scores 99 percent under any harness has closed the gap that \(H\) measures, and closing that gap is a real capability. Goodharting toward \(H = 1\) is Goodharting toward a model that does not need a scaffold, which is the model we actually want to know about.

Proposal 1. Every leaderboard that runs more than one harness should report \(H\) alongside the score. Every leaderboard that runs only one harness should say which.

2.5 The half-life of a benchmark

There is a second, cruder number that I think matters more for anyone deciding what to build.

ARC-AGI-1 was published with Chollet’s “On the Measure of Intelligence” in November 2019. It was effectively broken in December 2024, when OpenAI’s o3 posted 87.5 percent on the semi-private set at very high test-time compute (and 75.7 percent at lower compute). Call it 61 months.

ARC-AGI-2 was published in March 2025 with a stated grand-prize threshold of 85 percent. In May 2025 the best frontier score was 3 percent. By the start of 2026 GPT-5.2 Pro was at 54 percent; by late February 2026 Gemini 3 Deep Think had reached 84.6 percent and Gemini 3.1 Pro 88.1 percent on the public set, with Imbue’s evolution harness pushing the latter to 95 percent. Astra now sits near 95 percent on public aggregators. Whether one dates the crossing to February or April 2026, ARC-AGI-2 lasted roughly 11 to 13 months. (Two caveats: these are unconstrained frontier scores, not the compute-limited Kaggle track, and the semi-private verification queue was paused for part of this period, so the exact crossing date is soft.)

ARC-AGI-3 launched publicly in March 2026, with a developer preview in 2025. It was saturated on September 3, 2026. Six months.

Three points do not make a law, but they make a direction:

If each generation lasts a constant fraction of the previous one,

$$ L_n = L_1 \, k^{-(n-1)} $$

then a least-squares fit in log space across the three points gives \(k \approx 3.2\), with the individual ratios (about 5 and about 2) disagreeing enough that the exponent should be read as “somewhere between 2 and 5.” On that range, ARC-AGI-4, whatever it turns out to be, has an expected lifetime between one and three months.

I do not think the number is reliable. I think the shape is. ARC’s own strategy is explicit: Chollet said in March 2026 that ARC-AGI-4 will ship in early 2027 and that the foundation is aiming for a yearly release schedule, with each new benchmark fully unsaturated on release. General Intuition has already donated just over a million dollars toward building it. So the plan is one benchmark per year, designed to last. ARC-AGI-3 lasted six months. That strategy assumes you can build the next benchmark faster than the models can saturate it, and the current data says the gap between build time and saturation time is closing from the wrong direction. At a six-month cycle that is already implausible for an organization building hundreds of environments and running human baselines on each: ARC-AGI-2 alone required 407 participants across 515 sessions. At a two-month cycle it is not a strategy. It is a treadmill you fall off.

There is a second reading of the same table that is worth stating because it is more charitable to the models. Each ARC generation has been broken not by a bigger model alone but by a bigger model plus a new harness: o3 with massive test-time search on ARC-AGI-1, refinement and evolution loops on ARC-AGI-2, persistent reasoning state on ARC-AGI-3. Epoch-style analyses of algorithmic progress put catch-up efficiency gains since 2023, including post-training, far above the historical 3x per year. Benchmarks are not just being outrun by scale. They are being outrun by the scaffold, which iterates faster than the weights do.

The implication either way is that the next benchmark cannot be another ARC, a harder version of the same closed puzzle format. Whatever gets built next has to test something that does not saturate the moment a lab points a harness at it. That means, at minimum, that the thing being measured has to be a rate rather than a level: not “can it solve this” but “how fast does it get better at solving things it has not seen,” where the things it has not seen are regenerated every time.

2.6 What saturation did not show

To be precise about the residual: here is what Astra’s 99.9 percent demonstrates and what it does not.

It demonstrates that, given persistent state, a 2026 frontier model can enter an unfamiliar deterministic environment built on core knowledge priors, infer its rules and goal without instructions, build a compact symbolic model of it (ARC observed Astra inventing a domain-specific shorthand to track game state), and execute a solution in fewer actions than the median human. That is a real thing and it was not true in March.

It does not demonstrate:

• Adaptation. The rules of every ARC-AGI-3 environment are fixed for the duration of the episode. Nothing in the score tells you what happens when they change.
• Robustness. The environments are deterministic. Nothing tells you what happens under noise, partial observability, or a screen that lies to you.
• Transfer across environments. Each game is scored independently. Nothing tells you whether the model got faster at game 20 because of games 1 through 19. There is no learning-curve term in the metric.
• Open-ended goals. Every game has a win state. Nothing tells you what the model does when there is not one.
• Invention. The model modeled a world it was placed in. Nothing tells you whether it can propose a world, or a question, or an experiment.

ARC listed the first and fourth of these as the questions shaping their next generation. I want to argue that the first and third are the ones to build on, because they are the ones that can be turned into a rate.