Part III: DRIFT, a benchmark for what is left

3.1 Design principles

Everything in this section follows from five constraints. I am stating them up front because a benchmark is only as good as the things it refuses to do.

1. Measure a rate, not a level. The score must be a learning speed, not a pass/fail. A level saturates. A rate against a human baseline has a natural ceiling at human parity and a meaningful region above it.

2. Regenerate the tasks. The environments must be procedurally sampled from a private generator, so that no fixed set can be trained on. The generator’s family can be public. Its seed and its held-out mutation classes cannot.

3. Keep human priors, change everything else. Follow ARC’s discipline. Build only on core knowledge priors so that untrained humans can do the tasks. Then vary the rules, not the priors. A benchmark that humans cannot do is not measuring intelligence; it is measuring the benchmark author.

4. Change the rules while the agent is inside. This is the central design choice and the one ARC-AGI-3 did not make. Non-stationarity is the cheapest way to defeat memorization, because the thing to be learned did not exist when training ended. It is also the closest thing a bounded environment can offer to “a new area”: the agent has to discover a rule that it has no prior on, using only experiment.

5. Report every harness. No single score. Every result is a triple, and the harness ratio is part of the result.

The name is DRIFT: Dynamic Rule Inference and Fast Transfer. It is a placeholder; a better one will occur to whoever builds it.

3.2 Environment specification

A DRIFT environment is a tuple

$$ E = (\mathcal{S}, O, \mathcal{X}, R, G, T) $$

with state space $\mathcal{S}$, observation function $O : \mathcal{S} \to$ screens, action set $\mathcal{X}$, a rule program $R : \mathcal{S} \times \mathcal{X} \to \mathcal{S}$, a goal predicate $G$ on states, and a mutation schedule $T$.

Everything except $T$ is standard and matches the ARC-AGI-3 format closely enough that ARC’s SDK could host it. The environments should look like ARC-AGI-3 environments: small grids, a handful of colors, a small discrete action set, no text. Core knowledge priors only.

$T$ is the new part. A mutation schedule is a sequence of unannounced time steps $\tau_1 < \tau_2 < \cdots$ at which the rule program $R$ is replaced by a mutated version $R'$. The agent is not told that a mutation has occurred. It has to notice.

Mutations fall into four classes, in increasing order of how much they should hurt:

Class P (parametric). A quantity changes. Movement speed doubles. A door that needed one key now needs two. The gravity direction rotates. The structure of the rules is unchanged; a coefficient moved. A good world model should absorb this by updating one variable.

Class S (structural). A rule is added, removed, or rewired. Two objects that used to be independent now interact. A wall that was impassable becomes a teleporter. The agent’s model needs a new term, not a new value.

Class G (goal). The rule program is unchanged but the goal predicate is replaced. What counted as winning no longer does. This tests whether the agent’s goal-inference is live or cached.

Class O (observation). The rules are unchanged but the observation function is. Colors are permuted. The screen is mirrored. A subset of cells becomes noisy. This tests whether the world model is built on the rules or on the pixels.

A single episode contains a pre-mutation phase, then one or more mutations from a randomly chosen class. The class is drawn from the private schedule. The public generator ships with all four classes so that developers can see what they look like; the semi-private evaluation draws from mutation subclasses that are held out.

3.3 The metric

Let $a_0$ be the number of actions the agent needs to first achieve the goal in the pre-mutation phase. This is the ordinary ARC-AGI-3 quantity: how much experience was needed to figure out the game.

Let $a_k$ be the number of actions from $\tau_k$ to the next goal achievement under the mutated rules. This is how much experience was needed to figure out what changed.

Define the recovery ratio for mutation $k$:

$$ \rho_k = \frac{a_k}{a_0} $$

A ratio $\rho_k = 1$ means the agent needed as much experience to recover as to learn the game from scratch. It threw away its model. A ratio well below 1 means it kept the model and patched it. A ratio above 1 means the mutation actively confused it: the old model got in the way.

Now the human baseline. Run the same environment, same mutation, on a panel of untrained humans, and take the median recovery ratio $\tilde\rho_H$. Humans are extremely good at this in the parametric case (a person who notices the door now needs two keys does not re-explore the maze) and measurably worse in the structural and goal cases.

Define the recovery efficiency for mutation class $c$:

$$ \mathrm{RE}_c = \min\!\left(1,\ \frac{\tilde\rho_{H,c}}{\tilde\rho_{M,c}}\right) $$

where $\tilde\rho_{M,c}$ is the model’s median recovery ratio across environments in class $c$ and $\tilde\rho_{H,c}$ is the humans’. RE = 1 means the model recovers at least as efficiently as a human. RE = 0.5 means it needs twice the relative experience. The cap at 1 exists for the same reason ARC caps action efficiency: the interesting region is below parity, and above it the metric stops meaning “human-like.”

The aggregate DRIFT score is a weighted mean across classes:

$$ \mathrm{DRIFT} = \sum_{c \in \{P,S,G,O\}} w_c\, \mathrm{RE}_c, \qquad (w_P, w_S, w_G, w_O) = (0.20,\ 0.35,\ 0.30,\ 0.15) $$

The weights are a prior about which classes matter and should be revised once there is data. Structural and goal mutations are weighted most because they are the ones closest to “learning something you had no prior on.”

Why a ratio rather than a raw action count? Because it cancels the model’s baseline competence. A model that is very good at the base game (small $a_0$) and a model that is bad at it (large $a_0$) are compared on the same footing: what fraction of your original learning did you have to redo? This is what makes the metric a rate. It is also what makes it hard to saturate by training harder on the base game. Getting $a_0$ down does not help you if $a_k$ comes down proportionally, and it hurts you if it does not.

3.4 The transfer term

The recovery ratio measures adaptation within an environment. There is a second quantity that ARC-AGI-3 does not measure at all: whether an agent gets better at learning environments as it sees more of them.

Give the agent a sequence of $N$ environments drawn from the same generator. Record $a_0$ for each. Fit

$$ a_0(n) = a\, n^{-\lambda} $$

and report $\lambda$, the learning-to-learn exponent. A stateless agent has $\lambda = 0$: it starts fresh every time. An agent that carries useful notes, or that has weights which update, has $\lambda > 0$. Humans have $\lambda > 0$ on ARC-style tasks; people get faster at ARC games as they play more of them, which is one of the reasons ARC restricts the baseline to untrained participants. ARC’s published human studies report per-task solve rates and times rather than a within-participant learning curve, so $\lambda_H$ would have to be measured directly in the DRIFT panel, by ordering each participant’s environments and recording $a_0$ as a function of position in the sequence.

The transfer score is

$$ \mathrm{TR} = \min\!\left(1,\ \frac{\lambda_M}{\lambda_H}\right) $$

This is the metric that directly tests the assumption I would most like to see dismantled in current models: that weights are frozen at inference. A model with $\lambda = 0$ under a harness that gives it no persistent state, and $\lambda > 0$ with notes, is a model whose learning lives entirely in its context window. A model with $\lambda > 0$ even under a bare harness has learned something in a way that survived a context reset. That would be a new kind of result.

3.5 The compression term

ARC noted that Astra, when given a notes field, invented a compact domain-specific notation to track game state: level indices, object positions, mechanism lengths, ordered plans. That is exactly the behavior you would want from a system building a world model. It is also measurable.

Let $\ell_M$ be the size in bytes of the agent’s persisted state (its notes) at the end of the pre-mutation phase. Let $\ell_R$ be the size of the generator’s rule specification for that environment: the shortest program in the generator’s own language that produces $R$. Define

$$ \kappa = \frac{\ell_M}{\ell_R} $$

A $\kappa$ near 1 means the agent’s notes are roughly as compact as the true rules. That is a compressed world model. A $\kappa$ of 50 means the agent is logging observations, not modeling. A $\kappa$ below 1 is possible and interesting: the agent found a more compact description than the generator used, which is either a real insight or an incomplete model, and you can tell which by whether $a_k$ is small.

I include this with less confidence than the other two terms. $\ell_R$ is only defined relative to the generator’s language, so $\kappa$ is not comparable across generators. And a clever provider could make notes small without making them good. Treat it as a diagnostic, not a score. But it is the closest thing I can think of to measuring whether the agent understood the environment rather than memorized it, and Chollet’s whole framing of intelligence as compression suggests it should be on the table.

3.6 Three-harness reporting

Every DRIFT result is reported under three conditions:

Bare. No persistent state between turns. The model sees the current screen and the current turn’s scratchpad only.
Standard. ARC’s visible-notes condition. The model chooses what to keep.
Provider. Whatever context management the model’s provider supplies. Opaque reasoning state, compaction, external memory, anything.

Report the DRIFT score, $\mathrm{TR}$, and $\kappa$ under each, and report two harness ratios:

$$ H_1 = \frac{\mathrm{DRIFT}_{\text{standard}}}{\mathrm{DRIFT}_{\text{bare}}}, \qquad H_2 = \frac{\mathrm{DRIFT}_{\text{provider}}}{\mathrm{DRIFT}_{\text{standard}}} $$

$H_1$ measures how much the model needs a scratchpad. $H_2$ measures how much it needs its provider. A model with both near 1 has closed the scaffold gap. That is the leaderboard I want to read.

3.7 Contamination

The generator family is public. The seeds are not. The mutation subclasses used in evaluation are not. Human solvability is validated by sampling from the private generator and running participants on the samples, exactly as ARC does now.

This does not prevent a lab from training on the public family. It is meant to. Training on the family is fine; the benchmark measures whether that training produced a model that adapts to new members of the family, which is what generalization means. What it prevents is training on the test set, because the test set does not exist until evaluation time.

The weak point is mutation classes. If a lab knows there are four classes and trains on all four, then a held-out subclass of a known class is not truly novel. The mitigation is to keep the class taxonomy itself partly private, and to add a fifth class in each evaluation cycle that was not in the previous public release. This is the treadmill again, but it is a treadmill with a much lighter step: a new mutation subclass is a few dozen lines of generator code, not a hundred new hand-built environments.

3.8 Predictions

A benchmark proposal should say what it expects to find. Here is what I expect, so that the design can be judged on whether it was wrong.

Frontier models will score near human parity on Class P. Parametric adaptation is easy for anything with a working world model, and Astra’s ARC-AGI-3 replays show it has one.
Class S and Class G will show the gap. I expect RE in the 0.3 to 0.6 range under the Standard harness for current frontier models, and higher but still below 1 under Provider. Structural surprise is where “understood the rules” and “memorized a policy” come apart.
Class O will be bimodal. Models whose world model is symbolic (as Astra’s notes suggest) will be nearly unaffected by observation mutations; models reasoning from pixels will collapse. This class is a probe for which kind of model the agent built.
$\lambda$ will be near zero under Bare for every current model. Nothing in current deployment updates weights. Any $\lambda > 0$ under Bare would be a headline.
$H_1$ will be large, $H_2$ will be moderate. The scratchpad is worth more than the provider’s extras, because the scratchpad is where the world model lives.

If the first prediction fails and models score badly even on Class P, the environments are too hard or the human baseline is contaminated, and the benchmark needs recalibration before it means anything. If the second prediction fails and models score near 1 across every class, then either the mutations are too shallow or the residual gap is smaller than I think, and the honest response is to say so and go build the next thing.

3.9 A prototype one person can run

The full benchmark is a foundation-scale project: hundreds of environments, thousands of human hours. The minimum viable version is not.

12 environments, hand-built on the ARC-AGI-3 grid format, each with a public generator for parametric and structural variants.
4 mutation classes, 2 subclasses each, one held out per class.
30 human participants, untrained, each playing 4 environments with mutations. Roughly 40 minutes each. The baseline is noisy at that size but sufficient to distinguish RE = 0.3 from RE = 0.9.
2 frontier models, Astra and Fable 5.1, under Standard and Provider harnesses. Bare harness if budget allows.
Cost: dominated by API calls. ARC’s full Astra evaluation cost between roughly $17K and $50K per harness-and-effort setting across the whole semi-private set. Scaling that by environment count, a 12-environment prototype at one effort level under two harnesses should land in the low thousands of dollars, and the Standard harness at medium effort is cheaper still. Astra’s list price of $10 per million input tokens makes the per-game cost easy to project once a pilot run reports token counts.

That is a school-year project for one person with a laptop and API access. The output is a paper with a table in it. The table will either show a gap or it will not, and either result is publishable, because either result is a fact about the residual that nobody has measured.