Start here
About 60 secondsThe idea, in a minute.
- The question
- Does predicting how a model will change help it choose better learning updates?
- What we found
- A learned self-model predicted changes more accurately, but both controllers chose the same half-strength update on every step.
- Why it matters
- Better internal prediction is useful only if it changes a consequential decision. This experiment separates those two achievements.
One pretrained CIFAR-100 initialization; the useful prediction result used an exploratory higher learning rate. Production weights were not locally retrievable.
A concrete example
The self-model did not change the decision
Recorded revised controller experiment.
- 01
Improve the forecast
Learned residuals reduced validation prediction MSE by 25.5% at the higher update rate.
- 02
Choose updates
Both utility heads picked half-strength on all 500 steps.
- 03
Compare outcomes
Both reached JEPA loss 0.087218 and probe accuracy 19.42%.
Forecasting progress and decision progress need separate tests.
Archived experiment reports and measurementsRead the formal abstract
A model that can forecast its own learning might avoid harmful updates. But improved forecasting does not guarantee better control. We tested both claims separately using image representations and counterfactual update branches.
Per-block first-order forecasts plus a learned residual reduced prediction MSE by 25.5% on validation and 24.6% on held-out states at an exploratory higher update rate. Yet the analytic and learned-self utility controllers both chose 0.5-times updates at all 500 control steps. Their final JEPA losses and linear probes were indistinguishable. Prediction improved; the demonstrated decision reduced to lowering an aggressive learning rate.
01 Research question
Does predicting how a model will change help it choose better learning updates?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
Executed September 23, 2026 on DGX Spark GB10. All arms restore the same 5,000-step CIFAR-100 AdamW checkpoint, EMA teacher, and optimizer moments. The image encoder has width 192 and six blocks; its predictor has width 96 and two blocks. Eight parameter groups are controlled. The official CIFAR-100 test split remains unopened.
Eight fixed image probes produce 256 projected activation values across the encoder and predictor. A Jacobian–vector product, or JVP, approximates how activations change under a small parameter update. Version two computes a separate JVP for each parameter group, allowing any candidate gate vector to form an action-specific first-order forecast.
A small residual self-model learns the difference between that forecast and actual post-update state from 500 transitions, including 103 randomized block-gate interventions. Two utility heads then score nine candidate actions using either analytic or learned-self forecasts, trained on separate counterfactual outcomes. In control, actions are selected before drawing the fresh utility-evaluation batch.
The initial 3e-5 continuation rate made the analytic forecast almost exact and the learned residual failed its prediction gate. An exploratory 3e-4 rate increased nonlinearity and passed. Three additional audit trajectories evaluate eighty states and six held-out actions each. Control runs last 500 steps; paired branches at steps 100, 300, and 499 measure effects after one, five, and twenty further updates.
03 What happened?
A learned self-model predicted changes more accurately, but both controllers chose the same half-strength update on every step.
Held-out JEPA loss ↓ · scale shown from 0 to 0.1
How to read thisThe identical outcomes follow from identical action choices. A prediction gain should not be presented as a control gain.
The limitOne pretrained CIFAR-100 initialization; the useful prediction result used an exploratory higher learning rate. Production weights were not locally retrievable.
Single-initialization, exploratory 3e-4 continuation. JEPA loss is in the experiment loss units; linear-probe accuracy is percent.
Full results and data table
| Controller | Held-out JEPA loss ↓ | Linear-probe accuracy (%) ↑ |
|---|---|---|
| AdamW | 0.093969 | 19.16 |
| Analytic JVP | 0.087218 | 19.42 |
| Learned self | 0.087218 | 19.42 |
At 3e-5, validation JVP/self prediction MSE was 3.80e-11/3.97e-11: the learned residual was 4.6% worse. At 3e-4 it was 1.24e-6/9.21e-7, a 25.5% reduction; held-out error fell 24.6%. Three extra audit trajectories showed 14.2%, 18.8%, and 21.0% reductions.
At the higher rate, AdamW final JEPA loss was 0.093969 versus 0.087218 for each utility controller. Linear-probe accuracy was 19.16% versus 19.42%. Effective rank was 18.43 versus 17.81 and feature standard deviation 0.512 versus 0.480. Representation-quality measures did not uniformly improve.
Both heads selected the half-strength action on all eighty utility evaluation states and every one of 500 control steps. Exact counterfactual scoring also preferred it on 67/80 states. A fixed 0.5-times schedule would therefore enact the same decisions; the improvement over 1.0-times AdamW cannot be credited to learned self-model access.
The paired half-strength versus full-strength validation-loss advantage averaged 0.00237 after one update, 0.00063 after five, and 0.000050 after twenty. One of three twenty-step checkpoints had a negative effect. The first pilot also failed to distinguish the self-aware condition from its no-state-change ablation.
04 What does this mean?
What this suggests
- A learned self-model predicted changes more accurately, but both controllers chose the same half-strength update on every step.
- The study establishes a bounded prediction capability without establishing improved control. An action-dependent state forecast is more informative than a single forecast for the identity action, but the utility task must offer real selective choices.
What we still don’t know
- The higher learning rate was selected after seeing the lower-rate result, and all trajectories share one pretrained initialization.
- Final DGX model checkpoints remain outside this publication: the original remote machine was unreachable during packaging. The archive contains code, original reports, action logs, summaries, and audit records, not substitute smoke-test weights.
Read the full discussion
The study establishes a bounded prediction capability without establishing improved control. An action-dependent state forecast is more informative than a single forecast for the identity action, but the utility task must offer real selective choices.
A future test needs a calibrated base rate, stronger fixed-rate controls, longer-horizon action differences, and multiple initializations. The current result does not justify expanding controller capacity.
05 Limitations
One pretrained CIFAR-100 initialization; the useful prediction result used an exploratory higher learning rate. Production weights were not locally retrievable.
Read all study limitations
The higher learning rate was selected after seeing the lower-rate result, and all trajectories share one pretrained initialization.
Final DGX model checkpoints remain outside this publication: the original remote machine was unreachable during packaging. The archive contains code, original reports, action logs, summaries, and audit records, not substitute smoke-test weights.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
Production DGX checkpoints were not available for retrieval at publication. This download contains actual code, saved metrics, and action/audit records; it contains no trained production weights or replacement smoke-test checkpoints.