A model could learn corrections; its RL controller still lost

Start here

About 60 seconds

The idea, in a minute.

The question: Once a model can make useful corrections, does reinforcement learning choose better updates than a fixed recipe?
What we found: The RL controller acquired slightly more corrections but had 41% higher consequence error than fixed updates.
Why it matters: A more adaptive optimizer is not automatically better. It needs to beat the simple update policy already available.

Keep in mind

Five held-out synthetic models, two sixteen-correction streams each, and three policy seeds; the preregistered RL gate failed.

See the evidence

A concrete example

More corrections acquired, more damage left behind

Recorded final comparison on editable representations.

01
Use a fixed recipe
Direct acquisition succeeded 95.6% of the time.
02
Let RL choose
Success rose to 97.3%, but consequence error rose from 0.005581 to 0.007867.
03
Audit earlier edits
Their error rose from 0.009770 to 0.014059 under RL.

Acquiring the corrected example is only one part of a useful update.

Archived experiment reports and measurements

Read the formal abstract

An early update-policy experiment could appear safe by refusing to learn. This revision first made useful correction actions feasible and trained the policy on its own evolving model states.

The resulting fixed recipe achieved 95.6% correction success and consequence MSE 0.005581. RL increased success to 97.3%, but consequence MSE rose to 0.007867 and earlier-correction MSE to 0.014059 from 0.009770. Full-check greedy choice stayed close to the fixed recipe. The remaining obstacle was policy learning, rather than an inability to make useful model edits.

01 Research question

Once a model can make useful corrections, does reinforcement learning choose better updates than a fixed recipe?

02 Experiment setup

The full procedure, model settings, datasets, and evaluation choices are recorded here.

Explore the complete methodology

Executed September 18, 2026 on an RTX 4090 after local feasibility checks. The model is the inherited 39,121-parameter transformer in the artificial conductivity/circuit world. Every model receives 2,000 base AdamW steps; editable representations receive 600 episodes with three sequential corrections and first-order differentiation through SGD updates.

Development uses model 899; controller training uses 900–902 and validation 903. Final unseen models 60–64 each run two streams of sixteen corrections. Four editable materials recur in each stream, with updated labels replacing obsolete corrections. Final edit identities differ from those used in meta-training.

Sixteen direct evidence examples support each correction. Training rewards use full protected banks and earlier-correction probes; separate banks audit stable knowledge, new consequences, and earlier edits. Replay holds the latest direct evidence for previously corrected materials, excluding the current target.

Three RL policy seeds are compared with development-selected fixed actions, greedy full checks, and no update. The controller receives nine summary features. The final design trains policies on their own sequentially changing state rather than only reset states. Included artifacts contain final model and policy checkpoints, raw outcomes, protocol, and audit.

03 What happened?

The RL controller acquired slightly more corrections but had 41% higher consequence error than fixed updates.

Figure 01 / Observed resultsA strong fixed baseline

41% higherRL consequence error versus fixed updates

0.010.0050

0.005581

Fixed

0.007867

0.005664

Greedy checks

Consequence MSE ↓ · scale shown from 0 to 0.01

How to read thisA slightly higher success rate on direct corrections came with worse consequences and retention. The fixed and greedy conditions were much closer to each other.

The limitFive held-out synthetic models, two sixteen-correction streams each, and three policy seeds; the preregistered RL gate failed.

Five held-out model seeds; two sixteen-correction streams per seed; RL summarized across three policy seeds.

Full results and data table

Complete summary results
Update policy	Consequence MSE ↓	Earlier-correction MSE ↓
Fixed	0.005581	0.00977
RL	0.007867	0.014059
Greedy checks	0.005664	0.009831

Editable fixed / RL / greedy / no-update acquisition success was 95.6% / 97.3% / 97.5% / 5.6%. Consequence MSE was 0.005581 / 0.007867 / 0.005664 / 0.015489.

Stable-knowledge MSE for fixed / RL / greedy was 0.011245 / 0.017563 / 0.011090. Earlier-correction MSE was 0.009770 / 0.014059 / 0.009831. RL improved earlier-correction retention by at least 10% on zero of five models, failing the declared gate.

Compared with ordinary representations under the same fixed action, editable representations reduced consequence error from 0.011985 to 0.005581 and earlier-correction error from 0.024485 to 0.009770. This isolates a useful representation benefit from the failure of the learned controller.

Stable-knowledge MSE under RL also exceeded the declared 0.01 threshold. The no-update arm illustrates the opposite failure mode: it preserved the original stable world while acquiring almost none of the requested corrections. Successful direct acquisition or low old-world error alone is not a pass.

04 What does this mean?

What this suggests

The RL controller acquired slightly more corrections but had 41% higher consequence error than fixed updates.
Useful correction learning and useful policy learning are separate achievements. Here the fixed update was already strong, leaving less room for a controller to improve.

What we still don’t know

The experiment does not test pretrained language models, unreliable sources, or indefinite continual learning.
Nine state features and few training trajectories limit policy learning. Three policy seeds do not establish a general failure of RL-based optimization.

Read the full discussion

Useful correction learning and useful policy learning are separate achievements. Here the fixed update was already strong, leaving less room for a controller to improve.

Richer state and more varied trajectories are plausible next tests, but neither was demonstrated. Increasing controller complexity without establishing meaningful action choices would be premature.

05 Limitations

Five held-out synthetic models, two sixteen-correction streams each, and three policy seeds; the preregistered RL gate failed.

Read all study limitations

The experiment does not test pretrained language models, unreliable sources, or indefinite continual learning.

Nine state features and few training trajectories limit policy learning. Three policy seeds do not establish a general failure of RL-based optimization.