Start here
About 60 secondsThe idea, in a minute.
- The question
- Can slow shared updates plus a tiny amount of targeted replay preserve old text behavior while learning new text?
- What we found
- On three fresh 1.4M models, targeted soft replay reduced forgetting 10.4% versus matched random soft replay, with 1.1% more new learning.
- Why it matters
- A small old-data training budget can be valuable, but selecting that data may require many more forward passes than the budget suggests.
Small scratch-trained byte models on reconstructed text; a separate short native-checkpoint transfer favored random selection.
A concrete example
Half a percent of inputs, many more checks
Recorded larger-local-model accounting.
- 01
Spend old training inputs
The default trains on 5,184 old tokens for 1,048,576 new tokens.
- 02
Measure threatened memories
Ongoing verification forwards about 587,776 old tokens on average.
- 03
Judge the complete tradeoff
Retention improves over matching random, but online time rises from 15.61 to 17.51 seconds.
A training-input budget does not measure all of the work.
Archived experiment reports and measurementsRead the formal abstract
Can half a percent of replay training inputs meaningfully reduce forgetting? On small real-text models, combining a low shared learning rate with damage-based selection and frozen probability targets produced a useful retention–acquisition tradeoff.
In three fresh 1,412,608-parameter models, targeted soft replay reduced old-domain loss increase from 0.405419 to 0.363222 versus matching random soft replay, while new-domain learning increased from 0.573190 to 0.579731. Against low shared LR without replay, forgetting fell 63.0%. But the selector forwarded far more old tokens than it trained on, and a short native-model check lost to random. The evidence supports scarce-input allocation in this proxy, not universal or total-compute superiority.
01 Research question
Can slow shared updates plus a tiny amount of targeted replay preserve old text behavior while learning new text?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
The local program progressed from a 77,568-parameter byte model with 64-token context to a 1,412,608-parameter, three-block model with 256-token context. The larger local model has four experts and omits native adaptive halting, paging, and expert growth. All runs use the existing reconstructed real-text corpus and disjoint source regions.
In the larger study, old-domain pretraining uses 6,144,000 token exposures per model and new training uses 1,048,576 per arm. Development seeds 401–402 screen thirty-seven requested configurations, with a duplicated control deduplicated. The recipe is frozen before seeds 501–503 evaluate untouched test halves.
Shared parameters use 0.1 times the ordinary expert learning rate. The replay memory uses independent sixty-four-token windows. Selection ranks measured positive loss increase and refreshes every sixteen new steps, using each cached tile at most once per refresh. Soft replay preserves the starting model probability distribution with KL divergence instead of repeatedly fitting hard token labels.
All larger arms restore identical weights and Adam moments within seed, with corrected CPU optimizer counters. Matched random and targeted conditions consume the same 5,184 replay input/scored tokens in eighty-one extra updates, or 0.494% of new input. Comparators include low LR without replay, random and targeted hard targets, and random and targeted soft targets.
03 What happened?
On three fresh 1.4M models, targeted soft replay reduced forgetting 10.4% versus matched random soft replay, with 1.1% more new learning.
Old loss increase ↓ · scale shown from 0 to 1.1
How to read thisTargeted selection improves the equal-input comparison. Its ongoing verification cost and failed native transfer prevent a general compute-efficiency claim.
The limitSmall scratch-trained byte models on reconstructed text; a separate short native-checkpoint transfer favored random selection.
Three fresh 1.4M model seeds. Replay arms use 5,184 old inputs versus 1,048,576 new inputs; raw held-out loss changes are shown.
Full results and data table
| Method | Old loss increase ↓ | New loss reduction ↑ |
|---|---|---|
| No replay | 0.982307 | 0.557278 |
| Random soft | 0.405419 | 0.57319 |
| Targeted soft | 0.363222 | 0.579731 |
Old forgetting/new learning, in held-out loss changes, was low LR 0.1 without replay 0.982307/0.557278; random hard replay 0.487753/0.565575; random soft 0.405419/0.573190; targeted hard 0.397650/0.571003; and targeted soft 0.363222/0.579731. Positive old change is worse; positive new loss reduction is better.
Targeted soft replay reduced forgetting 10.4% versus matching random soft and 63.0% versus low LR 0.1 alone; new learning rose 1.1% and 4.0% respectively after rounding. Against the stronger LR 0.03 no-replay baseline (0.819051/0.553223), forgetting fell 55.7% and acquisition rose 4.8%. Targeted beat matched random retention in all three fresh models.
The selected arm took 17.51 seconds online versus 15.61 seconds for random soft and 14.65 seconds without replay. Its ongoing verification forwarded an average 587,776 old tokens; teacher distributions occupied about 14.5 MiB. Teacher setup and common pretraining are additional. Thus 0.494% replay input is not 0.494% computation.
A five-seed earlier 77,568-parameter laboratory found 65.0% less forgetting versus low LR alone and 52.6% less versus same-budget random, preserving 99.2% of new learning. That tiny-context experiment is a separate scale and is not pooled with the three-model result.
The separate 555M native-checkpoint diagnostic gave old forgetting 0.124964 without replay, 0.095715 with random soft replay, and 0.112596 with targeted soft replay. Targeted lost to random by 17.6%, with nearly unchanged new learning. It used one seed, a fourteen-window bank, 65,536 new tokens, and only 320 replay tokens. That diagnostic is reported as a transfer limit, not a positive replication.
04 What does this mean?
What this suggests
- On three fresh 1.4M models, targeted soft replay reduced forgetting 10.4% versus matched random soft replay, with 1.1% more new learning.
- Soft targets, consistent scoring/training context, and slower shared updates form a useful local recipe. Random soft replay is a strong baseline and should not be replaced by a weak hard-label or no-replay comparison.
What we still don’t know
- Three models on one reconstructed corpus do not establish broad capability retention, factual accuracy, or a universal optimal replay policy.
- The native transfer check favored random; its full native checkpoint is outside this small-model archive. Downloads include every local v1/v2 base checkpoint and cached inputs, but no 99M or native-model weights.
Read the full discussion
Soft targets, consistent scoring/training context, and slower shared updates form a useful local recipe. Random soft replay is a strong baseline and should not be replaced by a weak hard-label or no-replay comparison.
The method saves old training inputs more clearly than it saves total computation. Cost accounting should include verification, teacher construction, selection, and replay updates.
05 Limitations
Small scratch-trained byte models on reconstructed text; a separate short native-checkpoint transfer favored random selection.
Read all study limitations
Three models on one reconstructed corpus do not establish broad capability retention, factual accuracy, or a universal optimal replay policy.
The native transfer check favored random; its full native checkpoint is outside this small-model archive. Downloads include every local v1/v2 base checkpoint and cached inputs, but no 99M or native-model weights.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.
The complete archive has 7 parts. Download every part into the same folder, then open complete-study.zip with an app that supports split ZIP archives. The article, source archive, and individual checkpoints can also be downloaded separately.