Start here
About 60 secondsThe idea, in a minute.
- The question
- Can targeted replay search stay fast as memory grows without losing the retention benefit of full search?
- What we found
- A bounded shortlist made large-bank replay far faster, but exact shortlist repair still had about 23% higher old-behavior error than full search.
- Why it matters
- Memory retrieval can cost more than the training it saves. Search limits should be judged by retained behavior as well as latency.
A 1,033-parameter synthetic model; large-bank comparisons use two models. The million-key benchmark measures only warm index retrieval.
A concrete example
A fast shortlist misses useful repairs
Recorded large-bank aggregate comparison.
- 01
Search every memory
Full search cost 110.2 ms with protected MSE 0.01599.
- 02
Refine thirty-two candidates
Exact shortlist cost 3.97 ms with protected MSE 0.01974.
- 03
Interpret both outcomes
The update became much cheaper, but preservation degraded.
A speedup is valuable only at an acceptable retention cost.
Archived experiment reports and measurementsRead the formal abstract
A replay system cannot inspect every old example forever. We first replaced a learned independent selector with joint linearized repair scoring, then limited search through cached keys and a fixed shortlist.
The joint selector modestly improved both old and new error in four confirmation models. At 4,096 memories, exact shortlist scoring reduced amortized correction cost from 110.2 to 3.97 ms, while protected MSE increased from 0.01599 to 0.01974. A separate million-key benchmark achieved 0.20 ms median warm retrieval, but excluded model key generation and training. These are measured speed–quality tradeoffs, not evidence of billion-memory training performance.
01 Research question
Can targeted replay search stay fast as memory grows without losing the retention benefit of full search?
02 Experiment setup
The full procedure, model settings, datasets, and evaluation choices are recorded here.
Explore the complete methodology
Executed October 5, 2026. The inherited numerical model has 1,033 parameters, twelve contexts, independent two-step corrections at LR 0.02, four selected memories, and a ninety-six-memory starting bank. Each process has a 300-second watchdog; the longest completed scale interval took 79.6 seconds.
The linear_refined selector forecasts all candidate effects using an output Jacobian, chooses four memories jointly, and makes one round of single-memory replacements. It minimizes predicted training-bank error while penalizing support harm. It does not use final audit labels. Confirmation seeds 44–45 and 48–49 provide 256 fresh corrections, after development on 40–43.
Cached keys use 128-dimensional Jacobian sketches. Query work reads at most 512 index vectors, considers at most 128 memories, checks forty-eight fixed bank probes, and refines at most thirty-two candidates. The exact-shortlist variant recomputes precise effects for shortlisted candidates. The executed update needs exact gradients for only the four selected memories.
Each method calibrates an at-most-two-strength mixture on separate contexts 8–9 before final contexts 10–11. Scale comparisons include 96 and 4,096 memories. Memory-mapped files bind cached keys to model and code identities. A separate benchmark indexes one million synthetic keys; it does not train on one million memories.
03 What happened?
A bounded shortlist made large-bank replay far faster, but exact shortlist repair still had about 23% higher old-behavior error than full search.
Amortized ms/correction ↓ · scale shown from 0 to 120
How to read thisLower bars mean less correction time. The accompanying MSE column shows why speed alone is not sufficient: exact shortlist error remains about 23% higher.
The limitA 1,033-parameter synthetic model; large-bank comparisons use two models. The million-key benchmark measures only warm index retrieval.
4,096-memory comparison on seeds 70–71; setup spread over sixty-four uses. Displayed milliseconds and MSE retain their original units.
Full results and data table
| Selector | Amortized ms/correction ↓ | Protected MSE ↓ |
|---|---|---|
| Full bank | 110.2 | 0.01599 |
| Approximate | 2.98 | 0.02323 |
| Exact shortlist | 3.97 | 0.01974 |
At ninety-six memories, linear_refined versus learned repair had old MSE 0.023757 versus 0.024048 and new MSE 0.033887 versus 0.037511: average reductions of 1.2% and 9.7%. Online cost was 1.592 versus 1.139 ms. Old error improved in two of four models, so this is not a consistent per-model win.
For 4,096-memory seeds 70–71, full-bank / cached approximate / exact shortlist protected MSE was 0.01599 / 0.02323 / 0.01974, with new-rule MSE 0.03203 / 0.03231 / 0.03248. Setup-amortized correction times were 110.2 / 2.98 / 3.97 ms. The often-quoted roughly 42-fold gain refers to online exact-shortlist timing before setup amortization; the displayed amortized ratio is about 27.8-fold.
The million-key synthetic index built in 1.10 seconds and, after disk-cache warming, retrieved in 0.20 ms median and 0.23 ms at the 95th percentile. At most 512 vectors were read and 486 scored. This excludes key construction from a model, forward/backward computation, and replay training.
Three early disk-cache attempts failed during setup because of row-offset, temporary-path, and precision issues; they produced no accepted training results. Corrected memory-mapped runs completed on seeds 72–74. A billion-entry bank was not built or queried.
04 What does this mean?
What this suggests
- A bounded shortlist made large-bank replay far faster, but exact shortlist repair still had about 23% higher old-behavior error than full search.
- Bounding query work solves a real search-cost problem, but it can discard memories that matter for retention. The correct comparison is a frontier between error, setup cost, query cost, and refresh frequency.
What we still don’t know
- No sequential model drift or large language model was tested; two-model bank-scale screens are exploratory.
- Million-key warm retrieval is not end-to-end training or a billion-memory latency estimate. The method may lose retention when the shortlist misses useful repairs.
Read the full discussion
Bounding query work solves a real search-cost problem, but it can discard memories that matter for retention. The correct comparison is a frontier between error, setup cost, query cost, and refresh frequency.
Fast retrieval on synthetic cached keys cannot establish model-scale training efficiency. Cache staleness and rebuilding after model changes are unresolved costs in sequential deployment.
05 Limitations
A 1,033-parameter synthetic model; large-bank comparisons use two models. The million-key benchmark measures only warm index retrieval.
Read all study limitations
No sequential model drift or large language model was tested; two-model bank-scale screens are exploratory.
Million-key warm retrieval is not end-to-end training or a billion-memory latency estimate. The method may lose retention when the shortlist misses useful repairs.
06 References & materials
Keep the complete record
Take the research with you.
The full write-up, source code, and recorded experiment materials.
Download everything together or choose individual files.