Finding vulnerable memories is not the same as measuring forgetting

Start here

About 60 seconds

The idea, in a minute.

The question: Can targeting likely damaged examples make preservation checks both more informative and more accurate?
What we found: Indexed checks captured about four times more damage than random checks, yet their average-damage estimation error was 102.7% higher.
Why it matters: A test set selected for vulnerability can help find failures while giving a poor picture of how much the whole model changed.

Keep in mind

Five synthetic transformer models; the regret improvement was uncertain and indexing cost more than checking the entire small bank.

See the evidence

A concrete example

A better failure finder, a worse average

Recorded comparison of the same check budget.

01
Find damage
Indexed checks covered 26.6% of positive damage versus 6.5% for random checks.
02
Estimate the average
Despite weighting the sample, estimation RMSE rose from 0.003831 to 0.007766.
03
Check the cost
On this small bank, scanning all examples was cheaper than maintaining the index.

The right measurement depends on the question being asked.

Archived experiment reports and measurements

Read the formal abstract

Finding the worst failures and estimating typical harm are different statistical tasks. This study makes that distinction concrete by selecting preservation checks from a sensitivity index while retaining a weighted random complement.

Across five models and thirty-two corrections per model, the index captured 26.6% of positive damage versus 6.5% for random checks. Yet mean-damage estimation RMSE increased by 102.7%, and the 25.7% reduction in update-choice regret did not have a seed-bootstrap interval excluding zero. The main progression gate failed. The index located vulnerable examples, but did not deliver a dependable or cheaper reward signal.

01 Research question

Can targeting likely damaged examples make preservation checks both more informative and more accurate?

02 Experiment setup

The full procedure, model settings, datasets, and evaluation choices are recorded here.

Explore the complete methodology

Executed September 18, 2026 with a 39,121-parameter transformer and the twelve-material conductivity/circuit world. Final independent seeds are 40–44, with development seed 800. A bank has 512 protected examples and a separate audit has 256. Final trajectories contain thirty-two corrections per model.

Each common trajectory commits the 0.1 learning-rate action while all methods counterfactually evaluate five SGD steps at LR 0, 0.03, 0.1, or 0.3. Keeping trajectories common isolates measurement quality; the secondary learned policies are offline contextual-bandit evaluations, not live continual-learning policies.

The index stores projected loss gradients in 256 dimensions, refreshed every eight updates. It picks twenty-four highest predicted-damage examples and eight random examples from the complement. The population estimate weights the random complement by its known size; damage is clipped per example before averaging. Final audit examples never enter selection.

Comparators include random checks, hidden-representation similarity, stale keys, full-gradient diagnostic keys, and all 512 bank items. Reward combines new-validation error and five times estimated positive damage. Seed-cluster summaries and full selection/refresh costs are archived.

03 What happened?

Indexed checks captured about four times more damage than random checks, yet their average-damage estimation error was 102.7% higher.

Figure 01 / Observed resultsFinding the failures

4.1× coverageDamage found by indexed versus random checks

100500

6.491

Random

26.605

Indexed

Positive damage covered (%) ↑ · scale shown from 0 to 100

How to read thisThe index finds far more damaged examples. That does not make its estimate of average population damage accurate; the latter error more than doubled.

The limitFive synthetic transformer models; the regret improvement was uncertain and indexing cost more than checking the entire small bank.

Five model means over thirty-two sequential corrections each. Coverage is measured on the largest proposed update; all units are percentages.

Full results and data table

Complete summary results
Checks	Positive damage covered (%) ↑	Top-32 recall (%) ↑
Random	6.49097	6.28906
Indexed	26.60511	25.07813

Random versus projected-index damage coverage was 6.49% versus 26.61%; top-thirty-two damaged-example recall was 6.29% versus 25.08%. Those are selection-recall measurements, not global retention estimates.

Mean-damage estimation RMSE was 0.003831 for random and 0.007766 for the index. Update-choice regret fell from 0.000245 to 0.000182, but improved in only three of five models and its paired seed-bootstrap interval included zero. Hidden-similarity selection had lower mean regret, about 0.000054.

Independent protected MSE was 0.011112 for random and 0.011268 for the index; new-consequence MSE was 0.009874 and 0.009605 respectively. The modest acquisition difference does not rescue the failed estimation and replication gates.

Index selection and checks, including construction and refresh, took about 0.55 seconds per model stream versus about 0.08 seconds for full-bank checking. The time-matched random allocation therefore checked the entire bank. Secondary RL controllers mostly declined to update.

04 What does this mean?

What this suggests

Indexed checks captured about four times more damage than random checks, yet their average-damage estimation error was 102.7% higher.
Selection for failure discovery should not be confused with a representative evaluation of average damage. An unbiased estimator can still have high variance when only eight examples represent a large unsampled complement.

What we still don’t know

The bank covers a designed world, not all model capabilities; preserved bank behavior cannot certify universal retention.
Five models provide limited uncertainty estimates. The small bank favors batched full scans, and offline policies face reset-to-sequential distribution shift.

Read the full discussion

Selection for failure discovery should not be confused with a representative evaluation of average damage. An unbiased estimator can still have high variance when only eight examples represent a large unsampled complement.

A useful failure index might serve debugging or prioritization even when it is unsuitable as a population-damage estimator. The intended role should determine the benchmark.

05 Limitations

Five synthetic transformer models; the regret improvement was uncertain and indexing cost more than checking the entire small bank.

Read all study limitations

The bank covers a designed world, not all model capabilities; preserved bank behavior cannot certify universal retention.

Five models provide limited uncertainty estimates. The small bank favors batched full scans, and offline policies face reset-to-sequential distribution shift.