What different models learn in common is mostly simple

Start here

About 60 seconds

The idea, in a minute.

The question: Do independently trained models share a useful learning direction beyond simple token statistics?
What we found: Most early cross-model transfer came from vocabulary bias; the small contextual residual did not establish durable training savings.
Why it matters: A transferable signal may look like a general learning shortcut while mostly installing information ordinary training learns quickly.

Keep in mind

Small byte-level TinyStories transformers; eight held-out seeds per diagnostic and different transfer protocols are reported separately.

See the evidence

A concrete example

A head start that ordinary training catches

Recorded corpus-statistics persistence experiment; separate from the component figure.

01
Install token statistics
Unigram initialization reduces initial loss from 5.582 to 5.440.
02
Train twenty updates
The gap shrinks to 3.908 versus 3.898.
03
Train three hundred
The unigram condition reaches 2.188 versus 2.185 without initialization.

An immediate benefit is not the same as lasting acceleration.

Archived experiment reports and measurements

Read the formal abstract

Independently initialized models use different weight coordinates, but their outputs might change in common ways during learning. We measured that shared functional direction, removed simple components, and tested whether the remainder teaches fresh models.

The full first-update target improved held-out loss by 0.1779, while vocabulary bias alone improved it by 0.1699. Removing vocabulary, position, and current-token components left a consistent but small 0.0058 benefit. On unseen sequences, simple predictors explained over 92% of first-update shared energy; later residual transfer was negligible. Corpus-statistics initialization helped immediately but almost all of the advantage disappeared by twenty ordinary updates.

01 Research question

Do independently trained models share a useful learning direction beyond simple token statistics?

02 Experiment setup

The full procedure, model settings, datasets, and evaluation choices are recorded here.

Explore the complete methodology

Eight oracle seeds define shared logit-change targets; eight distinct held-out students 3000–3007 perform component mappings. Logits are unnormalized next-token scores. Probe A defines and maps targets; independent Probe B measures whether the induced functional change generalizes beyond the mapping probe.

The component study compares full shared changes, global vocabulary bias alone, and residuals after removing vocabulary, position means, and current-input-token effects. A smoothed empirical unigram target supplies a no-oracle baseline. Intervals 0→1 and 5→10 are kept distinct.

A separate shared-rule study fits input-conditioned global, position, and token predictors on separate training, validation, and test sequences. Eight new students test mapped coordinate transfer. This avoids treating a fixed probe tensor as a general update rule.

A third persistence experiment gives eight unseen students one matrix-free corpus-statistics update, then resumes ordinary training for 300 updates. None/unigram/position-unigram/bigram/trigram conditions share the ordinary-training comparison. Published downloads include the saved shared target tensors, probes, component CSVs, general-rule outcomes, persistence measurements, and actual code.

03 What happened?

Most early cross-model transfer came from vocabulary bias; the small contextual residual did not establish durable training savings.

Figure 01 / Observed resultsSubtract the simple explanation

0.0058Residual gain after three simple components are removed

0.20.10

0.1779

Full delta

0.1699

Vocabulary bias

0.0058

Residual

0.1155

Unigram prior

Validation loss improvement ↑ · scale shown from 0 to 0.2

How to read thisVocabulary bias captures most of the full 0.1779 improvement. The remaining effect is positive but small and does not demonstrate durable compute savings.

The limitSmall byte-level TinyStories transformers; eight held-out seeds per diagnostic and different transfer protocols are reported separately.

Component mapping at 0→1: eight held-out student seeds, separate mapping and functional-generalization probes. Improvements are loss differences, not percentages.

Full results and data table

Complete summary results
Shared target	Validation loss improvement ↑	Between-seed SD
Full delta	0.1779	0.0172
Vocabulary bias	0.1699	0.0176
Residual	0.0058	0.001
Unigram prior	0.1155	0.0168

First-update held-out validation improvements, mean ± between-seed SD, were full delta 0.1779±0.0172, vocabulary bias 0.1699±0.0176, residual after vocabulary 0.0102±0.0018, after position 0.0083±0.0014, and after current-token effects 0.0058±0.0010. Each was positive in all eight students. The corpus unigram prior improved loss by 0.1155±0.0168.

For the later 5→10 residual, validation improvement was only 0.0003±0.0002, positive in six of eight students. Independent Probe-B functional-change cosine was near zero. A residual can be statistically consistent without being a useful, probe-generalizable shortcut.

On unseen sequences, global/position/current-token predictors explained 92.6%/95.3%/94.1% of first-update shared energy, and 99.5%/99.6%/99.2% for 5→10. In that separate mapping protocol, the general current-token predictor improved validation by 0.0135 and its residual by 0.0005; both were positive in eight of eight students. These magnitudes must not be pooled with the fixed-probe component experiment.

The persistence study began at loss 5.582 without initialization and 5.440 with unigram or position-unigram initialization. After twenty updates these were 3.908 and 3.898; after 300 updates none/unigram/position-unigram/bigram/trigram losses were 2.185/2.188/2.181/2.183/2.181. There was no demonstrated persistent material training speedup.

04 What does this mean?

What this suggests

Most early cross-model transfer came from vocabulary bias; the small contextual residual did not establish durable training savings.
Shared functional learning exists, but the practically useful early signal is dominated by low-order token-frequency and bias structure. Decomposition and unseen-probe tests prevent an average direction from being mistaken for shared contextual intelligence.

What we still don’t know

Small models and one corpus do not rule out richer shared directions at other scales or training stages.
The mapping protocols differ, Probe-B transfer remained weak, and coarse checkpoint spacing limits estimates of training steps saved.

Read the full discussion

Shared functional learning exists, but the practically useful early signal is dominated by low-order token-frequency and bias structure. Decomposition and unseen-probe tests prevent an average direction from being mistaken for shared contextual intelligence.

Installing easy statistics can improve initialization without saving much subsequent compute. The relevant objective is persistent progress against ordinary training, not a striking one-step loss change.