# Why an exact gradient trick failed to replace real training data A linear-layer inversion works almost perfectly; reconstructing useful updates through a whole transformer remains lossy. Author: Yehoyada Shtinmetz Date: 2026-10-18 Version: 1.0 ## The idea, in a minute ### The question Can a trained model provide synthetic inputs that teach a fresh model faster than ordinary data? ### What we found Frozen synthetic embeddings beat Gaussian noise, but real TinyStories updates remained substantially better at matched processed positions. ### Why it matters An exact algebraic result at one layer can hide a difficult reconstruction problem across an entire network. ### Keep in mind Two held-out student seeds in the controlled pilot; synthetic embeddings are not ordinary text and do not establish foundation-model speedups. ## A concrete example: Accuracy vanishes as the inverse reaches the input Recorded known-realizable-gradient diagnostic, separate from student-transfer endpoints. ### Inject at the output head The head-gradient cosine reaches 0.996 using a controlled adjoint. ### Require valid raw inputs Valid soft targets at raw embeddings yield full-gradient cosine about 0.104. ### Test actual learning The controlled student loss remains worse for synthetic embeddings than real data. A local reconstruction check is necessary but not enough. Evidence: [Archived experiment reports and measurements](/downloads/oracle-gradient-inversion-meets-the-transformer/evidence.zip) ## Reading the main result 4.793 vs 3.884 — Synthetic versus real-data validation loss Synthetic embeddings help compared with noise but remain worse than ordinary data. The exact linear-layer result does not imply an exact network-level inverse. Two held-out student seeds in the controlled pilot; synthetic embeddings are not ordinary text and do not establish foundation-model speedups. ## What this suggests - Frozen synthetic embeddings beat Gaussian noise, but real TinyStories updates remained substantially better at matched processed positions. - The failure is not explained solely by choosing an impossible desired gradient. Even a genuine language-model gradient became poorly reconstructed when forced through valid raw inputs and output probabilities. ## What we still don’t know - Two held-out students provide a pilot, not a statistically powered speedup claim. Byte-level TinyStories models are far from pretrained foundation models. - Processed positions are matched, but synthetic construction has additional costs and the embeddings do not correspond to ordinary printable training text. ## Abstract If a trained model knows where learning should go, perhaps a new model can skip some rediscovery. We tested a direct, low-rank inversion from desired weight updates to synthetic embedding and output-target pairs, without optimizing those synthetic inputs in an outer loop. The linear-layer reconstruction passed, and the output-head gradient could be reconstructed with cosine 0.996 at direct injection. But valid soft targets at raw embeddings reproduced known-realizable full gradients with cosine only about 0.10. In the corrected, fixed-validation pilot, twenty matched updates gave held-out losses 3.884 for real data, 4.793 for frozen synthetic embeddings, and 5.584 for Gaussian inputs. The promising algebra did not survive the transformer reconstruction bottleneck well enough to replace ordinary training. ## Research question Can a trained model provide synthetic inputs that teach a fresh model faster than ordinary data? ## Experiment setup The controlled pilot uses TinyStories with a shared 256-token byte vocabulary, four causal transformer blocks, width 144, four attention heads, and MLP width 576. The saved controlled-run summary records 1,078,560 parameters; this actual artifact supersedes a different default parameter count in the general README. Stage A verifies exact gradient factorization for a linear map. Later stages use low-rank factors of oracle directions to construct activation/adjoint pairs, then map them toward valid synthetic inputs and probability targets. Synthetic construction uses seeds 101–102; held-out student seeds 201–202 do not participate in construction or selection. The corrected comparison uses one fixed 512-sequence validation set. Every update processes sixteen sequences of forty-eight positions. Twenty updates therefore process 15,360 positions per condition. Conditions are ordinary TinyStories, frozen constructed embeddings, and Gaussian-embedding controls. Corrected output-target naming distinguishes soft/hard targets from input tokens. The oracle trains to step 300, with checkpoints at 0, 50, 150, and 300. Complementary known-realizable-gradient diagnostics use three separate students 301–303 and measure reconstruction at multiple injection depths. All controlled-pilot checkpoint and synthetic-dataset files are included alongside the diagnostic CSVs. ## Results Stage A cosine was 1.0 with relative error 3.95e-7. The oracle reached fixed-validation cross-entropy 2.1696 at step 300. These successful checks do not establish student transfer. At twenty matched updates, controlled held-out validation cross-entropy averaged 3.884 for real data, 4.793 for frozen synthetic embeddings, and 5.584 for Gaussian controls. Lower is better. Earlier pilot values used a different comparison and are preserved as historical rather than pooled into this endpoint. For known-realizable gradients, controlled head injection achieved head cosine 0.996, but full-gradient cosine only 0.353. Before final LayerNorm these were 0.910/0.322. At raw embeddings with valid soft targets they fell to 0.194/0.104. Oracle-direction raw-embedding full cosine was lower still, about 0.016. Trajectory-span diagnostics separately show that short actual optimizer trajectories are more aligned with ordinary-gradient spans than a raw endpoint-weight direction. This helps reject the idea that a weight difference should automatically be treated as an ideal gradient; it does not repair the synthetic-input reconstruction. ## Discussion The failure is not explained solely by choosing an impossible desired gradient. Even a genuine language-model gradient became poorly reconstructed when forced through valid raw inputs and output probabilities. Useful synthetic transfer must overcome activation realizability, target constraints, and cross-layer distortion. Beating Gaussian noise is a weak baseline; the relevant efficiency comparison is ordinary data under an explicit compute and exposure budget. ## Limitations Two held-out students provide a pilot, not a statistically powered speedup claim. Byte-level TinyStories models are far from pretrained foundation models. Processed positions are matched, but synthetic construction has additional costs and the embeddings do not correspond to ordinary printable training text. ## Summary results | Training input | Validation cross-entropy ↓ | Matched updates | | --- | --- | --- | | Real data | 3.884 | 20 | | Synthetic | 4.793 | 20 | | Gaussian | 5.584 | 20 | Controlled rerun: two held-out seeds, one fixed 512-sequence validation set, and 15,360 processed positions per twenty-update arm. ## References - [Original reports, protocols, and saved measurements](/downloads/oracle-gradient-inversion-meets-the-transformer/evidence.zip)