An independent research journal
Questions, experiments,
and what follows.
A growing collection of research, documented from the first question
to the full experiment. Open methods. Complete results.
The research
21 studies in the stash21 studies found
Machine learning / Research
Learning from reasoning mistakes with 0.1% of the weights
An exploratory Qwen3 study of self-correction, sparse training, and the gap between solving a mistake and learning a skill.
Key findingThe sparse training pipeline worked, but its measured gain was just one extra correct answer on 24 held-out questions.
Program learning / Research
When Compression Finds Programs That Generalize
Discrete searches recover compact ownership, containment, XOR, and parity rules that generalize beyond training examples, while showing how strongly outcomes depend on a supplied grammar.
Key findingCompression-guided search found reusable rules for ownership, nested containment, XOR, and running parity, but the representation and program grammar were supplied in advance.
Optimization / Research
Repeated Consensus Keeps a Small Optimizer Population Exploring
A frozen multi-seed Rastrigin study tests repeated explore-and-consolidate updates against independent runs and one-shot consensus.
Key findingOn Rastrigin, cyclic elite consensus reached the exact optimum in 98.54% of 10D groups and 53.93% at 20D, but fell to 3.50% at 30D.
Expert Routing and Systems / Research
Unrestricted Routing with Bounded Expert Paging Preserves E36 Quality
A failed hard prompt plan led to a bounded LRU paging design that recovered unrestricted routing while retaining the E36 model’s quality.
Key findingOn a one-gigabyte E36 expert store, unrestricted routing with a 12-page-per-pool LRU cache matched resident quality while reducing peak device allocation by 73.7%.
Optimization / Research
Can a Model Stop Training When Its Behavior Has the Right Shape?
A 90-episode parity study compares an unlabeled counterfactual-geometry stopping rule with fixed training and validation-based stopping.
Key findingAcross 90 small parity tasks, an unlabeled geometry rule kept the success rate unchanged while cutting conservatively counted total work by 33.8% versus fixed training.
Neural architectures / Research
Headwise Depth Routing: Quality Gains, Measurable Cost
Paired WikiText proxies show small quality gains from headwise routing, while eager runtime costs persist and the longer five-task RunPod screen misses efficiency gates.
Key findingHeadwise routing improved paired WikiText loss at depths 8 and 12, but raised eager inference latency about 1.30x; in the longer RunPod screen, quality gains on Dyck-2 and TinyStories came with failed latency guards and additional Dyck costs.
Expert Routing / Research
E64 Improves Tiny-Proxy Quality, but Route Concentration Blocks Bank Validation
Three paired seeds show a modest E64 quality gain over E16, while concentrated routing prevents validating useful bank scaling.
Key findingE64 lowered mean NLL by 3.62% over E16 across three seeds, but all six routed layers failed exposure gates; a 64-domain follow-up also failed quality, assignment, and cost thresholds.
Neural architectures / Research
Hard Neural Fabrics: Structured Success, Language Failure
A primitive-only routed recurrent fabric learns running parity across three seeds, but fails to find a useful input route on byte-level WikiText-2.
Key findingHard routing learned the parity circuit in three seeds, but direct transfer to WikiText-2 collapsed to dominant-byte prediction and ran about twelve times slower than a matched GRU.
Training efficiency / Research
Small updates, useful arithmetic: selective adaptation in Qwen3-0.6B
A controlled Mac study of full fine-tuning, single-layer updates, and low-rank adapters, followed by causal removal and compression tests.
Key findingLow-rank attention updates adapted this small arithmetic task with much less update cost, while gradient magnitude failed to identify a uniquely useful layer.
MoE Evaluation / Research
A Small MoE Proxy Finds Useful Routing but Fails Architecture Rank Validation
V3 separates routing effects and tracks a WikiText-2 anchor, yet its architecture ordering does not predict the existing 86M comparison.
Key findingV3 detects causal routing behavior and small-scale WikiText gains, but its four-architecture ranking has zero rank correlation with the available 86M suite.
Optimization / Research
When Do Shared Training Trajectories Help a New Model?
A low-rank trajectory controller saves online work on parity models, fails zero-shot on nested majority, then recovers a smaller gain after multi-family calibration.
Key findingA trajectory denoiser cut accounted online work 8.55% on held-out parity and 2.02% after nested-majority meta-training, while its frozen parity-only transfer failed.
Machine learning / Research
Can we train a neural network to be easier to correct?
A small transformer learns to propagate corrections while damaging fewer unrelated predictions.
Key findingTraining for editability reduced consequence error by 49.5% and unrelated-prediction error by 62.1% versus ordinary training.
Machine learning / Research
The best replay memories may be useful together, not alone
A controlled study separates individual repair scores from the value of complementary memories.
Key findingDirect complementary selection reduced old-behavior error by 12.4% versus random replay, while the learned selector failed to recover that advantage.
Machine learning / Research
Fast targeted replay exposes a speed–retention tradeoff
Joint repair scoring and cached shortlists make replay retrieval faster, but larger banks reveal a cost in preservation.
Key findingA bounded shortlist made large-bank replay far faster, but exact shortlist repair still had about 23% higher old-behavior error than full search.
Machine learning / Research
Finding vulnerable memories is not the same as measuring forgetting
An index finds more damaged examples while producing a worse estimate of average damage.
Key findingIndexed checks captured about four times more damage than random checks, yet their average-damage estimation error was 102.7% higher.
Machine learning / Research
A model could learn corrections; its RL controller still lost
Sequential edit training makes useful updates possible, but a learned policy fails to improve on a fixed recipe.
Key findingThe RL controller acquired slightly more corrections but had 41% higher consequence error than fixed updates.
Machine learning / Research
An AI learned to predict its own changes—but chose the same updates
Action-specific self-model forecasts improve, while update control collapses to a simple learning-rate choice.
Key findingA learned self-model predicted changes more accurately, but both controllers chose the same half-strength update on every step.
Machine learning / Research
More model capacity did not make a certainty budget more useful
A paired MESU pilot separates larger-model retention from the benefit of capping total certainty.
Key findingThe 20× certainty cap improved recent retention by 9.97 points in the narrow model and just 0.22 points in the wide model.
Machine learning / Research
Why an exact gradient trick failed to replace real training data
A linear-layer inversion works almost perfectly; reconstructing useful updates through a whole transformer remains lossy.
Key findingFrozen synthetic embeddings beat Gaussian noise, but real TinyStories updates remained substantially better at matched processed positions.
Machine learning / Research
What different models learn in common is mostly simple
Cross-seed functional transfer survives, but vocabulary bias explains most of its useful early effect.
Key findingMost early cross-model transfer came from vocabulary bias; the small contextual residual did not establish durable training savings.
Machine learning / Research
A little replay helps—but its data budget hides its compute cost
Small real-text models benefit from slow shared updates and targeted soft replay, with a failed native transfer check.
Key findingOn three fresh 1.4M models, targeted soft replay reduced forgetting 10.4% versus matched random soft replay, with 1.1% more new learning.
More questions to come.
This is a living journal. Each study adds another piece to the picture.