# Small updates, useful arithmetic: selective adaptation in Qwen3-0.6B A controlled Mac study of full fine-tuning, single-layer updates, and low-rank adapters, followed by causal removal and compression tests. Author: yoyo Publication date: 2026-10-18 Version: 1.0 ## Takeaway Low-rank attention updates adapted this small arithmetic task with much less update cost, while gradient magnitude failed to identify a uniquely useful layer. ## The idea, in a minute ### The question How much of a pretrained language model needs to change to improve a narrow skill? ### What we found Rank-8 query/value adapters averaged 81.5% addition accuracy against 76.3% for full fine-tuning, with approximately 13.5 times faster timed updates on the tested Mac implementation. ### Why it matters Cheap adaptation can come from changing a small part of a model, but the useful update location must be tested rather than inferred from large gradients. ### Keep in mind Three training seeds, one shared 128-question test set, a narrow addition distribution, and float32 Apple MPS do not establish general training or pretraining savings. ## A concrete example: Compressing an already learned addition update A recorded diagnostic on the replayed seed-101 adapter, after the addition evaluation set had already been used. ### Before adaptation The original model answered 32 of 128 addition questions correctly. ### Train rank-8 adapters The replayed query/value adapters reached 103 of 128. ### Compress each learned matrix to rank one The posthoc compressed updates reached 104 of 128, retaining 62.7% of summed squared update-matrix norm. A compact representation of one successful update is possible; this does not show that directly training rank one will find the same solution. ## Reading the main result 13.46× — Lower timed update cost for LoRA LoRA averaged 1.738 seconds versus 23.394 seconds for full tuning, while mean accuracy was 81.51% versus 76.30% across three training seeds. This ratio excludes loading and evaluation and applies to one float32 MPS implementation. Accuracy seed ranges overlap; the study does not establish general superiority. ## What this suggests - Parameter-efficient adaptation deserves further controlled testing on this checkpoint. - The high-gradient early MLP was not consistently better than a middle-layer control. - Update-state size and actual backward-pass time must be measured separately. ## What remains unknown - Whether the advantage persists with optimized GPU kernels, longer sequences, broader tasks, or equal end-to-end time. - Whether direct low-rank training can consistently match a higher-rank solution compressed afterward. ## Abstract Can a small learned change unlock a useful skill without paying to update an entire language model? In this experiment, low-rank attention adapters improved a pretrained Qwen3-0.6B checkpoint on three-digit addition while updating about 0.192% of its unique parameters. Their mean accuracy exceeded the tuned full-model baseline, and synchronized update time was much smaller on the tested Mac backend. The more surprising qualification came from analyzing where updates matter. The early MLP with the largest raw gradient energy was not uniquely effective, and compressing one trained adapter solution to rank one preserved its accuracy. Direct rank-one training on a fresh mixed-operation task was weaker. Efficient adaptation therefore depends on the learning path and task as well as the final size of the update. ## Research question Can selective or low-rank updates improve Qwen3-0.6B arithmetic adaptation at lower measured update cost, and do gradient profiles predict the best update location? ## Experiment setup Experiments were conducted on 2026-09-08. The original post-trained Qwen/Qwen3-0.6B revision was c1899de289a04d12100db370d81485cdf75e47ca, with 596,049,920 unique loaded parameters. Shared input/output embeddings were counted once. Runs used float32 Apple MPS, eager attention, PyTorch 2.13.0, Transformers 5.16.1, and non-fused, non-foreach AdamW. The original checkpoint must be obtained separately from its upstream publisher. An initial supervised-gradient diagnostic used twelve explicit addition, geography, and translation examples with answer-only cross-entropy. The block-2 MLP had the largest average parameter-group gradient energy across all three tasks. This motivated an actual intervention comparison rather than treating gradient energy as proof of useful learning. The development screen compared full-model tuning, the last seven decoder blocks, block-2 MLP, block-14 MLP, and rank-8 query/value LoRA, with three learning-rate trials per arm. Each run used 64 training examples, 32 development examples, and sixteen batch-four updates. Block indices are zero-based. Selected rates were respectively 3e-6, 3e-6, 1e-4, 1e-4, and 3e-4. The locked replication ran all five configurations at seeds 101–103 with fresh 64-example training sets and one shared independent 128-example evaluation set. Unordered operand pairs were disjoint across screen, replication training, and evaluation sets; digit patterns could overlap. Prompts had the form 123 + 456 = followed by a three-token answer. Only answer-token loss was optimized. Numerical audits checked labels, loss alignment, baseline equivalence, initial adapter equivalence, frozen-gradient exclusion, and data separation. Follow-up work replayed one adapter-training run and evaluated 38 removal, reconstruction, compression, and restoration conditions. A separate fifteen-run development screen tested five adapter configurations at three rates each on fresh mixed addition/subtraction examples. This later screen used seed 401 and 64 training plus 64 development examples; it was not an independent multi-seed confirmation. ## Results The common baseline scored 32/128, or 25.0%, with mean answer-token cross-entropy 1.265. Across three training seeds, full tuning averaged 76.30% accuracy, suffix tuning 46.09%, block-2 MLP 74.48%, block-14 MLP 75.78%, and rank-8 query/value LoRA 81.51%. Observed accuracy ranges were 73.4–81.2%, 45.3–47.7%, 62.5–80.5%, 70.3–79.7%, and 78.9–85.2%, respectively. These are seed ranges, not confidence intervals. Mean synchronized update totals were 23.394 seconds for full tuning and 1.738 seconds for LoRA, a 13.46-fold ratio. LoRA trained 1,146,880 parameters and used about 9.2 MB of optimizer tensors, compared with 4,768.4 MB for full tuning. The frozen backbone and activation gradients still cost memory and computation. Loading, evaluation, and other workflow costs were excluded from these update timings. In the seed-101 replay, intact adapters scored 103/128. Removing all query updates scored 104/128; removing all value updates scored 79/128. Rank-one posthoc compression scored 104/128. Replay, zero-adapter, full-rank reconstruction, and restoration controls passed. No individual layer-group removal increased cross-entropy by more than 0.00364, consistent with distributed interacting changes rather than a single localized arithmetic circuit. On the fresh mixed-operation development screen, selected query/value rank-8 adapters answered 24/32 additions and 17/32 subtractions; query/value rank one answered 23 and 11. Block-14 MLP rank eight answered 26 and 13. Lower rank reduced state size substantially but barely changed update time. Selecting rates on this same development set prevents treating these figures as independent test winners. All replication arms retained four geography and four copying answers in each seed. Those eight prompts, plus four subtraction checks, are only retention diagnostics; they do not establish preservation of broad capabilities. ## Discussion The experiment reproduces the practical value of low-rank adaptation in a specific model, task, and backend. It does not introduce LoRA as a new method. Its useful finding is that a narrow skill improved through a small update subspace, while the initially dominant gradient location did not uniquely predict the most reliable intervention. Posthoc compression and direct low-rank training answer different questions. Compressing a jointly optimized solution shows redundancy in that solution; it does not prove that a restricted optimizer can reach it. Here the follow-up also changed the task distribution, so the contrast cannot isolate optimization difficulty from task differences. A controlled same-task comparison remains open. ## Limitations The shared 128-question test set supplies 128 distinct questions, not 384 independent observations. Three training seeds and a modest rate search cannot establish a statistically significant or universally optimal winner. This is adaptation of an existing pretrained model, not from-scratch learning. Improvement could reflect better access to existing arithmetic machinery. No pretraining savings, equal-wall-time superiority, optimized GPU speedup, or comprehensive capability-retention claim follows. The downloadable trained adapter is the inspected seed-101 replay, not a full model or checkpoints for all replication runs. Per-run scripts, datasets, summaries, intervention outputs, and the adapter hash manifest are included. Full tuned model weights were not retained; reproduce those conditions by rerunning the saved scripts and data with the pinned upstream base. ## Summary results | Update strategy | Accuracy (%) | Update seconds | | --- | --- | --- | | Full model | 76.30208333333334 | 23.393943818945747 | | Last seven blocks | 46.09375 | 4.857854001359859 | | Block 2 MLP | 74.47916666666666 | 1.8611364313401282 | | Block 14 MLP | 75.78125 | 1.4873812660419692 | | Rank-8 Q/V LoRA | 81.51041666666666 | 1.7382574739943568 | Mean accuracy on the same 128 questions across three independent training seeds per arm. Third column is mean synchronized update total in seconds on float32 Apple MPS; loading and evaluation are excluded. Seed ranges and task limitations are detailed in the text. ## Available materials Includes the actual 4,598,968-byte seed-101 trained adapter and its SHA-256 manifest. The original Qwen base must be obtained separately; full tuned checkpoints for the other runs were not retained. The evidence archive preserves selected reports and result records. Source paths and SHA-256 hashes are listed in provenance.json. This publication rechecked recorded outputs and archive integrity; no research experiment was rerun. ## References - [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) - [Pinned original Qwen3-0.6B checkpoint](https://huggingface.co/Qwen/Qwen3-0.6B/tree/c1899de289a04d12100db370d81485cdf75e47ca)