# Unrestricted Routing with Bounded Expert Paging Preserves E36 Quality A failed hard prompt plan led to a bounded LRU paging design that recovered unrestricted routing while retaining the E36 model’s quality. Author: yoyo Publication date: 2026-10-18 Version: 1.0 ## Takeaway On a one-gigabyte E36 expert store, unrestricted routing with a 12-page-per-pool LRU cache matched resident quality while reducing peak device allocation by 73.7%. ## The idea, in a minute ### The question Can a prompt-derived expert plan safely cap memory without blocking experts needed later in generation? ### What we found Yes, as an optional hint: demand-paged fallback preserved all aggregate and category scores on the frozen 60-case suite. ### Why it matters A cache can bound working memory while ordinary routing retains access to the full expert bank. ### Keep in mind This validates one roughly 1 GB model; filesystem reads were cacheable, and this does not establish quality gains from much larger banks. ## A concrete example: The E36 decode evaluation In the final engineering comparison on the unchanged 60-case suite, the native-FP16 paged model was compared with the same resident teacher. ### Resident control 34/60 cases were fully correct and 202/228 checks passed; mean decode was 13.11 ms and peak device allocation 2.296 GB. ### Bounded cache Only 12 of 36 expert pages per pool could reside on device. Ordinary product-key routes remained unrestricted; misses loaded a page and evicted the least recently used page. ### Observed outcome The paged run also achieved 34/60 and 202/228, with identical category scores and no changed passing response. Mean decode was 20.91 ms, P99 34.80 ms, and peak allocation 0.603 GB. The memory bound did not require the prompt planner to predict every future expert. ## Reading the main result 73.74% — lower peak device allocation The paged mode matched the resident arm at 34/60 fully correct cases while reducing allocation from 2,295,897,088 to 602,836,992 bytes. One model and one 60-case evaluation; storage timing was cacheable and the large checkpoint is not locally archived. ## What this suggests - A bounded LRU demand cache can preserve correctness for this E36 inference path while substantially reducing peak device allocation. - Prompt-derived plans can remain telemetry or an optimization hint without becoming a routing restriction. ## What remains unknown - Whether the same quality and latency hold for larger cores or hundreds of gigabytes of expert weights. - Cold sustained storage behavior in this end-to-end E36 run, because the expert-store file was cacheable. ## Abstract Can a prompt planner safely choose a bounded set of experts for an entire response? An E36 hard-plan attempt showed why that is risky: blocking routes outside the selected communities erased task quality. We repaired the contract to preserve ordinary prompt routing and later tested a stronger fallback: keep the prompt plan nonbinding, permit normal product-key routing, and load missing expert pages through a bounded least-recently-used cache. On an unchanged 60-case suite, 12 resident pages out of 36 per pool matched the resident model’s 34 fully correct cases, 202 of 228 checks, and every category score. Peak device allocation fell from 2.296 GB to 0.603 GB, while mean and P99 decode were 20.91 ms and 34.80 ms. This is an engineering result for a one-gigabyte store, not evidence that much larger expert banks improve model quality. ## Research question Does a nonbinding prompt hint plus unrestricted product-key routing and bounded LRU demand paging preserve E36 quality while reducing device memory? ## Experiment setup The sequence of tests began with a hard E36 prompt plan that excluded every expert outside two planned communities. That approach failed quality and motivated a contract repair and then demand paging. The final matched engineering comparison used the unchanged seed-101 E36 teacher, 538,820,352 stored parameters, approximately 86.1 million active parameter applications per token, and an expert store of 1,018,552,320 native-FP16 bytes. The three physical pools contained 36 pages each; the cache held 12 pages per pool. Both resident and paged arms used identical converted weights, greedy generation, ordinary product-key top-1 routing during prefill and decode, and the same 60-case DGX Spark public development suite. The last-prefill-token top-two-community plan was recorded but never enforced. DGX work was admitted through the idle-GPU guard. No research code was executed while preparing this article. ## Results The resident arm passed 34/60 cases and 202/228 checks, with 13.11 ms mean and 15.29 ms P99 decode latency and 2.296 GB peak allocated device memory. The paged arm passed the same 34 cases and 202 checks, and the aggregate and category breakdowns were identical. Two already-failing responses differed numerically; none of the passing responses changed. Paging raised mean decode to 20.91 ms and P99 to 34.80 ms, corresponding to 47.81 tokens/s from mean decode time. Peak device allocation was 602,836,992 bytes, a 73.74% reduction. The system performed 24,444 page loads and 24,408 evictions while honoring the 12-page limit per pool. The nonbinding hint matched 39,262 of 71,652 decode routes (54.80%). Demand paging handled the remaining routes. Charging all prefill and decode reads to generated tokens gives 39.13 MB/token and 782.7 MB/s at 20 tokens/s. That is 3.28 times below the separate 2.57 GB/s no-cache storage measurement and 5.79 times below measured decode/upload throughput. These comparisons screen traffic feasibility; the E36 filesystem reads themselves were cacheable. ## Discussion The decisive change was making planning advisory. A plan estimated likely expert use but did not determine which experts were legal to execute. A cache miss triggered loading and LRU eviction, preserving the model’s original router choices while constraining the resident working set. The test therefore supports a systems mechanism: memory can be bounded independently of the total bank size without imposing a hard routing constraint. The path to this result matters. The initial full-prompt hard-plan conversion scored 0/60 and 0/228. A decode-only contract repair correctly kept ordinary routes during prompt processing, but the hard decode plan still scored 0/60 and 0/228. It failed dense-quality and utilization gates. The paging result does not reverse those failures; it avoids the premise that the plan must be authoritative. The result promotes this combination for continued engineering at E36. The prompt plan was recorded, not physically prefetched, in the final demand-only run. The benefit demonstrated is bounded allocation with quality matching; the data do not establish a quality improvement from adding stored experts. ## Limitations The evaluation contains 60 task cases and uses one trained seed-101 E36 model. It is an engineering screen, not a multi-seed quality study or a scaling law. The local store timing was not cold-SSD evidence because reads could be served from filesystem cache. Storage headroom uses an earlier no-cache physical probe; production performance will also depend on decoding, upload, overlap, OS caching, and sustained device conditions. The complete E36 model and optimizer checkpoint remain on the DGX, about 6.1 GB total, and are not in this checkout. Evaluation JSON and decision artifacts are present. They are evidence, not trained model weights. ## Summary results | Mode | Fully correct cases | Peak device bytes | | --- | --- | --- | | Resident | 34 | 2295897088 | | Paged LRU (12/36) | 34 | 602836992 | Same E36 checkpoint and frozen 60-case suite. First measure is fully correct cases out of 60; second is peak allocated device memory in bytes. ## Available materials Reproduction inputs include the shared baseline, proxy, evaluation and metrics packages, E36 configs, guarded runner, and exact frozen public suite. The complete E36 teacher checkpoint is documented as retained only on DGX (model plus optimizer about 6.1 GB); it is not present locally. Evaluation payloads/results are evidence, not trained weights. The separate E64 physical page replay supports storage context only, not the E36 checkpoint. The evidence archive preserves selected reports and result records. Source paths and SHA-256 hashes are listed in provenance.json. This publication rechecked recorded outputs and archive integrity; no research experiment was rerun. ## References - [Source reports and measured results](/downloads/demand-paged-experts-preserve-quality/evidence.zip)