Research · Mainrun assessment · 2026
Adapting frontier LLM-pretraining techniques under fixed compute
Seven epochs, one seed, baseline at 1.7533. Twenty-eight single-variable runs later: 1.1754. The ledger records every one, failures included. The overshoots are how you know the minimums were found, not assumed.
The experiment ladder
Final validation loss per run. Hover a column for the change, or find it in the ledger.
The diagnosis
Baseline underfits. Training loss falls; validation plateaus after step 352. Warmup, weight decay, dropout move the fourth decimal. Regularisation is not the constraint. Learning per step is.
The big lever
AdamW (lr 3e-4) took validation from 1.7533 to 1.3400 — roughly 73% of the total gain in one swap. Heterogeneous gradient scales across attention and MLP; everything else in the ledger stands on this.
Frontier methods, re-tuned
Muon orthogonalises 2D updates via Newton–Schulz. At its published lr (0.02, tuned for 124M-param runs) it plateaued above AdamW. Four times lower it took the lead. WSD removed the late rise cosine left behind; RoPE, ReLU² and value residuals stacked the rest. Frontier methods transfer in direction, not magnitude.
Depth over width
Extra width (8L/512) made things worse at the same step budget. Extra depth compounded: 16 layers landed at 1.1754 (~40M params, 9.1 min). Twenty layers bought 0.0009 for double the runtime. Declined.
The complete ledger
Every experiment, kept. Overshoots and regressions included.
| # | Experiment | Key change | Val loss | Δ base |
|---|---|---|---|---|
| 0 | Baseline | SGD lr=6e-3, CosineAnnealingLR, 27.2M params | 1.7533 | — |
| 1 | AdamW | SGD → AdamW lr=3e-4, betas (0.9, 0.95) | 1.3400 | -0.4133 |
| 2 | Warmup + cosine | + 10% linear warmup, manual cosine decay | 1.3195 | -0.4338 |
| 3 | Weight decay | + wd=0.1 on 2D params only | 1.3194 | -0.4339 |
| 4 | Residual scaling | GPT-2 init 1/√(2·n_layer) on projections | 1.2746 | -0.4787 |
| 5 | Dropout 0.05 | dropout 0.1 → 0.05 | 1.2744 | -0.4789 |
| 6 | Grad accum ×2▾ bracket | + SDPA + gradient accumulation (regression) | 1.3156 | -0.4377 |
| 6b | SDPA only | accumulation back to 1; cause isolated | 1.2744 | -0.4789 |
| 7 | LR 1e-3 | pushed AdamW LR above 3e-4 (underfitting) | 1.2383 | -0.5150 |
| 7b | LR 1.5e-3▾ bracket | overshoot; brackets the optimum at 1e-3 | 1.2457 | -0.5076 |
| 8 | Deeper / narrower | 8 layers, d_model 416 (23.4M params) | 1.2352 | -0.5181 |
| 9 | WD ablation▾ bracket | wd 0.1 → 0.0 at optimal LR (control) | 1.2388 | -0.5145 |
| 11 | Muon, default LR▾ bracket | Muon lr=0.02; calibrated for 124M runs, plateaus | 1.3285 | -0.4248 |
| 11b | Muon tuned | Muon lr=0.005, AdamW lr=1e-3 | 1.2253 | -0.5280 |
| 11c | Muon lr=0.004▾ bracket | confirms 0.005 as the 6-layer optimum | 1.2350 | -0.5183 |
| 12 | Muon + RoPE, lr 0.02▾ bracket | LR too high; RoPE's effect masked | 1.3326 | -0.4207 |
| 13 | WSD schedule | cosine → warmup–stable–decay, cooldown 0.70 | 1.2209 | -0.5324 |
| 13b | WSD cooldown 0.65 | earlier cooldown | 1.2199 | -0.5334 |
| 14 | RoPE | learned positions → rotary embeddings | 1.1985 | -0.5548 |
| 15 | QK-norm▾ bracket | parameter-free RMS-norm on q/k (regression, dropped) | 1.2178 | -0.5355 |
| 16a | Depth re-test | 8L / 416 on the full stack | 1.1933 | -0.5600 |
| 16b | Muon lr=0.006 | re-tuned for the 8-layer depth | 1.1879 | -0.5654 |
| 16c | Muon lr=0.008▾ bracket | overshoot; brackets 0.007 (8-layer) | 1.1875 | -0.5658 |
| 16d | Muon lr=0.007 | the 8-layer optimum | 1.1866 | -0.5667 |
| 16e | ReLU² MLP | squared-ReLU activation (Primer) | 1.1835 | -0.5698 |
| 16f | Value-residual | learnable v-mix to first layer; heads 8 → 4 | 1.1823 | -0.5710 |
| 17a | 16 layers▸ submitted | n_layer 8 → 16 (the submission) | 1.1754 | -0.5779 |
| 17b | 20 layers | 1.1745 for double the runtime; not submitted | 1.1745 | -0.5788 |
Final configuration
- Architecture
- 16 layers · d_model 416 · 4 heads (head_dim 104) · block_size 128
- Parameters
- ≈40.0M (~47% more than the 27.2M baseline, spent on depth)
- Optimiser
- Muon (lr 0.007, momentum 0.95, 5-step Newton–Schulz) on 2D hidden matrices; AdamW (lr 1e-3) on embeddings, head, norms, mixing scalars
- Schedule
- WSD: 6% linear warmup → stable peak → (1−√t) cooldown from 0.65
- Positions
- RoPE, applied to q/k inside attention
- MLP
- Squared-ReLU (ReLU²) activation, 4× expansion
- Attention
- PyTorch SDPA, causal, dropout 0.05 · value-residual: learnable per-layer mix to first-layer values
- Other
- GPT-2 residual init 1/√(2·n_layer) · weight tying · grad clip 1.0
What the failures bought
- Gradient accumulation
- +0.0412. Halved optimiser updates from 938 to 469 inside the fixed budget; a larger effective batch could not buy it back.
- Weight decay > 0
- Negligible either way. Confirmed regularisation was not the operative constraint.
- Muon at default lr 0.02
- Plateaued at 1.3285. Hyperparameters calibrated for 124M-param runs; re-tuned 4× lower to recover.
- QK-norm, no learnable scale
- +0.0193. Capped attention's dynamic range without the per-head gain needed to recover it.
- 8L/512 width
- Worse than 8L/416 at the same step budget. Capacity was not the bottleneck; optimiser steps were.
- Overshoot brackets (7b, 11c, 16c)
- Deliberate runs past each optimum, confirming every minimum was located empirically rather than assumed.
Honest limits
One seed (1337, fixed by the rules). Differences under ~0.005 sit in run-to-run noise. The submission optimises compute, not parameter count; a parameter cap would keep 8 layers and give back 0.012. Built alongside full-time work. The surface had not flattened at submission.