Research · Independent · Ongoing · 2026
Galah: compute-optimal scaling at byte level
Given a compute budget C, what model size minimises loss? The question is posed on a single GPU: byte-level GPTs from 0.1M to 200M parameters, seven iso-FLOP budgets, one fixed training configuration. The measured frontier starts steep and bends toward Chinchilla as compute grows, and training destabilised on the longest schedules. Both observations are documented below.
Active study · qk-norm resolves the instability (20 July) · deployment slice next
The setup
Hoffmann et al. fitted scaling laws with tokenizers, TPU pods and roughly 400 runs. This study reduces the apparatus to essentials: bytes in, bytes out, and a ladder of GPTs small enough that a full iso-FLOP sweep completes overnight on a single card. The configuration is held fixed; only N and C vary. Any structure in the fits therefore reflects scale, not changes to the training procedure.
IsoFLOP profiles
All 43 sweep runs, one profile per budget. Select a budget to highlight its curve; the others are dimmed. Each minimum marks a compute-optimal size. Hollow red points diverged and are excluded from every fit. The seventh budget, 1e18, runs 38M to a new 200M rung and is the deepest profile in the study.
Reading the minima
The clearest single result is at the largest budget. At C = 1e18 the 69M model reaches 1.173 bpb — the lowest validation loss in the study — with the profile bracketed on both sides: 38M (1.194) and 113M (1.182) sit just above it, and the new 200M rung (1.221) confirms the right arm. The fitted optimum is N_opt = 73M. The same structure held one budget down, where 38M (1.249) beat both 18M and 69M at C = 3e17. Every point of the 1e18 profile also landed within 5% of what the previous surface extrapolated for it, a half-decade beyond its fitted range. Compute-optimal training grows steadily more parameter-heavy: the optimal bytes-per-parameter ratio falls from 215 at 1e16 to 25 at 1e18 — approaching the ≈20 Chinchilla reports for BPE tokens.
The exponent
Chinchilla reports b ≈ 0.5, so that doubling compute grows the optimal model by √2. A single power law fitted across all seven budgets here returns b = 0.97 — but no single exponent fits. The local slope between consecutive bracketed budgets declines from 0.92 (1e16→3e16) through 0.87 and 0.74 to 0.61 across the final decade (3e17→1e18): the frontier starts in the steep Kaplan-like regime that small-scale studies report and bends toward Chinchilla's 0.5 within this study's own measured range. How much of that survives noise is the fair question, so each budget's minimum was resampled — bootstrapping the runs within it and refitting — 20,000 times. The verdict is split, and worth stating plainly: the net bend is robust — the top-decade slope is below the bottom-decade slope in 86% of resamples — but a strict step-by-step monotone decline holds in only 9%, because the per-budget slope confidence intervals overlap. The honest claim is therefore the endpoints, not the smooth path: byte-level scaling is steep at small compute and materially shallower by 1e18. The parametric surface, which averages the whole grid, lands between the extremes at implied b = 0.84.
The frontier
N_opt against C, log-log, with reference slopes anchored at the largest budget. The parametric surface is L(N, D) = 0.70 + 8.7·N⁻⁰·¹⁷ + 1.6e7·D⁻⁰·⁹¹ (Huber residual 0.0016 over 39 runs); its N and D exponents imply b = 0.84. The visible flattening at the top of the measured points is the bend toward Chinchilla. The hollow point marks an unbracketed edge budget and is flagged accordingly.
Where training destabilised
Four runs diverged, and each is the longest schedule at its width: 1.5M at 12k steps, 2.7M at 25k, 5.5M at 41k, 10M at 25k. Duration is the controlling variable, and the cleanest evidence is a controlled pair: at a fixed base learning rate of 2.1e-3, the 5.5M model trained cleanly for 13,858 steps (C = 1e17) but diverged at 41,576 steps (C = 3e17), with onset at step 24,400 — beyond the point where the shorter run had already finished. It is not a simple step threshold, though: the shortest diverged run (12,017 steps) is shorter than the longest clean one (14,925 steps). The separating variable is learning rate, which scales as 1/√d_model, so the smaller, hotter models are exactly the ones that fail — and only on their longest schedules. The instability is an interaction of high learning rate with long horizon, not either alone. The 1e18 budget sharpened this: 38M ran 25,263 steps without incident at d=512's colder rate — almost exactly the horizon (25,362 steps) that kills 2.7M at every constant rate tried.
What the trigger is
One run localises the mechanism. At 10M, the ×1.0 and ×0.5 runs share a seed and therefore see an identical batch sequence; both leave their loss floor at the same step (≈5,600) despite a factor of two in learning rate. At ×0.25 the same batches pass without incident and the run holds to the end — so the trigger is data-order-dependent, but only fires above a rate threshold. The full LR ladder settles the question of a constant fix: quarter-rate rescued 10M (1.42 bpb) and 1.5M (1.69) but left both ≈10% above the trend the surface predicts for their (N, D), and the 2.7M and 5.5M rungs needed ×0.125 before a run first survived — and even then the spike still fired (steps 17.3k and 35.2k) and both landed 18–24% off trend. Across an 8× range of constant rates, no long horizon ever produced an on-trend point: colder rates change whether the model survives its spike, not whether the spike happens. That is the closed case for a horizon-dependent schedule rather than a lower peak.
The stability annex
Smoothed train loss for each diverged rung, at LR ×1.0 down to ×0.125. Dotted verticals mark where each run first left its loss floor. Scrub for values.
×1.0 and ×0.5 share seed 1337 and identical batch order; raw loss leaves its floor at step 5,550 in both. At ×0.25 the same batches pass without incident and the run holds to 1.42 bpb — the trigger is data-order-dependent, but only above a rate threshold.
| config | steps | LR ×1.0 | LR ×0.5 | LR ×0.25 | LR ×0.125 | verdict |
|---|---|---|---|---|---|---|
| 1.5M · C3e16 | 12,017 | 3.22 / 3.16 bpb · diverged | 1.79 bpb · held | 1.69 bpb · held | — | rescued · ≈9% off trend |
| 2.7M · C1e17 | 25,362 | 1.86 bpb · spiked, limped in | 3.37 bpb · diverged | 2.13 bpb · diverged | 1.80 bpb · spike survived | stable only at ×0.125 · ≈24% off trend |
| 5.5M · C3e17 | 41,576 | 4.17 bpb · diverged | 1.71 bpb · diverged late | 2.21 bpb · diverged | 1.54 bpb · spike survived | stable only at ×0.125 · ≈18% off trend |
| 10M · C3e17 | 25,307 | 3.63 bpb · diverged | 2.63 bpb · diverged | 1.42 bpb · held | — | rescued · ≈10% off trend |
The intervention log
A running record of every deliberate change to the study's apparatus — what was changed, when, and what was predicted before the results existed. Fits never mix across entries: the 43-run main sweep stands as measured, and every intervention below is an annex against it.
- 16–17 July · divergence isolated
- The four longest-schedule runs fail the smoothed-loss criterion and are excluded from every fit. A seed repeat reproduces the failure — systematic, not seed luck.
- 18–19 July · constant-LR ladder closed
- LR ×0.5, ×0.25 and ×0.125 across the failed configs. Colder rates trade divergence for an off-trend tax that shrinks but never clears; no constant rate produces an on-trend point on any 25k+-step horizon. A uniform rescale is ruled out.
- 19 July · frontier extended to 1e18
- A seventh budget lands fully bracketed (69M optimum, 200M rung). The local exponent falls to 0.61 across the final decade — the bend toward Chinchilla, measured in-study.
- 20 July · pre-registered — two arms
- Registered before any run executes. Arm A, qk-norm: per-head LayerNorm on q and k, on the four failed configs plus two clean controls, at the full ×1.0 rate. Wortsman et al. (ICLR 2024) identify attention-logit growth as the spike mechanism and show this intervention cures it at small scale and high LR — precisely this study's failing corner. Prediction: no spikes at ×1.0, losses on or near trend, controls unmoved. Arm B, horizon-aware LR: peak rate scaled by (T_clean/T_target)^0.5 per rung, model untouched — recent theory has the optimal rate decaying ≈1/√κ as the horizon grows κ×, the term the width-only rule is missing. Prediction: stable and nearer trend than any constant scale. Decision rule, fixed now: if Arm A lands on trend, the deployment model trains with qk-norm and the paper closes on “one normalization changes the measured law.” If only Arm B holds, the width-only LR rule was the artifact and the schedule is the fix. Results append here as they land.
- 20 July · result — Arm A wins, Arm B falsified
- Ten runs, no crashes, no manual intervention. Arm A confirmed the prediction outright. qk-norm at the full ×1.0 rate held every one of the four failed configs flat — no spike in any of them — and each lands on or below its compute-optimal trend: 1.5M 1.493, 2.7M 1.397, 10M 1.261, 5.5M 1.296 bpb, all at or under target. The two controls were not merely unmoved but slightly improved (18M 1.280→1.247, 5.5M@1e17 1.410→1.349), so the intervention never costs loss on a horizon that was already fine. Arm B was falsified. Horizon-aware LR still diverged the 10M run outright (val 2.315, spike at step 6,750) — worse than the best constant scale for that rung — and left the survivors 7–13% off trend. Schedule alone does not fix it; the architectural change does. By the decision rule fixed in advance, qk-norm becomes the recipe for the deployment model, and the study's closing claim stands: one normalization layer moves where the measured law holds.
Compute-optimal is not deployment-optimal
The question the whole study is built to ask. Chinchilla minimises loss for a compute budget and is free to pick any model size. A model that ships to a browser is not: a WebGPU int8 runtime caps the parameter count it can hold and run at interactive speed. So the right objective is not minimise L over all N, but minimise L subject to N ≤ N_max. Below, N_max is the deployment cap; the surface L(N, D) is minimised under it and compared to the unconstrained compute-optimal point at the same budget. Values past 1e18 are the fitted surface extrapolated beyond the measured grid and are labelled as projections.
| Budget C | Compute-opt N · L | Ship at ≤ 25M · L | Deployment tax |
|---|---|---|---|
| 1e17 | 16M · 1.325 | 16M · 1.325 | none (fits) |
| 3e17 | 40M · 1.234 | 25M · 1.242 | +0.008 |
| 1e18 | 111M · 1.150 | 25M · 1.206 | +0.056 |
| 3e18 proj. | 279M · 1.084 | 25M · 1.194 | +0.110 |
| 1e19 proj. | 398M · 1.033 | 25M · 1.190 | +0.157 |
What the slice says
For a 25M-parameter deployment cap, the compute-optimal model outgrows the browser at C ≈ 1.7e17. Past that crossover the shippable model is pinned at N_max and every additional FLOP goes into data, not parameters — deliberately overtraining a small model, which is precisely the regime compute-optimal training calls wasteful. The gap between the two columns is the price of the constraint: 0.056 bpb at 1e18, widening as compute grows. The practical consequence inverts the headline. The model this study ships is not its compute-optimal point (69M at the largest measured budget); it is the largest model a browser can run, trained far past its compute-optimal token count with qk-norm keeping the long schedule stable. Compute-optimal sizing answers a question the deployment target never asked.
Two honest caveats
First, accounting. N counts non-embedding parameters and FLOPs use the 6ND approximation plus the attention term; byte-level keeps embeddings negligible even at the 0.1M rung, which is the reason the ratio is comparable to Chinchilla at all. Second, units. A byte is not a BPE token, so the D/N ≈ 25 measured here and Chinchilla's ≈ 20 are the same quantity in different alphabets — roughly four to five bytes per English token means the two are close in tokens but not identical, and the comparison is directional, not exact. Both are stated so the ratio is not read as a like-for-like match it cannot be.
The apparatus
- Objective
- Next-byte prediction, vocabulary 256. No tokenizer intervenes between the data and the measured exponent.
- Corpus
- 11.9GB of bytes on local disk. Every run is sub-epoch, so data repetition does not confound the fits.
- Configuration, fixed
- AdamW (0.9, 0.95) · weight decay 0.1 · cosine to 10% · 2% warmup · gradient clip 1.0 · bf16.
- Learning rate
- Base LR ∝ 1/√d_model: width-aware but duration-blind. That property is the subject of the stability annex.
- Ladder
- 13 widths × 7 iso-FLOP budgets, populated where size and budget are compatible. Seed 1337 throughout.
- Hardware
- A single shared 96GB GPU. Each budget's profile completes in one overnight session.
Status
- Next
- The recipe is settled: qk-norm stabilises the long schedules at full learning rate, so the deployment model trains with it. Remaining: a couple of seed repeats on the load-bearing optima to put error bars on the minima, then the WebGPU int8 build of the largest browser-runnable model, trained past its compute-optimal token count.
- Completed
- Main sweep across seven budgets (43 runs, 1e15–1e18, with the 1e18 profile bracketed 38M–200M), divergence screen, frontier and parametric fits, seed-repeat control, the full LR annex ladder ×0.5 → ×0.125, a 20,000-sample bootstrap on the frontier bend, both pre-registered intervention arms (qk-norm and horizon-LR, 10 runs), and the deployment-optimal slice. Every annex point stays out of the fits, which rest on the 39 stable main-sweep runs.
- Constraints
- A single seed at nearly every point, seven budgets, byte-level data only. The reported exponents characterise this regime specifically and are not extrapolated beyond it.