Research · Independent · Ongoing · 2026
Galah: compute-optimal scaling at byte level
Given a compute budget C, what model size minimises loss? At byte level N_opt still tracks C, but the exponent bends — local b 0.98 → 0.68 — and a browser parameter cap inverts the optimum. The model this site can actually run is not the compute-optimal one.
Byte-level scaling · 4 seeds at every interior optimum · WebGPU next
The setup
Hoffmann et al. fitted scaling laws with tokenizers, TPU pods and roughly 400 runs. This study reduces the apparatus to essentials: bytes in, bytes out, and a ladder of GPTs small enough that a full iso-FLOP sweep is a workstation-scale job, not a TPU campaign. The configuration is held fixed; only N and C vary. Any structure in the fits therefore reflects scale, not changes to the training procedure.
IsoFLOP profiles
All 43 sweep runs (seed 1337), one profile per budget. Select a budget to highlight its curve; the others are dimmed. Each diamond marks a compute-optimal size from the fitted vertex. Vertical bars are L_std across four seeds at the winning rung. Hollow red points diverged and are excluded from every fit. The seventh budget, 1e18, is bracketed 38M / 69M / 113M — there is no 200M rung in this reconstruction.
Reading the minima
The clearest single result is at the largest budget. At C = 1e18 the 69M model reaches 1.166 bpb — the lowest validation loss in the study — with the profile bracketed on both sides: 38M (1.208) and 113M (1.181) sit just above it. The fitted optimum is N_opt = 74.75M. The same structure held one budget down, where 38M (1.249) beat both 18M and 69M at C = 3e17. Compute-optimal training grows steadily more parameter-heavy: the optimal bytes-per-parameter ratio falls from 237 at 1e16 to 24 at 1e18 — approaching the ≈20 Chinchilla reports for BPE tokens.
Fitted optima
Quadratic-in-log-N vertex per budget, on clean seed-1337 runs. C = 1e15 is censored — the vertex sits at or beyond the sampled range, so it is an upper bound only and carries no error bar. Every interior budget has four seeds.
| Budget C | N_opt | L_opt | Seeds | L_std |
|---|---|---|---|---|
| 1e15 | 0.22M | 2.1234 | — | censored |
| 3e15 | 0.35M | 1.8216 | 4 | 0.0108 |
| 1e16 | 1.88M | 1.6130 | 4 | 0.0220 |
| 3e16 | 5.52M | 1.4618 | 4 | 0.0152 |
| 1e17 | 13.80M | 1.3439 | 4 | 0.0109 |
| 3e17 | 32.97M | 1.2501 | 4 | 0.0066 |
| 1e18 | 74.75M | 1.1658 | 4 | 0.0060 |
The exponent
Chinchilla reports b ≈ 0.5, so that doubling compute grows the optimal model by √2. A weighted least-squares power law over the six interior budgets here returns N_opt = 2.499e-10 · C^0.991 (σ_b = 0.094) — but no single exponent fits. Consecutive local slopes are 1.39, 0.98, 0.76, 0.79, 0.68. The newly uncensored 3e15 hop is the 1.39; the mid-range is not monotone (0.76 then 0.79), so the public claim is the endpoints: local b 0.98 → 0.68 across the measured interior, not a smooth decline. The parametric surface, which averages the whole grid, lands between the extremes at implied b = 0.83.
The frontier
N_opt against C, log-log, with reference slopes anchored at the largest budget. Error bars are log-mapped from nOpt ± nOptStd across four seeds. The parametric surface is L(N, D) = 0.72 + 10.1·N⁻0.18 + 1.69e+7·D⁻0.91 (Huber residual 0.0015 over 36 runs); its N and D exponents imply b = 0.83. The hollow point marks the censored edge budget and is flagged accordingly.
Compute-optimal is not deployment-optimal
The question the whole study is built to ask. Chinchilla minimises loss for a compute budget and is free to pick any model size. A model that ships to a browser is not: a WebGPU int8 runtime caps the parameter count it can hold and run at interactive speed. So the right objective is not minimise L over all N, but minimise L subject to N ≤ N_max. Below, N_max is the deployment cap; the surface L(N, D) is minimised under it and compared to the unconstrained compute-optimal point at the same budget. FLOPs use the 6ND approximation (no attention term). Values past 1e18 are the fitted surface extrapolated beyond the measured grid and are labelled as projections.
| Budget C | Compute-opt N · L | Ship at ≤ 25M · L | Deployment tax |
|---|---|---|---|
| 1e17 | 16M · 1.323 | 16M · 1.323 | none (fits) |
| 3e17 | 40M · 1.232 | 25M · 1.241 | +0.009 |
| 1e18 | 110M · 1.147 | 25M · 1.204 | +0.057 |
| 3e18 proj. | 276M · 1.083 | 25M · 1.193 | +0.110 |
| 1e19 proj. | 755M · 1.023 | 25M · 1.189 | +0.166 |
6ND approximation · attention term omitted
Two estimators, not a disagreement. The isoFLOP table above locates N_opt as a quadratic vertex on the measured grid (74.75M at 1e18). This table minimises the fitted surface L(N, D) under 6ND accounting (110M at 1e18). The vertex is what was run; the surface is what the slice extrapolates.
What the slice says
For a 25M-parameter deployment cap, the compute-optimal model outgrows the browser at C ≈ 1.7e17. Past that crossover the shippable model is pinned at N_max and every additional FLOP goes into data, not parameters — deliberately overtraining a small model, which is precisely the regime compute-optimal training calls wasteful. The gap between the two columns is the price of the constraint: 0.057 bpb at 1e18, widening as compute grows. The practical consequence inverts the headline. The model this study ships is not the unconstrained surface minimum (110M at 1e18); it is the largest model a browser can run, trained far past its compute-optimal token count. Compute-optimal sizing answers a question the deployment target never asked.
Two honest caveats
First, accounting. N counts non-embedding parameters. The table above uses the 6ND approximation; the isoFLOP budgets themselves include the attention quadratic, so the ratio in the fitted optima is not the same quantity as the 6ND slice. Byte-level keeps embeddings negligible even at the 0.1M rung, which is the reason the ratio is comparable to Chinchilla at all. Second, units. A byte is not a BPE token, so the D/N ≈ 24 measured here and Chinchilla's ≈ 20 are the same quantity in different alphabets — roughly four to five bytes per English token means the two are close in tokens but not identical, and the comparison is directional, not exact. Both are stated so the ratio is not read as a like-for-like match it cannot be.
Where training diverged
That over-training is not only the deployment recipe — it is where this frozen setup fails. 7 of 61 completed runs failed the smoothed-loss screen: final EMA more than 15% above the run's own best. All 7 are mid-size rungs trained far past that budget's compute-optimal token count, not a simple high D/N rule. The smallest rungs stay clean at far higher D/N when they sit near their own optimum. None of the diverged points land on a fitted vertex, so no reported N_opt is affected. The July stability annex that originally probed this envelope was not reconstructed.
Provenance
The original cluster that ran the July 2026 sweep became unavailable, and with it the run artefacts. The study was rebuilt from the frozen recipe on rented cloud GPUs — an RTX 4090 24GB for rungs through 18M, an A100 SXM4 80GB for 38M and above — against FineWeb-Edu prepared independently on both boxes and verified byte-identical by SHA-256 between those two boxes. 61 completed runs (43 sweep + 18 seed repeats), 27.5 GPU-hours, about 14.8 hours of wall-clock, roughly sixteen dollars of compute. That is a reconstruction, not a second experiment: same code, same hyperparameters, new silicon.
Bracketing the low edge
A budget whose observed minimum sits on the low edge of the ladder is extended downward one existing rung at a time until the minimum is bracketed or the ladder is exhausted — a sampling rule fixed before any reconstruction run executed, so the outcome could not be cherry-picked. C = 3e15 is now an interior optimum (0.3M, bracketed by 0.2M and 0.8M). C = 1e15 remains censored even after that extension: the fitted parabola vertex still falls outside the sampled range. One extra point beyond the stopping rule (0.1M at 3e15) is kept out of the fits.
July annex — archived
The stability lab and intervention log below ran in July 2026 on the original cluster. That cluster is gone; these curves were not reconstructed. They are a dated historical record, not the current recipe. qk-norm is not how the reconstructed sweep was trained.
Smoothed train loss for each diverged rung, at LR ×1.0 down to ×0.125. Dotted verticals mark where each run first left its loss floor. Scrub for values.
×1.0 and ×0.5 share seed 1337 and identical batch order; raw loss leaves its floor at step 5,550 in both. At ×0.25 the same batches pass without incident and the run holds to 1.42 bpb — the trigger is data-order-dependent, but only above a rate threshold.
| config | steps | LR ×1.0 | LR ×0.5 | LR ×0.25 | LR ×0.125 | verdict |
|---|---|---|---|---|---|---|
| 1.5M · C3e16 | 12,017 | 3.22 / 3.16 bpb · diverged | 1.79 bpb · held | 1.69 bpb · held | — | rescued · ≈9% off trend |
| 2.7M · C1e17 | 25,362 | 1.86 bpb · spiked, limped in | 3.37 bpb · diverged | 2.13 bpb · diverged | 1.80 bpb · spike survived | stable only at ×0.125 · ≈24% off trend |
| 5.5M · C3e17 | 41,576 | 4.17 bpb · diverged | 1.71 bpb · diverged late | 2.21 bpb · diverged | 1.54 bpb · spike survived | stable only at ×0.125 · ≈18% off trend |
| 10M · C3e17 | 25,307 | 3.63 bpb · diverged | 2.63 bpb · diverged | 1.42 bpb · held | — | rescued · ≈10% off trend |
The intervention log
A running record of every deliberate change to the study's apparatus on the original cluster — what was changed, when, and what was predicted before the results existed. Historical; not reconstructed.
July 2026 · original cluster, not reconstructed
- 16–17 July · divergence isolated
- The four longest-schedule runs fail the smoothed-loss criterion and are excluded from every fit. A seed repeat reproduces the failure — systematic, not seed luck.
- 18–19 July · constant-LR ladder closed
- LR ×0.5, ×0.25 and ×0.125 across the failed configs. Colder rates trade divergence for an off-trend tax that shrinks but never clears; no constant rate produces an on-trend point on any 25k+-step horizon. A uniform rescale is ruled out.
- 19 July · frontier extended to 1e18
- A seventh budget lands fully bracketed. The local exponent falls across the final decade — the bend toward Chinchilla, measured in-study. (The reconstruction later dropped the 200M rung; see the current 38M / 69M / 113M profile above.)
- 20 July · pre-registered — two arms
- Registered before any run executes. Arm A, qk-norm: per-head LayerNorm on q and k, on the four failed configs plus two clean controls, at the full ×1.0 rate. Wortsman et al. (ICLR 2024) identify attention-logit growth as the spike mechanism and show this intervention cures it at small scale and high LR — precisely this study's failing corner. Prediction: no spikes at ×1.0, losses on or near trend, controls unmoved. Arm B, horizon-aware LR: peak rate scaled by (T_clean/T_target)^0.5 per rung, model untouched — recent theory has the optimal rate decaying ≈1/√κ as the horizon grows κ×, the term the width-only rule is missing. Prediction: stable and nearer trend than any constant scale. Decision rule, fixed now: if Arm A lands on trend, the then-planned deployment model trains with qk-norm. If only Arm B holds, the width-only LR rule was the artifact and the schedule is the fix. Results append here as they landed.
- 20 July · result — Arm A held, Arm B falsified
- Ten runs, no crashes, no manual intervention. Arm A confirmed the prediction outright on that cluster: qk-norm at the full ×1.0 rate held every one of the four failed configs flat, and each landed on or below its compute-optimal trend. The two controls were not merely unmoved but slightly improved. Arm B was falsified. Horizon-aware LR still diverged the 10M run outright and left the survivors off trend. Schedule alone did not fix it; the architectural change did — in July, on hardware that no longer exists. That finding is archived here. It is not the recipe the reconstructed sweep uses, and it is not claimed as settled.
The apparatus
- Objective
- Next-byte prediction, vocabulary 256. No tokenizer intervenes between the data and the measured exponent.
- Corpus
- FineWeb-Edu, prepared independently on both reconstruction GPUs and verified byte-identical by SHA-256 between those two boxes. Every run is sub-epoch, so data repetition does not confound the fits.
- Configuration, fixed
- AdamW (0.9, 0.95) · weight decay 0.1 · cosine to 10% · 2% warmup · gradient clip 1.0 · bf16.
- Learning rate
- Base LR ∝ 1/√d_model: width-aware but duration-blind. The frozen recipe was not changed for the reconstruction.
- Ladder
- Widths × seven iso-FLOP budgets, populated where size and budget are compatible. Seed 1337 for the sweep; seeds 2337 / 3337 / 4337 at every interior optimum.
- Hardware
- Reconstructed on cloud GPUs (RTX 4090 24GB + A100 SXM4 80GB) after the original training cluster became unavailable.
Status
- Next
- WebGPU int8 build of the largest browser-runnable model, trained past its compute-optimal token count.
- Completed
- [x] sweep (reconstructed on cloud GPUs after original cluster loss)[x] seed repeats at every interior optimum (4 seeds)[x] paper fits (frontier WLS + parametric surface)[x] deployment slice (6ND-approx)
- Constraints
- Four seeds only at the load-bearing optima, not the full grid. 7 diverged runs excluded; none sit on a fitted vertex. C = 1e15 remains censored. The July stability annex was not reconstructed. The reported exponents characterise this regime specifically and are not extrapolated beyond it.