Delphi midtraining · case file

The law didn't break at 1022.
The validation did.

The old 4plus math-validation target stacked a midtraining token budget that grew with scale on top of scale-dependent exposure to near-duplicate math documents. Fixed-token controls and actual-seen clean validation make the endpoint fits smooth again.

Exhibit A · the miss, reconstructed
0.6 0.7 0.8 0.9 3e18 1e20 1e22 fit cutoff · 3e20 predicted 0.665 actual 0.561 +18.6% base pretraining compute (FLOPs, log scale) old 4plus math val loss
fit through 3e20 extrapolation predicted at 1022 actual at 1022
Old 4plus target
+18.6%
K=0.20 lr0.50 at 1e22
→
Clean-seen target
+2.83%
same fit, re-measured
The leak, isolated
+0.0999 loss
dropped-contaminated complement keeps almost the entire original miss
Glossary — terms used in this case file
term meaning
old 4plus validation The original Nemotron-CC-Math 4plus math validation anchor used by the early Delphi midtraining scaling reports.
clean-seen validation A validation set decontaminated against documents actually seen by the 1e22 p33m67 K=0.20 math midtraining run.
dropped contaminated The complement of the clean-seen set: validation documents removed because the seen training stream contained same-source or near-duplicate evidence.
K=0.20 A midtraining budget equal to 20% of the base model pretraining token budget. It is not a fixed-token condition.
iso-token A control ladder where every base scale gets the same total midtraining token budget, such as 1B, 2B, 4B, or 8B tokens.
p33m67 The midtraining mix with about 33% pretraining-like data and 67% math data.
heldout endpoint The 1e21 and 1e22 points, excluded from fits trained through 3e20.
Jaccard near-duplicate A fuzzy overlap measure over normalized 5-character shingles. High values mean two extracted documents share substantial text.
same-source/window leakage The split excluded validation windows, but other windows from the same source document could still appear in the training stream.
prediction error Prediction minus actual. Positive error means the fit predicted too high a loss; the model did better than expected.
01
The symptom

The old 4plus target made 1022 look too good

The frozen original report fit endpoint laws through 3e20 and held out 1e21/1e22. The p33m67 K=0.20 ladder was close at 1e21 but badly high at 1e22: the fit predicted loss around 0.665 for lr0.50 while the measured value was 0.561.

The target was eval/nemotron_cc_math_v1/4plus/loss_anchor. Throughout, the sign convention is prediction minus actual: positive error means the model did better than the fit expected.

Exhibit 01At 1e22 the old-target fit predicted ~0.66 loss for every p33m67 learning rate, while the runs measured ~0.56 — a uniform ~18% overshoot.
Exhibit 02Fit through the 3e20 cutoff: 1e21 lands on the curve; 1e22 falls far below it. The law looked sound right up to the largest scale.

Original p33m67 K=0.20 old-target 1e22 numbers

series old_1e22_actual prediction prediction_error_pct loss_error
p33m67 lr0.33 0.572544 0.681570 19.04 0.109026
p33m67 lr0.50 0.561019 0.665204 18.57 0.104185
p33m67 lr0.67 0.559539 0.661742 18.27 0.102203
p33m67 lr0.83 0.563027 0.663669 17.88 0.100642
02
Ruling out the easy answers

The base models were smooth, and no fit form rescued the old target

The step-0 base loss showed no such failure. A Chinchilla-style fit through 3e20 predicted base step-0 math loss at 1e22 within about +2.4%, while the endpoint p33m67 old-target fit missed by +18.6%. The break is in the post-midtraining endpoint, not the base.

Exhibit 03The base model's step-0 math loss extrapolates cleanly; only the post-midtraining endpoint loss misses. The functional form is not the culprit.

We then tried per-recipe power laws, Chinchilla floor-plus-power fits, pooled LR-aware fits, log-log fits, parameter/data axes, base rows at D=0, and separate base/improvement components. These fits described the fixed-token series well, but the old K=0.20 target stayed an outlier.

Exhibit 04Swapping endpoint functional forms barely moves the 1e22 error — the miss survives every reasonable law we tried.
Exhibit 05The same fit families that fail on the old target behave on iso-token and clean-seen views. The forms work; the target was the problem.
03
Confound one

K=0.20 was never a fixed-token ladder

K=0.20 spends 20% of the base model's pretraining token budget on midtraining. In p33m67 the total midtraining budget grows from about 0.245B tokens at 3e18 to about 32B tokens at 1e22 — and about 67% of that budget is math.

The iso-token controls held the midtraining token budget fixed while sweeping base scale. On the old target, those fixed-token ladders had small 1e22 errors around -3% to -4%; only K=0.20 carried the large positive error.

Exhibit 06Hold midtraining tokens fixed and the ladders stay smooth; only K=0.20 (red, dashed) — whose token budget grows with scale — breaks away.
04
Confound two

The old validation split had fuzzy and same-source leakage

The exact-duplicate scan found zero duplicate document hashes across the 45.1M-doc corpus — and that result was not enough. Fuzzy MinHash/LSH plus exact 5-character-shingle Jaccard verification found substantial near-duplicate overlap between train and validation documents.

At verified Jaccard ≥ 0.75, 9,757 / 57,243 validation docs were implicated, touching 6,839 / 12,500 validation windows and 9.53M / 51.20M validation tokens.

Exhibit 07Exact dedup found zero duplicates, but verified Jaccard overlap exposes thousands of near-duplicate validation docs and pairs.

An actual-exposure replay made the mechanism scale-dependent. For p33m67 K=0.20, combined exposed validation tokens grew from 0.635M at 3e18 to 20.165M at 1e22; at 1e22 the exposure also tracked math fraction across mixes.

Exhibit 08Replaying the actual stream, exposed validation tokens climb with both compute and math fraction — the leak grows exactly where the miss grows.

The curated perplexity-gap study showed the same mechanism at the document level. High-Jaccard documents improved far more at 1e22 than clean documents — consistent with memorization or near-twin exposure rather than a generic base-scaling effect.

Exhibit 09High-Jaccard documents drop in loss far faster than clean ones by 1e22 — the fingerprint of near-duplicate exposure, not generic scaling.
05
The correction

The endpoint fits go smooth on the actual-seen clean target

The final clean-seen set was built against documents actually seen by the 1e22 p33m67 K=0.20 math midtraining stream. It kept 3,367 docs, 2,265,243 tokens, and 553 eval sequences.

The K=0.20 lr0.50 1e22 error moved from +18.56% on old 4plus to +2.83% on clean-seen. The dropped contaminated complement retained a large absolute miss: +0.0999 loss at 1e22, nearly the old target's +0.1042.

Exhibit 10On the actual-seen clean target the K=0.20 outlier collapses onto the fit, while the iso-token ladders stay smooth.
Exhibit 11Splitting the seen set: retained-clean docs fix the miss; the dropped contaminated complement keeps almost all of it.
Exhibit 12After removing the target and token-budget confounds, every 1e22 error sits in the low single digits.

Final compact facts

fact value source
old K=0.20 lr0.50 1e22 error +18.56% old 4plus target
clean-seen K=0.20 lr0.50 1e22 error +2.83% clean-seen target
dropped contaminated 1e22 absolute loss error +0.0999 seen-partition complement
retained clean 1e22 absolute loss error +0.0233 seen partition
iso-token clean-seen 1e22 errors -2.31% to -2.82% 1B/2B/4B/8B fixed-token ladders

Seen-partition summary

target_label actual_1e22 pred_1e22 error_1e22_pct abs_error_1e22 heldout_mae_pct
old full 4plus 0.561143 0.665313 18.563965 0.104170 10.743642
retained clean 0.824991 0.848331 2.829145 0.023340 1.665596
dropped contaminated 0.665261 0.765120 15.010509 0.099859 8.775780

Clean-seen iso-token summary

series_label actual_1e22 pred_1e22 error_1e22_pct heldout_mae_pct
iso-token 2B 0.958299 0.936194 -2.306719 1.850800
iso-token 1B 0.996622 0.972886 -2.381625 1.911204
iso-token 4B 0.926756 0.901372 -2.739014 2.125796
iso-token 8B 0.894756 0.869519 -2.820558 2.160137
K=0.20 iso-FLOP 0.824991 0.848331 2.829145 1.665596
06
Current interpretation

An eval-target confound stacked on a token-budget confound

The evidence supports a validation/measurement confound rather than a broken law of midtraining. The old K=0.20 target mixed base scale, midtraining token budget, math exposure, and near-duplicate validation exposure. Fix the token budget or move to actual-seen clean validation, and the 1e22 endpoint errors return to low single digits.

This is not a claim that every old-target artifact is fully explained. The clean-seen target is built against the 1e22 p33m67 K=0.20 seen set; per-mix actual-seen clean sets would be the stricter follow-up for p50m50 and p67m33.

Artifacts & provenance

artifact location status
Original public report delphi-midtraining public dashboard public
GitHub tracking issue marin-community/marin#6742 public
Contamination branch deconamint public branch
Final retrospective .agents/logbooks/midtraining_prediction_final.md local
Clean-seen K=0.20 summary gs://marin-us-east5/scratch/ahmed/midtrain_dedup/decon_val_sets/evals_clean_seen_1e22_k020/summary_p33m67_clean_seen_1e22_k020.csv GCS
Clean-seen iso-token summary gs://marin-us-east5/scratch/ahmed/midtrain_dedup/decon_val_sets/evals_clean_seen_1e22_isotoken_p33m67_lr50/summary_p33m67_isotoken_clean_seen_1e22.csv GCS
Seen-partition output root gs://marin-us-east5/scratch/ahmed/midtrain_dedup/decon_val_sets/evals_seen_partition_1e22_k020_lr50 GCS