Forecasting the Effects of Midtraining on Delphi

TL;DR. We midtrained every model on the Delphi ladder (3e18 to 1e22 FLOPs, 447M to 9.7B parameters) on a mixture of 67% web mathematics and 33% pretraining data for 20% of its pretraining compute. Ben Feuer then ran SFT on every checkpoint and RL on the 1e22 models, and we measured four stages of the same models: base, midtrained, after SFT, and after RL. Midtraining loss and coverage scale smoothly with base compute, and fits through the seven smaller scales predict the held-out 1e22 model within 3%. Midtraining raises MATH-500 pass@128 at 1e22 from 10.4 to 79.6 before any post-training. The same SFT recipe reaches 1.6 pass@1 on the base model and 45.0 on the midtrained model, and RL adds a further 8.4. Midtraining raises the coverage of correct answers, SFT converts that coverage into single-sample accuracy, and RL adds a further gain.

“Midtraining” is loosely defined as a stage of training on a specialized data mixture between pretraining and post-training. It is typically done by changing the data mixture during the cooldown of pretraining, but it can also be done as continued pretraining on a finished base model. DeepSeek-R1 trained on math and code data late in pretraining, then applied reinforcement learning (RL), and reached reasoning performance beyond the base model [1]. OctoThinker reported that midtraining incentivizes RL scaling [2]. Both results indicate that midtraining matters for post-training, but few controlled studies of it exist, and each of these describes one model at one scale.

Ideally, we'd have a way to measure the impact of midtraining across different scales. A scaling ladder is the standard tool for this kind of analysis, and the team at Marin has successfully applied it to predict pretraining loss for dense and MoE models up to 1e23 FLOPs [3]. Around May, I realized that we didn't have a similar set of analyses for midtraining, and in fact that (to my knowledge) no such scaling law study existed for midtraining. This seemed like a large gap, since pretty soon we'd want to move to post-training and it would be helpful to predict how midtraining affects post-training before paying for it at full scale. So, I started running experiments to determine how mid-training composes with pretraining, and affects downstream SFT and RL. I began with the compute optimal points across the Delphi scaling ladder, and in classic Marin fashion I documented the entire process publicly. This post is what came out of it, and what we learned about midtraining.

MATH-500, GSM8K, and AIME24 by pipeline stage, Delphi 1e22 (9.7B)

Midtraining and post-training results at 1e22 (9.7B): midtraining raises pass@128 eightfold, SFT raises pass@1 from 6.0 to 45.0, and RL adds 6–8 points. The dropdown selects the benchmark. Hatched bars are pass@128 where it was measured. Orange: base model and base model after SFT. Blue: midtrained model and the midtrained after SFT. Green: midtrained+SFT'd model after GRPO RL. The midtraining mix is nemotron-cc-math and the SFT mix is WildChat.

show data

Related work

DeepSeek-R1 [1] trained one model on math and code data late in pretraining and then applied RL. It reports the end result at one scale and does not separate the contribution of the data stage from the contribution of RL.
OctoThinker [2] midtrained Llama models on math data and reported that midtraining incentivizes RL scaling. It compares midtraining recipes at one or two model sizes after RL, and does not measure coverage before post-training or fit scaling laws.
Zhang, Neubig, and Yue [11] built a fully synthetic testbed (arithmetic dependency graphs rendered into templates, with disjoint splits for pretraining, midtraining, and RL) and trained one 100M-parameter model on 10B tokens, then GRPO with process-verified rewards. They found that RL raises pass@128 only when RL data sit at the edge of the pretrained model’s competence, that about 1% pretraining exposure to a context is enough for RL to transfer to it, and that midtraining plus RL beats RL alone at equal compute (+10.8 points on their hardest split). Their design gives causal control that real data cannot, at one small scale, with no SFT stage and no scaling axis. This post measures the same three stages on real pretraining and math data across nine compute-optimal model sizes, adds the SFT stage, measures coverage before any post-training, and tests whether loss and coverage extrapolate from small scales. The two studies agree on the central claim: midtraining sets what RL can reach, and RL alone is a weaker use of the same compute.

The ladder and the recipe

Delphi is Marin’s open scaling suite: Qwen3-architecture models pretrained on Nemotron-CC at nine compute budgets from 3e18 FLOPs (447M parameters, 1.2B tokens) to 1e22 FLOPs (9.7B parameters, 160B tokens), plus a partial run at 1e23 FLOPs (25B parameters) [3]. Technically we trained using Chinchilla method 2 so we have over-trained models for each compute budget, but to keep things simple I decided to use the compute-optimal checkpoints (I return to this choice in the Reflections at the end of the post.) Going forward I'll refer to each checkpoint by its compute budget.

I decided to fix the midtraining dataset as nemotron_cc_math_v1/4plus [4], the highest-quality band of Nemotron-CC-Math: 52B tokens of web mathematics, extracted with a layout-aware browser and cleaned by Phi-4. Marin had already tokenized this corpus, and it contains no Qwen-generated text, which permits later comparisons with Qwen models. The other candidate was Nemotron-Math-v2, about 347K human-authored problems with 7M model-generated solutions. It is high-quality math but far fewer tokens, and its solutions are generated by gpt-oss-120b, which is a different style of data than web mathematics. I wanted the volume of the web corpus for a 32B-token midtraining run at 1e22. I mixed the math corpus with the delphi pretraining distribution at three ratios, named by their pretraining-to-math split: p33m67 (67% math), p50m50 (50% math), and p67m33 (33% math).

I ran each midtraining cell as continued pretraining (CPT) using AdamH and a linear learning rate schedule with 10% warmup. It wasn't clear to me which learning rate would be best compared to pretraining, so I experimented with a few options. Larry Dial suggested that the optimal peak learning rate using a linear schedule would be higher than WSD, but we also add a moderate prior that the learning rate for midtraining should be smaller than during pretraining, owing to the fact that we're changing the data mixture. Given this, I ran multiple experiments varying the peak learning rate as a fraction of the base model’s pretraining peak over {0.33, 0.5, 0.67, 0.83}. Each run trains for a fixed fraction K = 0.20 of its base model’s pretraining token budget. I chose K = 0.20 because it equals the length of the decay phase in the bases’ warmup-stable-decay (WSD) schedule: details on all runs and experiments are below

Recipe, launcher, and data identity

Base models: the canonical Delphi checkpoints registered in experiments/delphi_models.py (Hugging Face: marin-community/delphi-{scale}-…). The 1e21 and 1e22 bases are adamh-scaling-ladder-nemotron-optimal-{1e+21,1e+22}; the seven smaller bases are the AdamH-v6 isoflop bucket winners.
Launcher: experiments/midtrain_specs/delphi_small_cpt_k020.py on branch midtrain_data. One invocation launches one cell (--base 3e18 --mix p33m67 --lr 0.5); --budget-tokens N selects a fixed token budget for the iso-token controls. The typed helpers are in lib/marin/src/marin/midtraining/.
Data identity: the data: block of each mix is a frozen JSON copied from the 1e21 reference runs. The math validation split (12,500 sequences, 51.2M tokens, a fixed pseudo-random carve-out of the 4plus-2c5519 cache) is therefore byte-identical across every scale, mix, and control.
Optimizer: AdamH, weights-only initialization, 10% linear warmup, linear decay to 0.1× the peak, batch size and sequence length (4096) inherited from the base model.
Steps at K = 0.20: 3e18 7,399 · 9e18 8,818 · 2e19 10,982 · 3e19 7,573 · 9e19 8,032 · 2e20 11,277 · 3e20 7,081 · 1e21 4,410 · 1e22 7,646.
Logging: W&B project marin-community/delphi-midtraining. Every run exports Hugging Face checkpoints at about ten intermediate steps and at the final step.

Continued pretraining versus in-schedule midtraining

Every run above is CPT: a new optimizer and a new warmup on a fully decayed base model. In-schedule midtraining swaps the data during the decay phase of pretraining and keeps the optimizer state. To compare the two, I rebuilt every Delphi pretraining run to its exact 80% step, the start of the WSD decay phase, by re-running prefixes of the nine pretraining runs to materialize full-state checkpoints that did not exist. I then completed the last 20% of steps with the data swapped to p33m67. The 3e18–3e20 runs completed, and their math loss decreases monotonically with scale, from 1.408 at 3e18 to 0.904 at 3e20. The 1e21 and 1e22 prefix checkpoints were completed (the 1e22 prefix required seventeen attempts across two regions), but their decay-phase runs were not launched before the compute allocation ended. The SFT and RL results below use the CPT checkpoints.

Midtraining Scaling Ladder

I started in April at the two largest scales, as we were fortunate enough to have lots of v5p nodes available. By early May I had a 24-cell sweep at 1e21 and 1e22: 3 mixes × 4 learning rates × 2 scales, each at K = 0.20 where K is the fraction of compute used in proportion to pretraining. I then ran the seven smaller scales, 3e18 through 3e20, for a further 84 cells. As a sanity check, I also ran iso-token experiments from 1B to 8B midtraining tokens at each scale, for a further 45 cells.

I measured each cell on two held-out losses. The math validation loss is computed on a held-out split of the 4plus corpus totaling 51.2M tokens. To measure how much each model regresses on the pretraining distribution, I also evaluated each cell on Paloma, a perplexity benchmark over the pretraining distribution that measures retention. The findings were recorded (#4547):

Optimal learning rate differs across scale. On math validation loss, p33m67 is (unsurprisingly) best at every scale. On Paloma retention, p67m33 (33% math) at learning-rate factor 0.33 is best at every scale. However, the best learning-rate factor does not stay fixed and the larger models, tend to perform better with lower learning rates.
The mix gap is largely scale-invariant. Across the nine scales, p67m33 trails p33m67 by 0.090 ± 0.012 nats of math validation loss, and p50m50 trails by 0.038 ± 0.004 nats.
Learning rate mostly affects pretraining retention. Within each mix, the Paloma loss increases monotonically with the learning-rate factor from 0.33 to 0.83, while the math validation loss varies by under 0.04 nats across factors {0.5, 0.67, 0.83}. The retention cost also grows with scale: for p33m67 at factor 0.5, the Paloma loss exceeds the base model’s by 0.033 nats at 1e21 and by 0.045 nats at 1e22. This suggests my hunch on the midtraining learning rate needing to be lower is true.

Is loss after midtraining predictable through a scaling law? I fit a floor-plus-power law through the seven smaller scales and held out 1e21 and 1e22. The fit predicts the held-out 1e22 loss within 3% for the K = 0.20 ladder (+2.8%) and for every fixed midtraining budget (−2.3% to −2.8%) (figure below). The base models’ math loss before midtraining extrapolates with +2.4% error at the same scale.

Endpoint math validation loss vs base compute

Math validation loss after midtraining decreases smoothly with base compute, for the K = 0.20 ladder (dashed) and four fixed midtraining token budgets (solid), all p33m67 at learning-rate factor 0.5. Dotted lines are floor-plus-power fits through 3e18–3e20. Open markers are the held-out 1e21 and 1e22 points. Hover over an open marker for the fit’s prediction, the measured loss, and the error. Held-out 1e22 errors: +2.8% for the K = 0.20 ladder and −2.3% to −2.8% for the fixed budgets.

show data

Reaching that result wasn't straightforward. My first fits missed 1e22 by +18.6%, and it took three weeks to find out why. The dropdown below records how I found and fixed the cause, but press on if you're impatient to learn about how Post-training went.

The investigation: a failed extrapolation and a contaminated validation split postmortem

The failed extrapolation

In June I fit endpoint scaling laws through the seven smaller scales (3e18 to 3e20) and held out 1e21 and 1e22. Throughout, prediction error is the predicted loss minus the measured loss, as a percentage of the measured loss. The 1e21 error was under 3%. The 1e22 error was +18.6%: the fit predicted a math loss of 0.665 for the p33m67 model, and the model reached 0.561. Every learning rate and every mix erred in the same direction. The base models’ math loss before midtraining extrapolated with +2.4% error at 1e22. The error appeared only after midtraining.

The figure below is the loss figure above evaluated on the original validation split. The K = 0.20 1e22 endpoint falls below its fit, while 1e21 lies on it. The frozen report with the Chinchilla-style fits quoted below is at ahmeda14960.github.io/delphi-midtraining.

Endpoint math validation loss vs base compute, original validation split

The same series and fit protocol as the loss figure above, evaluated on the original validation split. The K = 0.20 ladder misses 1e22 by +18.6%. The fixed-budget ladders miss by −3.1% to −3.8%.

I first tested whether the functional form caused the error: per-recipe power laws, Chinchilla floor-plus-power fits, pooled fits with learning-rate terms, log-log fits, parameter and data axes, and separate base-loss and improvement components. The best form reduced the 1e22 error to +13.8%. The same forms fit the fixed-token ladders with under 1% error. The K = 0.20 series, not the fitting procedure, was the anomaly.

Functional forms tested

Fit	Held-out endpoint MAE	K=0.20 1e22 error
per-recipe power law	1.1% at 1e21	+10.7%
pooled N × D_math + LR	1.1% at 1e21	+9.0%
per-recipe Chinchilla floor+power	2.3% at 1e21	+15.8%
Chinchilla endpoints: N + D_pre + D_mid (best)	1.99%	+13.8%
separate base loss + saturating improvement	—	+15.8%
same families, iso-token runs only	0.93%	+2.4% (8B budget)

MAE is the mean absolute percentage error over the held-out endpoints. The full reports are in the retrospective.

Confound one: K = 0.20 is not a fixed-token condition

A sweep at K = 0.20 is iso-FLOP in one sense: each cell spends the same fraction of its base model’s compute. The absolute budget grows with the base model. The 1e22 model trained on 32B midtraining tokens and the 3e18 model on 0.24B. A fit of loss against base compute therefore combines three variables: model size, pretraining data, and midtraining tokens.

Midtraining tokens per cell at K = 0.20

The K = 0.20 midtraining budget spans 130× across the ladder, from 0.24B tokens at 3e18 to 32B at 1e22. Dashed lines mark the fixed budgets of the iso-token ladders in the loss figure above.

I ran the control in June. Each iso-token ladder holds the midtraining budget fixed and sweeps the base model: five budgets (0.5B, 1B, 2B, 4B, and 8B tokens) at nine scales, 45 cells, all p33m67 at learning-rate factor 0.5. For 1B tokens, the 3e18 model runs 30,518 steps at batch size 8 and the 1e22 model runs 238 steps at batch size 1024. Every iso-token ladder decreases monotonically with scale, with no crossover, and the held-out 1e22 errors are between −3% and −4% at every budget. Only the K = 0.20 ladder has the large positive error.

Iso-FLOP vs iso-token endpoint scaling under one fit protocol

Under the same fit and held-out protocol, the K = 0.20 ladder (left) misses the held-out 1e22 point and the 8B iso-token ladder (right) does not. The log-log fits drawn have 1e22 errors of +15.3% (left) and −3.2% (right). The frozen report’s Chinchilla-style fits give +18.6% and −3.1%. Filled markers are the fit points (3e18–3e20). Open markers are the held-out 1e21 and 1e22 points.

The token budget explains most of the error but not all of it. The fixed-token errors are consistently negative, and the size of the K = 0.20 error increases with the math fraction of the mix (p33m67 largest, p67m33 smallest). Confound two identifies the second cause.

Operational note: eight attempts at one cell

The 1e22 cell at 1B tokens required eight attempts. An int32 overflow in the data loader, host-memory exhaustion, a hang at step 99, and TPU quota limits across three regions each ended an attempt before attempt a008 finished on a v5p-32 in us-central1. The midtraining logbook records every budget, scale, and attempt.

Confound two: the validation split contained near-duplicates of training documents

The math validation split is a random subsample of the 4plus corpus: a fixed pseudo-random permutation orders the documents, the first 12,500 windows form the validation split, and the remainder forms the training split. The split is stable and byte-identical across every run. An exact-duplicate check found zero duplicate document hashes among 45.1M documents.

A fuzzy-duplicate scan found many near-duplicates. Nemotron-CC-Math extracts each page by rendering it and cleaning the text with a language model. A page crawled in two snapshots yields two near-duplicate documents rather than one exact duplicate. I measured document similarity by Jaccard similarity over 5-character shingles, the fraction of shingles two documents share, using MinHash-LSH to find candidate pairs and exact computation to verify them. 17% of validation documents (9,757 of 57,243) had a training near-duplicate at Jaccard ≥ 0.75. These documents cover 55% of validation windows and 18.6% of validation tokens. At Jaccard ≥ 0.5, 36% of validation documents had a training near-duplicate.

Verified near-duplicate pairs, validation vs training split

The validation split has 374,929 verified near-duplicate pairs with the training split at Jaccard ≥ 0.5 (left) and zero exact duplicates. The most common maximum similarity per validation document is 0.75–0.85 (right).

Corpus-level contamination is constant across scales and cannot by itself produce a scale-dependent error. Exposure varies with scale. I replayed the data stream of each K = 0.20 run and counted the validation tokens whose near-duplicate the run had trained on. For p33m67, exposure grows from 0.6M validation tokens at 3e18 to 20.2M at 1e22, and at 1e22 it increases with the math fraction of the mix. Exposure grows exactly where the extrapolation error grows.

Contamination exposure and the held-out 1e22 error

Exposure to near-duplicates of validation documents grows with scale (0.6M to 20.2M tokens) and with math fraction, and the extrapolation error follows it: on the original split the K = 0.20 1e22 error is +18.6%, on the validation documents with no near-duplicate among the 1e22 run’s training documents it is +2.8%, and on the dropped documents it is +15.0%. Every fixed-token ladder has an error between −2% and −4% on both splits. The dropdown selects the view.

I then evaluated every checkpoint on decontaminated validation splits that drop each validation document whose maximum training Jaccard exceeds a cutoff τ. At 3e18, dropping near-duplicates lowers the loss by 0.11 nats: near-duplicate documents are long, templated pages, which small models predict poorly. At 1e22, dropping them raises the loss by 0.04 nats: the largest model predicts near-duplicate documents better than clean documents. The improvement from 1e21 to 1e22 is 0.238 nats on the original split and 0.149 nats on the τ = 0.5 split. Near-duplicates account for 37% of the measured improvement.

Original vs decontaminated (τ = 0.5) validation loss, p33m67, learning-rate factor 0.33

Dropping near-duplicates (τ = 0.5) lowers the loss of the p33m67 model at 3e18 and 1e21 and raises it at 1e22. Bars are the p33m67 cells at learning-rate factor 0.33.

The cutoff sweep and the per-document mechanism exhibits

Validation loss vs decontamination cutoff, p33m67, learning-rate factor 0.33

Validation loss decreases as the cutoff τ drops near-duplicates at every scale up to 1e21 and increases at 1e22. Stars mark the original split. Each curve is one scale. The dropdown selects the near-duplicate index used to build the splits.

The per-document losses show memorization. I scored 30 curated validation documents token by token at six scales. Clean documents improve by 0.11 nats from 1e21 to 1e22. Documents with a training near-duplicate at Jaccard 0.88 improve by 0.73 nats. One such document, a page titled “Unit 14: Time is Money,” falls from a mean loss of 2.70 to 0.27, and 78% of its tokens have a loss below 0.05 nats at 1e22. The number of near-duplicate copies the run trained on predicts this collapse (correlation −0.39 with 1e22 loss); the similarity of the closest copy does not (+0.04).

Mean loss by Jaccard band vs compute

Documents in the highest Jaccard band have the highest loss at 3e18 and the largest loss reduction by 1e22. Each point is the mean over 5 curated validation documents.

Per-token loss, near-duplicate vs clean document

At 1e22 (green), the per-token loss of a validation document with a Jaccard-0.87 training near-duplicate (top) falls to near zero over most of the document, as does that of its training near-duplicate (middle). A clean document (bottom) improves by a constant factor.

The final test partitions the validation split by the documents the 1e22 p33m67 run trained on. A 13-gram containment pass over the run’s 21.7M training documents identified the validation documents with a near-duplicate among them. I call the 3,367 documents without a near-duplicate the seen-clean split and the 53,876 dropped documents the dropped split. I then refit the K = 0.20 ladder on each split. The error moved with the dropped documents. On the seen-clean split the K = 0.20 1e22 error is +2.8%, and the iso-token ladders have errors between −2.3% and −2.8%. On the dropped split the error is +15.0%.

K = 0.20 ladder refit on three validation targets

The K = 0.20 p33m67 ladder (learning-rate factor 0.5) refit on three validation targets with a floor-plus-power fit through 3e18–3e20: the original split has a 1e22 error of +18.6%, the seen-clean split +2.8%, and the dropped split +15.0%. Open markers are the held-out 1e21 and 1e22 points.

Scan procedure and its errors

Exact scan: all 45,096,087 xxh3_128 document ids in the normalized corpus are unique.
Fuzzy scan: MinHash over 5-character shingles, followed by locality-sensitive hashing (LSH) at two band settings. The 286×26 setting used for deduplication has 1.3% candidate recall at Jaccard 0.5. The 284×71 setting (about 99% recall) found 56% more contaminated documents at Jaccard ≥ 0.5. Exact Jaccard computation verified every candidate pair.
Exposure replay: the replay rebuilds each K = 0.20 run’s training index (mixture block rounding, block shuffle, seed 0) without reading tokens and enumerates the documents the run trained on. The 1e22 p33m67 run trained on 21.7M distinct math documents, 48% of the corpus.
Decontaminated splits: nine cutoffs, τ from 0.5 to 0.9, built against the 4plus corpus alone and against the union of all Nemotron-CC-Math subsets. The two constructions differ in about 6% of dropped documents and give the same conclusions.
Errors that cost days: an off-by-one from reading a document-end offset array as document starts, which an exact-count check caught. different text normalization in the scan and verification stages, which shifts the Jaccard scale by about 0.03 toward under-decontamination. Reuse of the scan band setting for deduplication marked 99.9% of a slice as duplicates.

The contamination logbook on branch deconamint records the scans, the dossier collects the exhibits, and issue #6742 tracks the investigation. The seen-clean split is defined by the 1e22 p33m67 run. Per-mix splits for p50m50 and p67m33 remain to be built.

I took two rules from this. First, deduplicate a corpus by fuzzy matching before carving a validation split from it. Second, pair any midtraining sweep at a fixed compute fraction with a fixed-token control before fitting a scaling law to it.

Coverage before post-training

While it was great that we found smooth scaling across various mid-training budgets, unfortunately loss rankings can be misleading for downstream evals. Rohith Kuditipudi suggested we evaluate the midtrained checkpoints on a solution-primed GSM8K task as a sanity check. In this task the worked solution is in the prompt and the model emits only the final number. He evaluated every 3e20, 1e21, and 1e22 cell (5-shot, 256 problems, 32 samples per problem, temperature 0.6) and reported sampled pass@1, the mean correctness over the 32 samples. Every p33m67 cell exceeded its base model, and so did 34 of the 36 cells overall. The two exceptions are p67m33 at 1e21 at learning-rate factors 0.5 and 0.83, which sit 5.9 and 0.8 points below the base. p33m67 was the best mix at every scale (#4549).

Solution-primed GSM8K: gain over the base model

Gain over the base model on solution-primed GSM8K (sampled pass@1, the mean correctness over 32 samples, in points) for every mix × learning-rate cell: 7–16 points at 3e20 and 5–8 points at 1e22. Two p67m33 cells at 1e21 fall below the base. The dropdown selects the scale. p33m67 is the best mix at every scale.

We found that the ranking from math validation loss transfers to a relevant, albeit simple downstream task. So our sanity check confirms that midtraining is effective, but the performance gain shrunk with scale, from +15.6 at 3e20 to +8.2 at 1e22, because the base model is already at 84.7% at 1e22. However, when looking at error rates the largest model gains the most: midtraining removes 41% of the base model’s errors at 3e20 and 54% at 1e22. The same run also reports pass@32, and it is saturated before midtraining: the base models solve 99.6% of problems at 3e20 and 1e21 and 100% at 1e22 with at least one of 32 samples. At this point it was clear we needed a harder task so we switched gears to MATH-500 and AIME.

Around the same time, Ben Feuer joined Marin and offered to post-train the models. He ran the 27 best K = 0.20 checkpoints (9 scales × 3 mixes) as measured by math validation loss and the 9 base models through one pipeline: learning-rate factor 0.83 at every scale through 3e20 for all three mixes; at 1e21, 0.67 for p33m67 and p67m33 and 0.83 for p50m50; at 1e22, 0.67 for p33m67, 0.5 for p50m50, and 0.33 for p67m33. He ran them through one pipeline (#6279): cold-start SFT on every checkpoint, then GRPO RL on the 1e22 models, with one evaluation protocol across every stage, with more details in the dropdown below.

Post-training pipeline and evaluation protocol

SFT: two cold-start recipes, one math-strong (Magpie-Llama-3.3-Pro [5]) and one math-weak (WildChat-386k [6]), each blended with 10% chain-of-thought warmup data, at learning rate 1e-5 for one epoch.
RL: GRPO, a policy-gradient method that normalizes rewards within the group of samples for each prompt, applied to the 1e22 models.
Evaluation: MATH-500, AIME24 (mean over 10 seeds), and GSM8K, zero-shot at temperature 0.7, with a grader that reads the final answer from \boxed{}, at Delphi’s 4k context length. The same protocol scores every stage.

Does midtraining raise coverage before any post-training? Before any post-training, we measured pass@k [7], the fraction of problems solved by at least one of k samples, with 128 samples per problem. Unsurprisingly, the base models have pass@1 near zero because they do not produce the \boxed{} format the grader reads. However, their pass@128 increases with compute to 10.4% on MATH-500 at 1e22. Midtraining shifts the whole pass@k curve upward. At 1e22, p33m67 midtraining raises MATH-500 pass@128 from 10.4 to 79.6 and pass@1 from 0.1 to 6.0. The increase holds at every scale (pass@128 is 12.2% at 3e18). On GSM8K the midtrained 2e20 model exceeds the pass@128 of the 1e22 base model (75.9 vs 61.5), and the midtrained 1e22 model reaches 93.6. On AIME24 the base models solve nothing at any scale below 1e22, where pass@128 is 3.3 (one of 30 problems). The midtrained models reach non-zero AIME24 coverage from 9e19 upward, 23.3 at 1e22.

MATH-500 pass@k, base vs p33m67 midtrained

Midtraining raises MATH-500 pass@128 at every scale, from 10.4 to 79.6 at 1e22 (left; ▲ base model, ● p33m67 midtrained model), and shifts the whole pass@k curve upward (right). The left dropdown selects the benchmark. The right dropdown selects the scale of the pass@k curve.

show data (MATH-500)

Is pass@k after midtraining predictable? A logit-linear fit through 3e18–3e20 predicts the held-out 1e22 pass@128 within 1 point and pass@8 and pass@32 within 4 and 6 points, both underpredicted. Rohith’s analysis in #6279 reaches the same conclusion with a shared coverage curve and a per-scale horizontal shift, which predicts MATH-500 pass@5 at 1e22 to within the noise of the measurement. The same procedure on the base models’ GSM8K coverage did not predict 1e22.

MATH-500 coverage of the midtrained models vs compute, fit through 3e20

MATH-500 pass@k of the p33m67 midtrained models increases smoothly with base compute. The dashed line is a logit-linear fit through 3e18–3e20. Open markers are the held-out 1e21 and 1e22 points. Hover over one for the fit’s prediction, the measured value, and the error. Held-out errors of the fit (predicted − measured, in points): pass@128 −3.3 at 1e21 and +0.3 at 1e22; pass@32 −4.9 and −3.6; pass@8 −2.9 and −5.1. The dropdown selects k.

Post-training: SFT Results

In June, Ben ran cold-start SFT on all 27 midtrained checkpoints and the 9 base models with two recipes, Magpie (math-strong) and WildChat (math-weak). MATH-500 pass@1 after SFT increases monotonically with compute, from 2.2% at 3e18 to 45.0% at 1e22 for the best cell, and the mix ranking favoring more math also holds. Magpie SFT on a base model reaches at most 3.6% through 1e21 and 10.6% at 1e22, and WildChat SFT on the 1e22 base reaches 1.6% but when composed with midtraining it reaches 45.0%.

pass@1 after cold-start SFT vs base compute

SFT on midtrained models (solid lines) raises MATH-500 pass@1 to 37–45% at 1e22, while SFT on base models (dashed orange) reaches at most 10.6%. The dropdown selects the benchmark and the SFT recipe.

show data (MATH-500, magpie / wc386k)

We found that the SFT recipe matters less at larger scale, though this is likely con-founded by the fact the larger models saw more math data during midtraining. At scales up to 3e20, the Magpie recipe scores 2 to 6 times the WildChat recipe on MATH-500 for p33m67. At 1e22 the two recipes converge, and the WildChat recipe scores higher on both MATH-500 (45.0 vs 44.2) and GSM8K flexible-extract (64.1 vs 62.4). On non-math (with a lower case 'm') tasks, the 1e22 p33m67 SFT model scores MMLU 58.6, HellaSwag 74.8, and ARC-easy 72.9, in the range of 2–3B-parameter instruct models. Paloma measures how much each checkpoint regressed against its base.

Is accuracy after SFT predictable? Now to answer a question the astute reader might be wondering: if we define 'SFT' as the most light-weight form of post-training, is the accuracy after SFT (and thus post-training) predictable from the midtrained model’s coverage? The results seem to indicate yes! But of course this comes with caveats on the actual SFT mix. Against midtrained coverage the fit works under Magpie: SFT pass@1 against the midtrained model’s pass@128 through the same seven scales predicts 1e21 within 1 point (25.8 vs 25.0) and 1e22 within 4 (47.5 vs 44.2). However, this falls apart for WildChat: the WildChat recipe sits at the format floor through 9e19 (under 1.5 pass@1) and no fit through those points extrapolates: against coverage it predicts 38.5 at 1e22 against a measured 45.0. So perhaps a narrow (but still interesting!) claim we can make is that once the SFT recipe clears the format floor, the midtrained model's pass@128 is a good predictor of the SFT pass@1 , despite model size.

MATH-500 pass@1 after SFT vs midtrained coverage, p33m67

Each point is one scale of the p33m67 ladder: x is the midtrained model’s MATH-500 pass@128, y is its MATH-500 pass@1 after SFT. The dashed line is a logit-logit fit through 3e18–3e20. Open markers are the held-out 1e21 and 1e22 points. Hover over one for the prediction and the error. Magpie: 1e21 +0.8, 1e22 +3.3 points. WildChat: 1e21 −8.8, 1e22 −6.5 points. The dropdown selects the SFT recipe.

Cold-start recipe selection and format effects at small scale

A screen of 4 instruction datasets × 5 learning rates on one anchor checkpoint (9e19 p33m67) selected the two recipes. tulu3 produced the answer format reliably but little math (MATH-500 at most 4.6). Magpie produced the most math (10.2 at learning rate 1e-5). The two WildChat variants scored lowest. Ben kept one math-strong and one math-weak recipe to separate the contribution of the SFT data from the contribution of midtraining.

At small scales, low MATH-500 and AIME24 scores reflect a format failure at the 4k context: the models often do not emit \boxed{} within the budget. GSM8K flexible-extract, which reads the last number in the response, is the least format-sensitive column. AIME24 exceeds zero only at 1e22 (4.9 ± 0.2 for p33m67 with the WildChat recipe). All 129 checkpoints (base, midtrained, SFT, and RL) are public on Hugging Face. The consolidated model map lists them, and penfever/delphi-rl-scaling-laws-6279-evals holds the raw evaluation artifacts.

How does Delphi compare to comparable models after midtraining and SFT? Under the same protocol and 4k context, the 1e22 p33m67 SFT model exceeds Gemma-2-2B-it and Llama-3.2-1B-it on MATH-500 by more than 30 points, matches Gemma-2-9B-it (45.0) at about 1/44 of its pretraining FLOPs, and trails only models pretrained with at least 70× its compute.

Delphi-1e22 SFT vs instruct models under one protocol (4k context)

The Delphi-1e22 SFT model (no RL) matches Gemma-2-9B-it on MATH-500 at 1/44 of its pretraining compute, under one protocol and one 4k context. The dropdown selects the benchmark. Raw is the mean of the three benchmarks.

show data

Post-training: RL Results

What is the gain post RL? Ben’s RL stage starts from the 1e22 p33m67 math-weak SFT checkpoint (MATH-500 45.0, AIME24 4.9, GSM8K 64.1). GRPO with forced thinking on RLVR-MATH [8] with a length penalty reaches 53.4 on MATH-500 (+8.4) and 68.5 on GSM8K (+4.4) at step 140. The same recipe on DAPO-17k [9] for 280 steps reaches 51.6 on MATH-500 and 6.6 ± 0.4 on AIME24 (+1.7), the only run that improves AIME24. In his first 100-step probe, five of six math cells collapsed to low entropy. His stabilized recipe (learning rate 1e-5, gradient clipping at 0.5, no entropy bonus, no KL term) produced the numbers above. An independent reproduction of the RLVR-MATH run in a different framework exceeded these gains (+12.3 on MATH-500, #6915).

RL from the 1e22 p33m67 SFT model, held-out scores

RL raises MATH-500 by 6–8 points and GSM8K by 4 points over the SFT model, and only the DAPO-17k run raises AIME24. The dropdown switches between absolute scores and the change from the SFT model.

Why did AIME24 get worse under RL? The two RLVR-MATH policies score below the SFT model on AIME24. The immediate failure mode in the original runs appears to be truncation: their longest AIME reasoning chains reach the 3,584-token generation limit before emitting a final answer. A length penalty raises AIME24 from 0.0 to 1.5, consistent with mitigating that failure, although both RLVR runs remain below the 4.9 SFT baseline. Truncation is not an inevitable consequence of forced-thinking RL. The cross-framework reproduction uses the same starting checkpoint, RLVR-MATH data, forced-thinking template, length penalty, and 4k budget, but its completions settle around 450 tokens and AIME24 does not regress. The original policy therefore developed a length pathology; why it did so remains unresolved and may involve optimization or implementation dynamics. The DAPO-17k run reaches 6.6 under the same context limit, but it changes both the dataset and the training duration, so it does not isolate the cause. Each condition is one training run, and AIME24 contains only 30 problems. Completion-length and truncation rates, an SFT baseline evaluated with the same forced-thinking template, and a longer-context evaluation are needed to distinguish an evaluation-time length failure from a change in hard-math capability.

Looking across the stages, midtraining did not teach the answer format and moved pass@1 by 6 points. It changed the distribution the model samples from. SFT then taught the format, and RL reinforced what the SFT policy already sampled. Without midtraining, the same SFT started from a model with 10% coverage instead of 80% and reached 1.6 pass@1. The one comparison we did not run is RL from that base → SFT model. I expect it to have little to reinforce, but that is the experiment, not the result.

RL recipe and the 1e23 run

Start: laion/delphi-1e22-p33m67-…-wc386k_lr1e5-sft. GRPO advantages, FSDP2, colocated policy, reference, and rollout, no KL loss, no entropy regularization. An earlier +0.01 entropy bonus raised the entropy from 0.13 to about 11.5 and drove the reward to −1.0 on the math cells.
Forced thinking: the chat template prefills <|start_think|>, and the model generates reasoning inside the tag instead of an empty think block.
Reward: the held-out evaluation’s \boxed{} grader plus a length penalty (weight 1.0) that rewards concise correct answers and penalizes truncation.
Budget: 256 prompts × 16 samples per step, 4k context (512 prompt + 3584 generation tokens), 128 A100 GPUs, about 146 steps. The best exponential-moving-average step is 140.
AIME24 under forced thinking at 4k: AIME24 falls from 4.9 to 1.5 on the RLVR-MATH run because long reasoning chains reach the generation limit before the answer. The 4k context limits every AIME24 result on this ladder.
1e23: Ben Feuer midtrained the 1e23 base model on the K = 0.20 recipe. We ran out of compute at 38% of the budget. After minimal SFT, the 38% checkpoint scored close to the 1e22 model after SFT and RL. In a 100-step RL probe (32 H100 GPUs per cell, about 100 GPU-hours per cell), the reward increased on every dataset, but only training on the test distribution changed the held-out benchmarks. RLVR-MATH moved held-out MATH-500 by +0.008, and RL on IFEval reduced MATH-500 by 0.258. The 1e23 midtraining, SFT, and RL ran on the Marin stack (Levanter and Iris on the TPU and GPU clusters). RL throughput: 100–121 s per step, about 33 steps per hour, 2.2–4.8k generated tokens per second over the whole step, 512 rollouts per step. The 1e23 results describe a partial run.

Reflections: compute-optimal training when post-training matters

What should compute-optimal mean when post-training matters? In July, after most of the runs above were in, I had a long discussion with Will Held and Ben Feuer about this. Marin’s procedure (the Delphi post) picks the compute-optimal point at each budget by pretraining loss. When considering how this should compose with post-training, Will proposed a simplifying assumption I think is basically right: with enough RL, a model’s pass@k at large k converts into pass@1. Under that assumption the quantity to optimize before post-training is coverage on the skills we plan to train. This makes midtraining incredibly important since it seems to be the stage that moves coverage the most for these key downstream tasks. Zhang, Neubig, and Yue showed in a controlled synthetic setting that midtraining followed by RL beats RL alone at equal compute [11]. We can't directly answer that from my findings, but it's clear that mid-training is quite important and we now have results to quanitfy this across various math tasks.

The same assumption changes the compute accounting, because RL pays for inference. An overtrained model is smaller at equal loss so every RL step on it is cheaper in proportion. For Delphi, training 5× past the compute-optimal token budget costs about 3× more pretraining compute to reach the same loss. If coverage behaves like loss under overtraining, a 3× pretraining cost that makes every RL step 5× cheaper would be a good trade. If I were starting over I would probably have midtrained the overtrained checkpoints when I had the chance, though we still learned a lot through this exercise.

Artifacts

Tracking issues: #4547 (midtraining recipes), #6742 (validation contamination), #4549 (solution-primed GSM8K), #6279 (SFT and RL), #4511 (umbrella project).
Interactive scaling report (frozen): delphi-midtraining. Contamination exhibits: dossier.
Checkpoints: base models under marin-community/delphi-*; midtrained, SFT, and RL models under laion/delphi-* on Hugging Face (129 public repositories).
Code: branch midtrain_data (launchers, iso-token controls, in-schedule runs), branch deconamint (contamination scans, decontaminated splits, per-document study), branch aa_fork_rk_downstream (downstream evaluation).

Acknowledgements

Thank you to Ben Feuer who ran the SFT and RL stages and the evaluation pipeline, Rohith Kuditipudi who ran the solution-primed GSM8K evaluation and inspired me to go down this particular rabbit hole, and Kaiyue Wen, Will Held, David Hall, and Percy Liang for helpful discussions. Google’s TPU Research Cloud (TRC) program provided the TPU resources for the midtraining ladder. The LAION, Leonardo, and CoreWeave allocations provided the compute for the SFT and RL stages.

References

DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
Wang, Z. et al. (2025). OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling. arXiv:2506.20512.
Marin Community (2026). Delphi: an open scaling suite. Hugging Face collection. Tracking in marin#1337.
Kumar, R. et al. (2025). Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset. arXiv:2508.15096.
Xu, Z. et al. (2024). Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. arXiv:2406.08464.
Zhao, W. et al. (2024). WildChat: 1M ChatGPT Interaction Logs in the Wild. arXiv:2405.01470.
Brown, B. et al. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787.
Lambert, N. et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.
Yu, Q. et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
Khatri, D. et al. (2025). The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786.
Zhang, C., Neubig, G., and Yue, X. (2025). On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models. arXiv:2512.07783.