In the last post I talked about running inference with small language models, where I tinkered with a few quantized versions of a model, studied and tried to explain the different prefill and decode speed behaviour through llama.cpp style benchmarks, and then landed on a question, prefill and decode speed gains are desirable but not at the cost of quality, which is what pushed me to build my own benchmark datasets for testing a model, something I could trust rather than borrow. Other benchmarks out there are completely fine if what you want is the delta uplift from post training, that is a different question. You can read that one here.
Now say I have a model in place that I trust the numbers on. Can I trust it to run inside a harness, something like Pi, at 70 to 80 percent of the capacity Claude Sonnet gives me on the same harness. That question is what pulled me into this next rabbit hole.
There is a way of looking at self improving agents where you split the agent into 2 parts, Agent = Model + Harness/Scaffold. If you want the agent to get better at something, you either improve the model, or you improve the harness, the ecosystem sitting around it, while keeping the model frozen. Both are real paths and both are being poked at right now, as small toy sized experiments rather than one big attempt.
This post is the model side. There is a second track running alongside it on the harness side, a frozen 8 billion parameter model with the harness getting built up one layer at a time, agent loop, tool calls, memory, added one piece at a time and measured after each piece. That track will show up on this same page too, once it has enough runs behind it to say anything.
Why this model, why this size
The model is Qwen2.5 1.5B Instruct. Picked small on purpose. At this size a full training run and evaluation cycle is cheap and quick, which means several attempts fit into a couple of hours a night instead of needing a weekend each. There is also a plain compute constraint behind that choice, my own machine has a 5060 Ti with 16 gigabytes of memory, which is slow enough at this that I depend on rented cloud compute instead.
Before committing to it I checked that the model actually has room to improve on the problems I wanted to use, since training a model that already gets everything right or a model that gets nothing right teaches you nothing either way. I also wanted to start from first principles rather than jump straight to agentic coding traces, so this first pass uses plain math word problems, cheap to check, cheap to reason about, a way to see whether the whole setup works before pointing it at anything I actually use day to day.
The dataset
Training uses GSM8K, grade school level word problems, 1500 of them, frozen once before any arm started training. Every training step samples 8 answers per problem, so that rollout budget, 8 attempts per problem, is the same across all 6 arms.
Evaluation runs on 2 sets. In domain evaluation stays on GSM8K, 32 attempts sampled per problem at every checkpoint, since more attempts give a steadier read on how reliable the model actually is. Out of domain evaluation moves to MATH-500, a harder competition style math set the model never sees during training, restricted to its 2 easiest difficulty levels so the untrained model has some chance on it at all. Out of domain sampling drops to 16 attempts per problem, since the harder set is more expensive to evaluate and 16 still supports a first try and an 8 try number, just not a 32 try one.
A few terms before the 6 runs
These come up a lot below, worth naming once plainly rather than leaving them as acronyms.
GRPO is Group Relative Policy Optimisation. Sample several answers to the same problem, judge each one against how the whole group of answers did, and push the model toward the ones that beat the group. No separate scoring model needed, the group itself is the yardstick.
RFT here means rejection sampling fine tuning. Sample several answers, keep the ones that check out against the answer key, and train the model directly on one of them with ordinary next token prediction, the same kind of loss used in plain supervised training.
OPD here means on policy self distillation. The model writes an answer on its own, and a better informed version of that same model, not a separate larger model, grades that answer token by token, rather than handing the model a target answer to copy directly.
The 6 runs
All 6 start from the same untrained model, use the same problem set, and the same rollout budget per training step, 8 sampled answers per problem. 3 families, 6 arms.
Arm 0, untrained, baseline. No training at all. Every other number in this post gets read against this one.
Arm 1, shortest correct, no brake, rft_shortest. This is rejection sampling fine tuning. Sample 8 answers to a problem, keep the ones that check out, train on the shortest one with ordinary next token prediction. The loss is
L(θ) = − Σ_t log π_θ(y*_t | y*_<t, x)
where y* is the shortest answer among the group that passed the checker, x is the problem, and the sum runs over the tokens of that one selected answer.
Arm 2, shortest correct, braked, rft_shortest_braked. The same as arm 1, with one thing added, a term pulling the weights back toward the starting model.
L(θ) = − Σ_t log π_θ(y*_t | y*_<t, x) + β · D_KL( π_θ( · | x) ‖ π_ref( · | x) )
π_ref is a frozen copy of the model as it was before any training started, β is 0.05 here. Arm 1 has no such term at all. That is the only difference between the two, on purpose, so whatever separates them is the brake and nothing else.
KL from the frozen starting policy, tracked over training. Arm 1 has no such term at all so it sits flat at 0 by definition. Arm 2’s smoothed line drifts up from near 0 to a little over 0.003 nats by the end, individual training steps spike higher, briefly past 0.01 a few times in the back half. The brake is doing something measurable, it is just a small something on this problem set at this size.
Whether that small something changed anything else, tokens or pass@1, is barely visible either. Arm 1 and arm 2 end within a couple of tokens and about a percentage point of each other on most of the numbers further down. At this scale the brake looks present but weak, not absent and not dominant.
Arm 3, vanilla GRPO, grpo. For each problem, sample a group of 8 answers, score each one correct or not, and compute how each one did relative to the group.
A_i = (r_i − mean(r_1, ..., r_8)) / std(r_1, ..., r_8)
Then push the policy toward answers with a positive A_i and away from the rest, using a clipped update so a single batch cannot move the weights too far in one step.
J(θ) = E[ min( ρ_i(θ) A_i, clip(ρ_i(θ), 1 − ε, 1 + ε) A_i ) ]
ρ_i(θ) is how much more or less likely answer i has become under the updated weights compared to the weights that generated it, ε is 0.2. No pull back toward the starting model here, this family gets its stability from the clip instead of a KL term.
Arm 4, GRPO with a length penalty, 2 strengths, grpo_length_penalty. Same family as arm 3, same advantage and the same clipped update above, just built on a different reward. The reward for a correct answer now also depends on how long it is.
r_i = 1 − λ · L_i if answer i is correct
r_i = 0 if answer i is not correct
L_i is the answer’s token count divided by the maximum allowed length, so it sits between 0 and
- λ has to stay under 1, otherwise a short wrong answer can outscore a long right one and the model learns to just guess short. 2 versions of this arm ran, one at λ = 0.2, one at λ = 0.5, everything else identical between them so the only thing that changed is how hard the length pressure pushes.
Arm 5, on policy self distillation, opd. The student writes an answer on its own. A teacher, which is the same starting weights but with the correct final answer, not the reasoning that led to it, placed in its prompt, reads the student’s own tokens and grades each one. The loss pulls the student toward the teacher token by token.
D_KL( π_s( · | x, y_<t) ‖ π_t( · | x, hint, y_<t) ) = Σ_v π_s(v) [ log π_s(v) − log π_t(v) ]
summed over every generated token in the batch. The same forward pass also gives the student’s own entropy at each position.
H(π_s) = − Σ_v π_s(v) log π_s(v)
The teacher and student are the same weights and the same size, the teacher is only better because it gets to see where it is going. That also means 1 frozen copy of the model, kept in memory, covers both this arm’s teacher and arm 2’s brake, so the extra cost of running 6 arms instead of 4 is smaller than it sounds.
Left, the actual loss being minimised, reverse KL from student to teacher, falling across the run, steepest in the first 25 to 50 steps, slower and a little bumpy after that. Right, the student’s own token entropy across that same run, noisy step to step but flat on average, no rise, no fall. The loss the model is chasing keeps improving while the spread of the student’s own token probabilities barely moves, one way to read why opd’s accuracy gain stays small even as its reverse KL keeps dropping, the thing being optimised and the thing pass@1 depends on are not the same quantity.
What keeps the comparison fair
Every arm trains on the same 1500 problems, the same amount of generated text, and gets evaluated the same way, several attempts per problem rather than one, since checking a single attempt tells you almost nothing about how reliable a model actually is. The measure used throughout is
pass@k = 1 − C(n−c, k) / C(n, k)
averaged across problems, where n is how many attempts were sampled per problem, c is how many of those came back correct, and k is the number of tries actually being asked about. Generating exactly k attempts and checking whether any worked would be a much noisier number, so n stays well above k everywhere, 32 attempts per problem in domain, 16 out of domain.
The problem pool itself was frozen once, before any arm started training, using the untrained
model. Every candidate problem got 8 rollouts from that untrained model, and only problems where
between 1 and 7 of those 8 rollouts came back correct made it into the training pool. Look back
at the advantage formula from arm 3, A_i = (r_i − mean(r)) / std(r). If all 8 rollouts on a
problem are correct, the mean reward is 1 and every r_i is also 1, so every advantage comes out
- If all 8 are wrong, the mean is 0 and so is every advantage. Either way the group has nothing to teach, no rollout gets pushed toward or away from anything. Filtering out both extremes and keeping only problems where the untrained model sometimes got it right and sometimes did not is what leaves every kept problem with a group that actually disagrees with itself, and it is the same pool for every arm here, rft_shortest and opd included, not only the GRPO family that computes an advantage this way.
The build, the crash, the fix
All 6 arms do full parameter training, every weight in the model updates, no smaller add on adapter standing in for it, on a rented H100 with 80 gigabytes of memory, billed by the hour. Once everything is warmed up the pace runs at a bit under a minute per training step. Total spend across the whole block of runs, including a couple of restarts, came to about 45 dollars.
Partway through one of the runs it crashed with an out of memory error, asking for another 4 gigabytes on a card that already had 77 out of 80 gigabytes in use.
My first fix was to split the heaviest computation into smaller pieces and only stitch them back together at the end before running the learning step once on the whole thing. That changed nothing. Peak memory stayed exactly where it was. The reason is that the framework underneath keeps everything it might need for the learning step regardless of how the forward pass was chopped up, so splitting the forward pass and reassembling it before the learning step buys nothing at all. The fix that actually worked was to run the learning step on each small piece immediately, before moving to the next piece, so nothing ever needs to be held onto for the whole batch at once. That single change took the peak memory this step used from about 35 gigabytes down to about 15, with no change to what the model was actually learning, same loss, same gradient, just computed piece by piece instead of all at once.
The baseline
Before any training, the untrained model on the in domain problems gets 58 percent right on a single try, 91 percent right within 8 tries, 97 percent right within 32 tries. On the harder out of domain set it is at 61 percent on a single try and 88 percent within 8 tries.
What came out of it
A few things stood out enough to write down, along with what I think might be causing each one. None of this is settled, every arm here is 1 training run, not several, so treat the direction as a lead worth checking again rather than a proven effect.
The training reward for the reinforcement learning arms bounced around with no visible trend, staying somewhere between 0.6 and 0.8 the whole way through. That is expected, since each step scores a fresh random batch of 8 problems and the spread between batches is wide enough on its own to hide any slow underlying trend.
Training reward for each reinforcement learning arm, noisy the whole way through, no clear upward or downward trend.
Looking at answer length against first try accuracy over the course of training, rft_shortest and rft_shortest_braked both shrink steadily, going from around 283 tokens down to somewhere in the mid 260s by the end. Vanilla GRPO grows instead, ending up around 288 tokens, and its first try accuracy climbs the most of any arm, from about 58 percent to about 64 percent. grpo_length_penalty also grows a little in length early on then settles back down close to where it started, while its accuracy climbs almost as much as vanilla GRPO. opd grows by about the same amount as vanilla GRPO, ending around 288 tokens too, but its accuracy barely moves, climbing from about 59 percent to about 60 percent, roughly 0.6 of a percentage point. Spending the same extra length as vanilla GRPO and getting a much smaller return on it is worth sitting with.
One way to read the direction of that length change is to look at what each loss actually
optimises against. rft_shortest and rft_shortest_braked train directly on the shortest correct
rollout out of the 8 sampled, every single step, with ordinary cross entropy pulling the model
toward that one sequence, and a loss that repeatedly points at a short target has a direct
reason to shrink. GRPO has no target sequence at all, it only pushes probability toward whichever
rollouts already scored above the group mean and away from the rest, nothing in that update cares
about length one way or the other, so whatever the group happens to reward early on is what
grows. grpo_length_penalty is the one arm that puts length directly into the reward,
r_i = 1 − λ · L_i for a correct answer, so it has an explicit reason to come back down once accuracy is
secure, roughly the shape the plot shows. opd is different from all 3, its loss is a per
token match to a teacher that already has the final answer sitting in its prompt, not a match to
one short target and not a scalar reward shaped by length, so nothing in its objective points
toward short completions the way rft_shortest’s does.
How answer length and first try accuracy move together over the course of training, 1 line per arm.
The same length story looks different tracked step by step instead of only start to end. Tokens per solved, total tokens generated on an evaluation pass divided by how many problems it actually got right, is a joint measure, it moves with both the token cost and the accuracy at once, which is the question I had sitting with this plot the first time I looked at it. Every arm starts close together, in the upper 530s, and every arm brings that number down at least a little since every arm gets at least some more accurate over training. grpo_length_penalty at λ = 0.5 comes down the furthest, ending near 508. opd comes down the least of any arm, ending near 544, barely below where it started, the same read as the frontier plot above, an arm that spends length without buying much of it back.
Median tokens on correct answers, left, against tokens generated divided by problems solved, right, both tracked over training. The right panel is length weighted by how often the arm is even getting the problem right, not length on its own, which is the part that was not obvious to me from the axis label alone.
Token level entropy, which is a way of measuring how undecided the model is at each position it generates, moves in opposite directions for different families. Worth being precise about what the 2 panels below actually show, both are measured at evaluation checkpoints during training, not during training itself, and training only ever touches GSM8K, so the MATH-500 panel is entropy on a set none of these arms ever trained on, checked periodically to see how behaviour on unseen problems shifts while the model only ever learns from GSM8K.
On the GSM8K panel, entropy climbs for both rft_shortest and rft_shortest_braked across training. It falls for vanilla GRPO and both grpo_length_penalty runs. opd stays close to where it started the whole way through. Going in, I expected rft_shortest and rft_shortest_braked to be the ones narrowing the model’s options the most, since they train on one single self generated answer at a time with plain cross entropy. What actually happened points the other way, at least at this scale and this training length.
One way to make sense of that is to look at what each update actually touches inside a batch. rft_shortest only ever takes a gradient from the one rollout it kept, the other 7 sampled rollouts for that problem are thrown away and never enter the loss at all, so nothing in the update explicitly pushes probability away from any of the paths the model did not pick. GRPO scores the whole group of 8 against each other every step and pushes probability away from every rollout that landed below the group mean, correct or not. Repeating that suppression every step for most of a run this long is a lot of pushing away, against rft_shortest which never pushes away from anything, it only ever pushes toward the one target it liked. opd sits in the middle of both mechanically, its loss matches the student to a teacher token by token rather than picking one target sequence or scoring a whole group, and its entropy sits in the middle too, close to flat, neither climbing like rft nor falling like grpo. The MATH-500 panel splits across the same families in a similar pattern, which is worth noting precisely because none of these arms ever saw a MATH-500 problem during training.
How undecided the model stays at each generated position, measured at evaluation checkpoints across training. GSM8K on the left is the training set. MATH-500 on the right is held out, never trained on, evaluated only to see how training on GSM8K alone shifts behaviour on unseen problems.
The gap between how often the model gets a problem right within 32 tries and how often it gets it right on the first try tells a similar story. For rft_shortest and rft_shortest_braked this gap widens as training goes on. For vanilla GRPO it narrows steadily. grpo_length_penalty narrows it too, a little less sharply. opd stays roughly flat. A narrowing gap on its own does not say whether that is because first try reliability genuinely improved or because the wider coverage got smaller, and here it looks like a mix of both for vanilla GRPO, mostly the former.
How often the model gets a problem right within 1 try, 8 tries, and 32 tries, tracked across training, along with the gap between the widest and narrowest of those.
Worth checking that widening gap against how much confidence the numbers actually support. Final checkpoint pass@k on the evaluation set, with a 95 percent bootstrap confidence band around each line, tells a more careful version of the same story. At k = 1 the arms are spread out enough to separate, the grpo family and opd sit clearly above rft_shortest and rft_shortest_braked. By k = 32 almost every band overlaps almost every other band, rft_shortest_braked edges out the rest at 0.98 against grpo’s 0.95, a 3 point gap sitting inside overlapping confidence intervals on 100 evaluation problems. The direction across checkpoints, rft_shortest_braked climbing from laggard to leader while grpo’s lead erodes, is a steadier signal than that one final number, but a single final number this close, on 1 seed, is not something to lean on by itself.
pass@1, pass@8, and pass@32 for every arm at its last checkpoint, band is a 95 percent bootstrap confidence interval over the 100 evaluation problems. The bands overlap almost entirely by k = 32, which is the honest version of any claim about which arm pulls ahead at high k.
Answer level diversity, counting how many distinct final answers show up across the attempts on one problem, drops for vanilla GRPO and both grpo_length_penalty runs as training goes on, rises for rft_shortest and rft_shortest_braked, and opd sits in between the two, ending a little below where it started, the same split the entropy mechanism above points at.
How many different final answers show up across the attempts on one problem, tracked across training.
On the out of domain set, vanilla GRPO and grpo_length_penalty post the largest single try accuracy gains over the untrained baseline. At 8 tries though, opd has the largest gain of any arm, ahead of every reinforcement learning variant. So which family looks best out of domain depends on whether the question is about the first try or several tries.
opd’s own gain is also the most lopsided of the 6 arms between the 2 ways of counting it. At 1 try its in domain gain and out of domain gain sit close together, about 1.4 points and 1.7 points over baseline. At 8 tries the gap opens up, in domain gain drops under half a point while out of domain gain grows to about 2.4 points, the widest split between the 2 axes of any arm here, not the most even one, even though the 1 try picture alone makes it look that way.
How much each arm gained over the untrained baseline in domain against out of domain, at 1 try on the left and 8 tries on the right.
On whether a correct answer tends to be shorter than a wrong one to the same problem, this only holds up clearly for grpo_length_penalty and opd. For vanilla GRPO, rft_shortest, and rft_shortest_braked, correct and incorrect answers to the same problem come out close to the same length, sometimes even the wrong one is shorter.
Whether a correct answer tends to be shorter or longer than a wrong answer to the same problem, 1 distribution per arm.
On whether the shortening rft_shortest and rft_shortest_braked show in domain carries over to the out of domain set, it does, both arms get shorter on both sets by a comparable amount. opd does the opposite of shortening, it grows on both sets by a comparable amount too. Vanilla GRPO barely changes in domain but grows a little out of domain.
Whether a change in answer length in domain shows up as the same kind of change out of domain.
Every rounded percentage above comes out of one table, final checkpoint numbers for all 6 trained arms plus the untrained baseline, across both evaluation sets, GSM8K token cost, and GSM8K entropy, colour coded against the baseline in the same row. It is the plot I go back to most while writing this, since every other figure here is one slice of what this table holds together in one place.
Raw values in each cell, colour is the fractional change against the untrained baseline row. The GSM8K entropy column lines up with the trajectory plots above it, green for both rft arms and opd, red for grpo and both grpo_length_penalty runs, the same split the training plots show, just at 1 final point instead of across training.
What this set of 6 runs cannot tell
A few things by design. Both rft_shortest and rft_shortest_braked pick the shortest correct answer to train on, so nothing here separates whether the changes seen are caused by training on a short answer specifically or just by fine tuning the model on any one of its own answers at all. Answering that needs a seventh arm that picks a random correct answer instead of the shortest one, which was left out on purpose to keep this block at 6 runs.
Rejection sampling and vanilla GRPO also differ from each other in more than one way at the same time, how the training example gets picked instead of a whole group being scored, which direction any KL term pulls, whether wrong answers get used at all, and what kind of brake if any is present. Any comparison between the 2 families in this post is describing what each one does, not explaining which of those differences is responsible.
And everything above comes from 1 seed per arm, run once, for a little under 200 training steps on a model chosen specifically because it is small. Whether any of these directions hold at a larger model, a longer run, or a second random seed is the next thing to check, not something this post can answer on its own.
Where things stand
6 runs, 1 seed each, done. A second seed on the arms with the more surprising direction is the next thing planned before treating any of the above as more than a lead. The harness side track mentioned at the top is running in parallel and will get its own write up on this same page once it has something to show.
One more thing on the list, separate from either track above. The same idea behind opd, a model generating something on its own and then learning from a graded version of its own output, is something I want to try on my own coding traces next. And that is where it would become useful to me day to day. Whether any of the patterns in this post carry over from math problems to code is genuinely unknown to me right now, it is on the list to try, not something I have started.
If any part of this was useful to you, or something in here does not hold up, I would like to hear about it either way.