RL by Subtraction: Everything We Stopped Needing

On 15 September 2026, TypeSafe announced Jev, a model that returns typed decisions instead of generated text. Give it context and a question; it returns a choice, score or yes/no answer with probabilities. It can't explain its decision.

TypeSafe calls Jev's training method RLCD, or Reinforcement Learning for Calibrated Decisions. Its public AI primer describes RLCD as training for calibrated decisions and probabilities, but does not specify the reward, loss or training procedure.

RLHF, RLVR, GRPO and RLCD change different parts of training. They aren't four steps in one pipeline. I built small toys to compare the methods and see where each one fails.

TL;DR

RLHF learns a reward model from preferences. RLVR replaces it with a verifier. GRPO replaces the learned critic with a group baseline. Jev returns typed decisions with probabilities instead of free-form text, while TypeSafe has not disclosed enough to reproduce RLCD. I compare the objectives, build NumPy toys for the reward signals and GRPO, and probe Jev on 120 support tickets.

These methods aren't a ladder of capability. They expose different assumptions in the reward and optimiser.

The standard diagram leaves things out

The familiar diagram has three boxes: supervised fine-tuning, a reward model, then PPO. It describes one influential recipe, not the start of preference-based reinforcement learning.

Christiano et al. 2017 had no SFT stage and no PPO. It ran A2C and TRPO on Atari and MuJoCo, learning a reward from 5,500 human comparisons of trajectory segments, not text, across the Atari games. Bai et al. 2022 at Anthropic had no SFT stage on human demonstrations either: "our finetuning occurs purely through RL (we perform context distillation, but this is much more like simple prompting)." The clean three-stage shape is Stiennon et al. 2020, canonised a year later by InstructGPT.

The mechanism underneath all of them is the Bradley-Terry model. Given two outputs, the probability a human prefers one is a logistic function of the reward gap, so you fit a scalar by maximising the likelihood of the labels you collected. Ouyang et al. 2022 write it with the pair-count normaliser:

$$\mathrm{loss}(\theta )=-\frac{1}{\left(\genfrac{}{}{0ex}{}{K}{2}\right)}{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \left(\sigma \left(r_{\theta }(x,y_w)-r_{\theta }(x,y_l)\right)\right)\right]$$

The key assumption is easy to miss. The model learns which of two outputs a person preferred, then scores arbitrary outputs, including ones no person ranked or would produce. The loss says nothing about whether those scores generalise.

Christiano's team noticed the economics early. Five thousand labels cost roughly five hours of human work, about $36 at US minimum wage, against about $25 of compute. The expensive part was never the GPUs.

The reward model was always the weak link

Stiennon et al. saw the failure in 2020 and described it without ceremony:

"Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences."

Two years later Gao, Schulman and Hilton gave it a functional form. Holding a 6B "gold" reward model as ground truth and training proxy models from 3M to 3B parameters, they found the true score follows

$$R_{\mathrm{RL}}(d)=d\left({\alpha }_{\mathrm{RL}}-{\beta }_{\mathrm{RL}}\log d\right),d=\sqrt{D_{KL}(\pi \Vert {\pi }_{\mathrm{init}})}$$

The shape is the point. Optimise a little and the proxy and the truth move together. Optimise a lot and they come apart.

I wanted to see this happen in something I could hold in my head, so I built a deli counter. A policy composes a sandwich order from a twelve item menu. The customer's true satisfaction is a fixed hidden function: tastiness per item, a bonus for a coherent sandwich, a quadratic crowding penalty past four items, and a charge per pound over budget. A reward model is then fit to noisy pairwise preferences over orders of four items or fewer.

That last constraint is the entire experiment. Below four items, more really is better, so the reward model correctly learns a positive length coefficient of +0.38. It has never seen a seven item sandwich and has no way to represent the crowding term.

True satisfaction climbs to +1.27, peaks at 4.3 nats of KL from the starting policy, then falls to −9.71. The reward model's score rises from +0.11 to +8.13 and never wavers. If the proxy were all you could see, and in production it is, this run looks like a triumph throughout.

The final policy makes the failure concrete. By the end, 99.9% of its orders were twelve slices of halloumi. No bread. Twenty-eight pounds eighty against a six pound budget. The reward model scores that at +13.59, the highest it ever gave anything, because every slice adds tastiness and length and the model has no term for crowding, or budget, or the concept of a sandwich. The customer rates it −4.36.

Gao's most uncomfortable finding is about the fix everyone reaches for. A KL penalty "increases the proxy reward model score that can be achieved for a given KL divergence, but this does not correspond to a measurable improvement in the gold RM score–KL_RL frontier." Read that carefully. The gold score still improves. It just doesn't improve per unit of drift, so the penalty only makes the gold score "converge earlier," and its effect "is akin to early stopping." It buys you time, not safety. The authors add their own hedge, that the result "could be particularly sensitive to hyperparameters."

Which might explain why nobody agrees on how hard to pull it.

Fifty times, between labs, on the same knob. And the lab at the bottom says its value "might actually be wholly unnecessary."

The best illustration of what a reward signal can do when it points the wrong way is still Ziegler et al. 2019, who flipped a sign by accident:

"One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while still regularizing towards natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form, regardless of how innocuous the starting point was. This bug was remarkable since the result was not gibberish but maximally bad output. The authors were asleep during the training process, so the problem was noticed only once training had finished."

A single sign error, and the KL term kept the output fluent as the reward pushed it in the wrong direction.

Delete the reward model

RLVR removes the learned reward model and uses a verifier instead.

Tülu 3 introduced the term in November 2024 and described the change plainly: "RLVR leverages the existing RLHF objective but replaces the reward model with a verification function." Set the two objectives side by side and one symbol moves.

$$\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{y\sim {\pi }_{\theta }(x)}\left[r_{\varphi }(x,y)-\beta \mathrm{KL}\left[{\pi }_{\theta }(y\mid x)\Vert {\pi }_{\text{ref}}(y\mid x)\right]\right]$$ $$\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{y\sim {\pi }_{\theta }(x)}\left[v(x,y)-\beta \mathrm{KL}\left[{\pi }_{\theta }(y\mid x)\Vert {\pi }_{\text{ref}}(y\mid x)\right]\right]$$

Where $r_{\varphi }$ was a neural network trained on human opinion, $v$ is a program:

$$v(x,y)=\{\begin{array}{ll}\alpha & \text{if correct,}\\ 0& \text{otherwise.}\end{array}$$

Tülu 3 sets $\alpha =10$, not 1, "based on pilot experiments," which is a small detail worth keeping because people quote this reward as binary and it's binary in structure rather than magnitude.

DeepSeek-R1 took the same position and said why, in a sentence that reads like a warning label on everything above:

"Notably, we abstain from applying neural reward models—whether outcome-based or process-based—to reasoning tasks. This decision is predicated on our observation that neural reward models are susceptible to reward hacking during large-scale reinforcement learning."

So the reward is now code. Which means the reward now has bugs.

The best example comes from Tülu 3 itself, in the same paper that introduced the method. Training against instruction-following verifiers at a low KL penalty, they logged this:

User: Measure the length of the given object in cm. Pen. In your response, the letter e should appear 14 times. Assistant: e, e, e, e, e, e, e, e, e, e, e, e, e, e

The verifier counts fourteen e's and pays out. The question isn't answered. Their own caption concedes that the model "does not output meaningful content."

My deli has a rule checker too. An order passes if it has exactly one bread, at least one filling, and costs £6 or less. The policy never gamed it. The trouble was that, for long stretches of training, it gave the policy nothing to learn from.

GRPO samples sixteen orders at a time and scores each one against the group's average. If all sixteen earn the same reward, every advantage is zero and the update does nothing. The share of groups like that traces a U over training.

At the start, every group is like that, because almost no order passes and every reward is zero. As the pass rate climbs, the share falls. It reaches 0% at step 380 in this run, and on the other seeds I tried its lowest point was between 2% and 6%. Then it climbs back to 92%, because the policy now passes almost every time and the rewards are all the same again.

After I published this, Chen Yang pointed out that the U follows directly from probability. If each of $G$ samples passes independently with probability $p$, the chance that all $G$ rewards are the same is

$$P(\text{all equal})=p^G+(1-p{)}^G$$

For any group larger than one, this is lowest when $p=0.5$. With sixteen samples it is about 0.6 when 3% of orders pass, close to zero at 50%, and 0.94 at 99.6%. I plugged this run's pass rates into the formula, and the prediction matched the observed curve with a correlation of 0.92. Most of the difference comes from the chart, which averages over the last 50 steps. Larger groups make mixed rewards more likely, but every extra sample costs another rollout.

So a binary reward only teaches while the policy still sometimes fails. Outside that band the advantages are zero and there is no gradient. DAPO handles this by resampling until each group contains at least one pass and at least one failure.

The verifier avoids the halloumi order. The policy settles on focaccia and turkey: £4.20, every rule passed, true satisfaction +3.00. A real sandwich.

Then it stops improving. The best order that passes every rule is sourdough with two slices of halloumi, exactly £6.00, worth +4.95. A binary verifier gives no further signal when every order in a sampled group passes. Once all the samples pass, their relative advantages are zero.

One method optimised its proxy into a bucket of cheese. The other cleared the bar and lost interest. Neither found the sandwich.

Verifiers also have to run fast enough for RL training. R1's own paper says it plainly: "Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks." In the v1 preprint's comparison table its SWE-Verified score of 49.2 sits below Claude 3.5 Sonnet's 50.8. The binding constraint is latency, not possibility.

Delete the critic

GRPO is easier to describe than to derive. It scores each response relative to the other responses for the same prompt.

LLM PPO often uses a critic, a second network that can be comparable in size to the policy, to estimate expected return and provide a baseline for the advantage. DeepSeekMath removed it. Their stated reasons start with cost, since the critic "is typically another model of comparable size as the policy model," and move through the difficulty of training a per-token-accurate critic when the reward only arrives at the end. The reason that actually matters is the last one. The critic's only job is variance reduction.

The baseline can be any value $B$ that doesn't depend on the action. Its expected contribution to a policy gradient is zero:

$${\mathbb{E}}_{o_t\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(o_t\mid q,o_{<t})B(q,o_{<t})\right]=\left[{\nabla }_{\theta }1\right]B=0$$

The critic is one choice of $B$. A group of $G$ outputs gives a Monte Carlo estimate of expected reward for that prompt. Subtract that mean from each reward:

$${\hat{A}}_{i,t}=\frac{r_i-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})},\mathbf{r}=\{r_1,\cdots ,r_G\}$$

Subtracting the group mean removes the reward shared by the samples, including the effect of prompt difficulty. The remaining differences show which responses did better than their peers.

Drag the group average up to meet the score and the advantage goes to zero no matter how well the answer did. An answer isn't good or bad here. It's better or worse than its siblings, and that's all the gradient ever knows.

That claim is checkable, so I checked it. Running the deli and estimating the same gradient three ways, measuring the spread across the group:

That table is one seed, and the most flattering of the five I ran. Across all five the group baseline cuts the spread to between 36% and 64% of no baseline, and always lands at or just under the critic, between 0.94 and 0.99 of it. So the honest summary is that it matches the critic, with no parameters and no training, rather than beating it. It tracked the expected reward better in four of five seeds. The group mean uses current rollouts; the learned critic has to track a policy that keeps changing. This toy is a bandit, so the critic has no generated prefix to condition on. A per-token critic in a language model has more information and can do better. The point is only that the baseline was never the hard part.

Dr. GRPO makes the connection exact: the group mean baseline is REINFORCE Leave-One-Out up to a scale factor "which can be subsumed into the learning rate without affecting the RL dynamics." A textbook estimator, arrived at by deleting a neural network.

The full objective keeps PPO's clipped ratio and adds a KL term:

$${\mathcal{J}}_{GRPO}(\theta )=\mathbb{E}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\sum _{t=1}^{|o_i|}\left\{\min \left[{\rho }_{i,t}{\hat{A}}_{i,t},\mathrm{clip}\left({\rho }_{i,t},1-ϵ,1+ϵ\right){\hat{A}}_{i,t}\right]-\beta D_{KL}\left[{\pi }_{\theta }\Vert {\pi }_{ref}\right]\right\}\right]$$

Two structural details are readable straight off the brackets. The KL sits inside the token sum and is subtracted from the loss, where PPO-RLHF folds it into the reward. That matters more than it sounds: as a per-token reward it flows through the advantage estimator and gets summed over the trajectory, so longer responses accrue more penalty. As a loss term it's averaged and never touches the advantage at all.

And the clipping is the same pessimistic bound PPO always used.

Push the ratio past $1+ϵ$ with a positive advantage and the objective flattens. Past that shelf, moving further from the old policy stops paying. Flip the advantage negative and the shelf moves to the other side.

The response-length normaliser $\frac{1}{|o_i|}$ has a cost. Dr. GRPO shows it's not neutral:

"For positive advantages […] this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages […] longer responses are penalized less due to their larger $|o_i|$, causing the policy to prefer lengthier responses among incorrect ones."

Short when right, long when wrong. That's a mechanical property of the divisor, and it produces exactly the curve that got read as emergent reasoning: response length climbing steadily through RL training. Some of that climb is the optimiser, not the model learning to think. R1's own bibliography cites the paper making this argument.

DeepSeekMath's GRPO and DeepSeek-R1's GRPO are different equations. The first is token-level, the second sequence-level. Same name, two algorithms, and the second sits closer to GSPO than to the first.

Replace prose with a typed decision

Jev returns a typed decision rather than a paragraph. The SDK makes that contract explicit. NoulAnswer carries one yes-probability; ChoiceAnswer and ScoreAnswer include probability distributions and a confidence value. Constraining the output type prevents malformed answers. It does not establish that the probabilities are calibrated.

The type contract is straightforward. Calibration is harder to judge.

A scoring rule is proper when a forecaster maximises their expected score by reporting their true belief. Gneiting and Raftery formalised the family. The log score is strictly proper; maximising it corresponds to maximum likelihood. The Brier score is the mean squared error between a stated probability and the outcome. It is also strictly proper and bounded.

A binary RLVR reward isn't a proper scoring rule at all. It's indifferent to confidence. Answer correctly while certain and answer correctly while guessing, and it pays the same.

RLCR, published in July 2025, gives a concrete version of this idea. The model emits an answer and a confidence $q$, and the reward is correctness minus the Brier score on that confidence:

$$R_{\text{RLCR}}(y,q,y^{\ast })=1_{y\equiv y^{\ast }}-{\left(q-1_{y\equiv y^{\ast }}\right)}^2$$

Their Theorem 1 proves expected reward is maximised when $q$ equals the true probability of being right, and they show it generalises to any bounded proper scoring rule. The bound is essential. Swap in the unbounded log score and, as the paper puts it, "there exist ranges of $p_y$ where the model receives higher expected reward by reporting answers with lower true correctness probability."

The results are the argument.

On HotpotQA, RLVR reaches 63.0% accuracy with ECE 0.37; RLCR is slightly less accurate at 62.1%, with ECE 0.03. On MATH, the accuracy scores are nearly tied, while RLCR again has lower ECE. These are results on the reported datasets, not a guarantee about another model or task. If probabilities drive routing or triage, measure calibration as well as accuracy.

Accuracy alone misses whether confidence tracks correctness.

My deli has an acceptance head that does this in forty lines: emit P(customer accepts), train on Brier, and because the environment defines the true acceptance probability, measure real calibration error rather than a binned approximation of it. It lands at Brier 0.150 and ECE 0.043.

The toy lets me compare stated probabilities with the environment's known acceptance rates. On production data, I can only compare stated probabilities with observed outcomes across groups. A reliability diagram shows whether those frequencies track the model's claims.

TypeSafe's docs describe calibration as matching probabilities to outcome frequencies across groups. They do not name a scoring rule, publish the training loss, or report an ECE or reliability diagram. The goal is clear. The method is not.

What the scores don't tell us

That is a clean story. I don't fully trust what aggregate scores leave out.

Start with the finding that should unsettle anyone quoting an RLVR benchmark. Spurious Rewards found GRPO improved MATH-500 on Qwen2.5-Math-7B by 21.4 points using randomly assigned rewards, against 29.1 for ground truth. It fails on Llama3 and OLMo2, which suggests the training surfaces a pretrained prior rather than teaching anything. If random rewards recover most of the benchmark gain on one model, accuracy alone can't tell us what the reward taught.

Base models also beat RLVR-trained ones at large pass@k. Yue et al. conclude that an RLVR model's reasoning abilities "originate from and are bounded by the base model." RL may be sharpening a distribution rather than extending it.

I wanted to test Jev on my own task, so I spent about two tenths of a cent on 240 questions.

The setup is 120 support tickets with two questions about each. The first, "is this customer unhappy?", is answerable from the message. The second, "should this be escalated to tier three?", depends on a routing policy I never showed the model. Same text, same call, one question answerable and one not.

On the answerable question Jev is excellent. 100% accuracy, Brier 0.006, ECE 0.054.

On the unanswerable one it's 40% accurate and 84% confident. Sixty of the 120 answers came back below 0.1, which is to say near-certain. Nineteen sat in the 0.4 to 0.6 band where an honest "I cannot know" would live.

The mechanism shows up in the averages. Mean P(escalate) was 0.287 for the unhappy customers and 0.045 for the pleased ones. Jev didn't decline the question it couldn't answer. It quietly answered a different one, about tone, and reported the substitution as confidence. A second run on a fresh sample reproduced it: ECE 0.047 answerable against 0.344 unanswerable.

The routing policy I chose is arbitrary and uncorrelated with tone, so near-chance accuracy is expected. Accuracy was never what I was testing. I wanted to know whether the probability would tell me it was guessing. It didn't. Other independent probes report a similar split. Archer Hume measured ECE 0.0313 on an MMLU sample. The jev-ood-calibration study reported ECE 0.325 on a priority task whose policy was missing from the input. A Banking77 probe found 29 misclassifications at confidence 1.00 among 3,080 support queries. These are separate experiments with different setups. In my probe, Jev was miscalibrated when the ticket lacked the rule needed to answer.

In my probe, calibration held when the ticket contained the answer and broke when it depended on a policy I withheld. That is a different question from the launch chart's type-safety claim.

TypeSafe's docs also distinguish calibration from individual correctness: "Calibration is measured across groups of predictions" and does not guarantee one answer is right. On the launch chart's 0% hallucination figure, the company says:

"Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots."

The marketing is less careful than the docs, though the caveats are disclosed if you scroll. The blog claims "40x-200x faster for the same levels of frontier intelligence for System One shaped queries," and that last clause carries more weight than the number. The comparison workflows "were made by individuals on our model capabilities team, so some bias could exist," and the LLM baselines run through a wrapper TypeSafe admits "tends to be slower and more expensive." Their separate evals page adds that the reference labels are not ground truth but "an average of the responses of GPT-6 Astra and Claude Fable 5.1."

The objection I keep returning to isn't about calibration. Simon Willison puts it as a return to black boxes: "put in all the text you want, the only thing you're going to get back is a floating point number. If Jev marks something as spam, which content signals tipped it off?" He follows it with the case that should worry you: "I really hope nobody uses Jev to rank job applicants, that floating point number could conceal all manner of unseen bias baked into the models." Every other objection here asks whether the numbers are right. That one asks whether you could ever find out.

One point I agree with: many production LLM calls are classification tasks. A typed decision model may be a better fit when software needs a bounded answer rather than generated prose. A week-old open replication on Qwen3.5 got within 3.5 Brier points of Jev, which suggests the idea travels further than the moat does. The idea being right doesn't make the chart true.

Where this leaves you

Ask for a reliability diagram alongside accuracy. A 63% accuracy score and ECE 0.37 describe different things; neither number tells the whole story alone.

Measure calibration on your own data. In my probe, ECE was eight times higher on the unanswerable question than on the answerable one, in the same API call.

If you are implementing GRPO, read Dr. GRPO before the blog posts. The length normaliser and the standard deviation divisor are both contested, and the two leading fixes disagree with each other about what to do.

Do not quote a GRPO memory saving. No primary paper publishes one. The arithmetic is defensible, roughly 44% of weight and optimiser state, but it's arithmetic, not a citation, and it ignores that sampling $G$ responses per prompt costs you elsewhere.

Test your verifier like production code. If it rewards the wrong behaviour, optimisation will find and repeat that mistake.

Each method moves the failure somewhere else. Jev's typed output is easy to parse; its confidence still needs to be checked on the task you plan to automate.

Part 2 asks whether a task reward can train the internal update that decides what a model keeps in memory.

Sources

• Deep Reinforcement Learning from Human Preferences: Christiano et al., 2017
• Proximal Policy Optimization Algorithms: Schulman et al., 2017
• Fine-Tuning Language Models from Human Preferences: Ziegler et al., 2019
• Learning to Summarize from Human Feedback: Stiennon et al., 2020
• Training Language Models to Follow Instructions with Human Feedback: Ouyang et al., 2022
• Training a Helpful and Harmless Assistant with RLHF: Bai et al., 2022
• Scaling Laws for Reward Model Overoptimization: Gao, Schulman and Hilton, 2022
• DeepSeekMath: Shao et al., 2024, the GRPO objective
• Tülu 3: Lambert et al., 2024, coins RLVR
• DeepSeek-R1: DeepSeek-AI. The v2 / Nature version; v1 words the reward-model passage differently
• Understanding R1-Zero-Like Training: Liu et al., 2025, Dr. GRPO
• DAPO: ByteDance Seed, 2025
• Does RL Really Incentivize Reasoning Capacity Beyond the Base Model?: Yue et al., 2025
• Spurious Rewards: Shao et al., 2025
• Group Sequence Policy Optimization: Qwen Team, 2025
• Beyond Binary Rewards: Damani et al., 2025, RLCR
• Strictly Proper Scoring Rules, Prediction, and Estimation: Gneiting and Raftery, 2007
• Introducing System One Models and Jev: TypeSafe's launch description of Jev
• TypeSafe AI primer: its public account of RLCD
• TypeSafe workflow evals: the reference-label methodology behind the speed claims
• Noul primitive: the typed output contract
• Jev's Architecture Unmasked: independent architecture and calibration probe
• Jev out-of-distribution calibration study: raw results on held-out policy questions
• Jev: Simon Willison, 2026
• JevBench: independent Banking77 run, including confidence-1.00 failures

Companion code for every experiment here, including the live Jev probe: github.com/Serendeep/rl-by-subtraction. make test covers the math, make charts regenerates the figures.