The Unlearnability Phenomenon in RLVR for Language Models — Reading

GRPO learns from a prompt only when its rollouts disagree, so the natural assumption is that once a hard problem starts producing some correct rollouts, RL will eventually learn it. This paper shows that's often false. A large share of hard examples receive positive rewards throughout training yet never improve. The authors rule out the usual optimization explanations and argue the model has flawed representations for these examples, which RL can't repair.

It's a findings paper: no new method, just a phenomenon, a process of elimination, and a diagnosis.

Setup

Training is GRPO with dynamic sampling (from DAPO): prompts whose $k$ rollouts all get the same reward are filtered out, since their advantages are all zero:

\hat{A}_{i} = \frac{r _{i} - mean ( r _{1} , \dots , r _{k} )}{std ( r _{1} , \dots , r _{k} )}

Examples with initial success rate of at least 10% are easy. The rest are hard, split by what happens during training:

Learnable: reward improves steadily.
Unlearnable: pass@1 stays below $0.1$ at the end of training (32 samples), despite at least one correct rollout during training.

Groups are intersected across three independent runs to reduce noise. Among hard examples, unlearnable ones are roughly as common as learnable ones:

Model	Unlearnable (%)	Learnable (%)	Never a positive reward (%)
Qwen2.5-0.5B	30.2	25.6	23.5
Llama-3.2-3B-Instruct	21.9	31.6	37.7
Qwen2.5-3B	16.7	14.2	47.2

What they rule out

Too few positive rollouts. Forcing exactly 1 correct and 7 wrong rollouts per group (oversampling with replay) doesn't close the gap. Neither does a rollout group of 64, nor even SFT on correct solutions generated by Qwen2.5-7B. The SFT result is the strongest evidence that the problem isn't specific to RL.
Clipping and KL. Correct rollouts on unlearnable examples are not low-probability under the initial model, clipping rates are similar across groups, and clip-higher or removing KL changes nothing.
Gradient interference (Appendix A.3). Gradients of correct and incorrect rollouts on unlearnable examples stay aligned rather than opposing each other.

The diagnosis: flawed representations

Gradient outliers. For each example, they average gradients over its correct rollouts (through a fixed random LoRA adapter) and compute cosine similarity between examples. In my notation, with $g_{i}$ the averaged gradient for example $i$ :

sim (i, j) = \frac{g _{i}^{⊤} g _{j}}{∥ g _{i} ∥ ∥ g _{j} ∥}

Unlearnable examples are outliers: average similarity within the group is about $0.46$ , versus $0.67$ for learnable and $0.75$ for easy examples. What the model learns elsewhere doesn't transfer to them.

Lucky answers. GPT-5-mini rates the reasoning behind correct answers on a 0 to 5 scale. On unlearnable examples, correct answers often come from broken reasoning, and the quality gap widens over training.

Augmentation doesn't help. GPT-5-generated similar problems and subproblems are learned fine (subproblems even faster than learnable examples) but nothing transfers to the originals, and validation pass@16 drops after about 20 steps. Most striking: near-duplicates that differ only in their numbers have gradient similarity around $0.3$ with the original. Semantically similar isn't similar in optimization space.

Mid-training helps gradient similarity. OctoThinker models (Llama-3B mid-trained on 20B tokens) show higher gradient similarity on difficult MATH examples than Llama-3.2-3B-Base.

Connection to the Reasoning Boundary Paradox

Both papers study RLVR through gradient dot products. Here, similarity is cosine-normalized, while the Reasoning Boundary Paradox uses the raw kernel $K$ . They describe different failures, though:

The Reasoning Boundary Paradox is about losing solutions: low-likelihood correct answers get squeezed out. Here, correct rollouts on unlearnable examples aren't low-likelihood, and the failure is never gaining the example at all.
Appendix A.3 tests the Reasoning Boundary Paradox's interference mechanism and finds little evidence of it for unlearnable examples. That conclusion depends on whether the gradients include the advantage's sign, which the paper doesn't make clear. If they're raw log-probability gradients, near-parallel correct and incorrect gradients combined with zero-sum advantages would mean the updates cancel, the opposite conclusion.

My take

The elimination is careful: three seeds, several interventions per hypothesis, and an SFT control. The near-duplicate result is genuinely surprising. But the headline claims go further than the evidence:

"Representation issue" is inferred, not measured. The evidence is gradient similarity through a random LoRA adapter and LLM-judged reasoning quality.
A simpler explanation is on the table. If the "correct" rollouts are mostly lucky guesses, reinforcing them teaches nothing transferable, with no appeal to representations needed.
The definition is partly circular (groups are selected by the outcome being explained), and footnote 1 notes that with longer training some unlearnable examples do spike in reward while validation drops, which suggests memorization.
The mid-training claim stops short. It's measured on Llama-3.2-3B-Base while the main experiments use the Instruct model, on "difficult" rather than unlearnable examples, with modest and overlapping distributions. There's no RL run showing the hard examples actually become learnable after mid-training, which is the experiment the paper's main recommendation needs.
Scale. Nearly all the mechanism analysis is on Qwen2.5-0.5B.

How I'd cite it: strong evidence that correct rollouts alone don't guarantee RLVR learns an example, and that neither more positives, looser regularization, nor SFT fixes it. The representation diagnosis and the mid-training remedy are plausible but not established.