arXiv:2604.25907v2 Announce Type: replace-cross
Abstract: SFT-then-RLVR is widely used for post-training reasoning models, but why this specific ordering, and why RLVR-only stalls at cold start, have lacked a unifying theoretical account. We provide that account under a unified loss family $J_Q$ using the Tsallis $q$-logarithm. $J_Q$ is a single-parameter family that interpolates between RLVR (at $q{=}0$, the textit{exploitation pole}) and the log-marginal-likelihood over latent trajectories (at $q{=}1$, the textit{density-estimation pole}), under which the standard pipeline corresponds to a stepwise $q{=}1 to 0$ schedule. All members share the same per-example gradient direction, differing only by a per-instance amplification $P_theta^{-q}$ that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires $Omega(frac{1}{p_0})$ time to escape cold start but is robust to label noise, while the density-estimation pole escapes in $Thetabig(log(frac{1}{p_0})big)$ but memorizes label noise. This separation explains how SFT ($q{=}1$) first moves the model out of the cold-start regime, followed by the more robust RLVR ($q{=}0$), under the SFT-then-RLVR paradigm. We further derive two Monte Carlo estimators that directly optimize fixed-$q$ on the $J_Q$ continuum, without annotated rationales: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias $Obig(frac{q}{M P_theta^q}big)$ but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high $q$ substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low $q$ dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes and PAFT at $q{=}0.75$ remains stable, reaching $47.9$ texttt{m@16} on HotPotQA ($+13.9$ over GRPO).