A research paper titled “Free Lunch for LLMs Confidence: The RL Critic Already Knows” was published on October 26, 2026, on the arXiv preprint server (ID 2610.31820v1), proposing a novel method to enhance the confidence estimation capabilities of large language models (LLMs) without requiring additional training.
Research has found that, within a reinforcement learning framework defined by correctness-based outcomes, the problem of confidence estimation for LLMs is mathematically equivalent to the Critic’s value-function learning—both aim to predict the probability that a given policy will yield the correct outcome. Consequently, for LLMs already trained using value-based RL, their built-in Critic can serve directly as an off-the-shelf confidence estimator, without requiring access to the model’s internal representations or incurring additional data or computational overhead.
Experiments show that this reuse scheme achieves the best confidence estimation performance across all in-distribution test sets in more than ten domains, including mathematics, code generation, tool invocation, and general reasoning. It also attains optimal calibration on 14 out of 16 evaluation subsets, without compromising the original task’s performance at all. Furthermore, Critic can provide fine-grained confidence outputs throughout the entire generation process, including intermediate reasoning steps.
This achievement provides a lightweight, plug-and-play technical pathway for the trustworthy deployment of LLMs; however, its applicability is explicitly limited to models trained using value-based reinforcement learning, and its conclusions are based on the experimental setup and evaluation protocols reported in the paper.