The Verification Boundaries of Language Model Reasoning: Sampling, Search, and Evidence of Correctness

1. Intermediate text is not proof of correctness

Chain-of-thought research shows that, for the models and tasks studied, examples with intermediate steps can improve performance on arithmetic, commonsense, and symbolic reasoning. This finding indicates that prompt organization may affect task solving; it does not guarantee that every generated step is valid. Readers should distinguish final-answer correctness, step validity, and whether explanations faithfully represent the actual computation, rather than substituting narrative coherence for testing.[1] Chain-of-Thought Prompting Elicits Reasoning in Large L

2. Self-consistency spends computation on multiple paths

Self-consistency samples different reasoning paths and then aggregates the answers to make a selection. It exploits the structure that “different paths may converge on the correct answer,” in contrast to generation along a single greedy path. However, if the model repeatedly reuses the same erroneous premise, increasing the sample count may merely repeat the error. Effective candidate diversity should be reported separately from the number of samples drawn.[2] Self-Consistency Improves Chain of Thought Reasoning in

3. Process supervision focuses on intermediate judgments

Let's Verify Step by Step compares outcome supervision with process supervision and reports advantages for process supervision on specific mathematical tasks and under particular candidate-selection settings. The 78% in its abstract pertains to a representative subset of the MATH test set and the paper's evaluation procedure; it must not be recast as “accuracy in solving all mathematical problems.” This article does not use that figure to construct a general capability ranking.[3] Let's Verify Step by Step

4. A transparent derivation for voting

If candidates are independent, their probability of being correct is a constant p, and all correct candidates can be grouped as the same answer, the probability of success for majority voting over an odd number n of attempts is the sum of binomial probabilities from$k=\frac{n+1}{2}$to n. This differs from “at least one candidate is correct”; even when the latter probability is high, a reliable selector is still needed. Figure 2 in this version shows at-least-once success and verifier positive predictive value, not empirical majority-voting results. Independence, identical distributions, and answer grouping are all assumptions that require testing.作者分析与复核建议 / Author analysis

5. Why the independence assumption is often insufficient

Outputs from the same model may share knowledge gaps, prompt biases, and misreadings of a problem, so errors can remain correlated even when sampling produces different wording. Having “nine answers” in a formal sense does not mean having nine independent pieces of evidence. External retrieval, program execution, or provable constraint checking may introduce information distinct from linguistic restatement; whether they actually help still requires task-specific measurement.[2] Self-Consistency Improves Chain of Thought Reasoning in[3] Let's Verify Step by Step

6. Comparisons must specify the budget

If a method uses more tokens, more candidates, or a stronger verifier, changes in quality occur alongside changes in computational investment. This article recommends reporting total tokens per problem, number of calls, timeout rate, final accuracy, and end-to-end latency together, while retaining results grouped by difficulty. Reporting only the average time for the easiest problems obscures the long tail; displaying only the best answer obscures the cost of selection.作者分析与复核建议 / Author analysis

7. Establishing inspectable verification boundaries

For arithmetic, execute the calculations and check units; for code, use isolated tests and expected behavior; for facts from the literature, check sources and versions. Verifiers should also undergo evaluation of their own misjudgments. If a verifier remains, in essence, a subjective scorer based on the same type of model, that dependency should be disclosed, rather than presented as confirmation by external experiments. These are design principles; this site does not claim to have conducted new model experiments on their basis.作者分析与复核建议 / Author analysis

Verifier capabilities are also conditional on budget

Process checking adds another form of computational investment. If the verifier can access reference answers, special tools, or additional context, these should be distinguished from the information available to the candidate-generating model. If the best answer is selected from a hundred candidates, final accuracy should be presented alongside the cost of generating a hundred candidates. Direct comparison with single-generation results can serve as a system-level comparison, but resource differences must not be omitted when claiming that the base model's reasoning ability has improved by the same amount.作者分析与复核建议 / Author analysis

8. The intuition and estimand of pass@k

If each attempt has success probability p and the attempts are mutually independent, the probability of at least one success is 1−(1−p)^k. This is useful for explaining the potential benefits of increased budgets, but does not imply that actual candidate sets satisfy independence. When sampling without replacement from the same fixed sample pool, a common estimator uses binomial coefficients, and its estimand is not exactly the same as that of the independent Bernoulli model. Papers must specify the number of samples, temperature, sample pool, and selection strategy, distinguishing “a correct answer was generated” from “the system can select the correct answer”; the latter also requires a selection or verification step.[2] Self-Consistency Improves Chain of Thought Reasoning in[3] Let's Verify Step by Step

9. Majority agreement may arise from a shared error mechanism

Candidates using the same training knowledge, prompt, and retrieved material may share the same erroneous premise. Different wording does not necessarily provide independent evidence. Error correlations can be studied by varying the surface form of the question, evidence sources, solution tools, and verification methods, but these changes may also alter the success rate of an individual attempt and therefore need to be recorded separately. For mathematics or code tasks, checks that independently recompute or execute are closer to external evidence than purely linguistic mutual agreement. For open-ended factual tasks, the versions and sources of verification material are equally important.[1] Chain-of-Thought Prompting Elicits Reasoning in Large L[2] Self-Consistency Improves Chain of Thought Reasoning in[3] Let's Verify Step by Step

10. Verifiers must be tested on the post-selection distribution

Suppose the proportion of correct candidates produced by the generator is p, the verifier's acceptance rate for correct candidates is t, and its acceptance rate for incorrect candidates is f. The proportion of correct answers in the accepted set is then pt/[pt+(1−p)f]. Even if f appears small, errors may still constitute a substantial share of the accepted set when correct candidates are extremely rare. The baseline correctness rate and acceptance threshold should be reported, and evaluation should use candidates produced by actual search, rather than testing only on manually selected examples that are obviously right or wrong.[3] Let's Verify Step by Step

11. Inference budgets and system optimization require separate attribution

Increasing search nodes, output tokens, or tool calls generally increases the computational budget; system methods such as speculative decoding may instead reduce waiting time under the same target sampling distribution. Both may help users obtain correct answers sooner, but their scientific explanations differ. A fair study should report total computation, wall-clock time, candidate count, and success rate together, and compare performance frontiers under fixed budgets. Do not attribute all of the higher success rate obtained through additional attempts to the base model being “better at reasoning.”[1] Chain-of-Thought Prompting Elicits Reasoning in Large L[2] Self-Consistency Improves Chain of Thought Reasoning in[4] Fast Inference from Transformers via Speculative Decodi

Core equations and derivations

Method and evidence workflow

Analytical plots and numerical examples

Reproducible worked example

With an independent success probability of 0.25, the probability of at least one success in 8 attempts is approximately 0.900; however, with verifier sensitivity 0.9, false-positive rate 0.1, and candidate correctness rate 0.1, its positive predictive value is only 0.5. The existence of a correct candidate does not mean it is ultimately selected.

Method comparison

Review and experimental protocol

1. Choose a task set with clearly assessable answers, and preserve answer criteria, data versions, and contamination checks; establish separate source and annotation rules for open-ended questions.
2. Fix the total candidate budget and separately compare single generation, sampling, voting, external tools, and verifier selection, retaining all failures.
3. Report pass@k, the accuracy of answers selected by the system, and the proportion that cannot be assessed; do not display only the best output.
4. Measure error correlations among candidates and check whether different phrasings still depend on the same erroneous premises or the same sources.
5. Sweep verifier thresholds, record true acceptance, false acceptance, and rejection rates, and validate on the actual search distribution.
6. Account separately for tokens, tool calls, actual computation, and wall-clock time in the budget; repeat independent held-out evaluation after changes to training or prompts.

Scope and limitations

• Independent sampling is only an analytical assumption.
• Open-ended questions often lack a complete verifier capable of deciding correctness.
• Generated explanations cannot replace external verification.
• No new reasoning benchmarks or model rankings are introduced.

Primary sources and corresponding claims

1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsWei et al. · 2022-01-28v6; method and limitations
2. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsWang et al. · 2022-03-21v4 abstract
3. Let's Verify Step by StepLightman et al. · 2023-05-31v1 abstract; process vs outcome supervision
4. Fast Inference from Transformers via Speculative Decoding, , · 2022-11-30Abstract; 原文方法、理论和实验章节见作者原始论文链接