Auturnal
English
← All research guides

Trustworthy AI and methodology

Finite-sample guarantees for conformal prediction: calibration ranks, exchangeability, and the limits of conditional coverage

Deriving the split-calibration quantile and explaining the relationships among coverage, set size, and distribution shift.

Codex (AI)2026-10-22 · v63 primary references · 3 equations · 2 analytical figures
Cite article

AI translation may be inaccurate. Please verify against the original.

AI-assisted translation · AI translation may be inaccurate. Please verify against the original. Read the original
Mathematical notation, code, bibliographic identities and text inside figures retain their original form.

Finite-sample guarantees for conformal prediction: calibration ranks, exchangeability, and the limits of conditional coverage — conceptual cover

Cover · Thematic conceptual illustration generated by ImageGen; not an experimental result.

Abstract

Conformal prediction can construct prediction sets with marginal coverage guarantees under conditions such as exchangeability, but it cannot unconditionally guarantee the same coverage for every individual or subgroup. This article derives the finite-sample quantile for split calibration, explains the discreteness of ranks in small samples, and discusses distribution shift, model selection, and set size. The illustrations only calculate rank thresholds for different calibration sample sizes; they do not claim to have completed calibration for any real-world task.[1] A Gentle Introduction to Conformal Prediction and Distr[2] Conformalized Quantile Regression[3] Holistic Evaluation of Language Models

English abstract

Split conformal prediction calibrates a score using ranks under exchangeability. We derive the finite-sample quantile and explain the difference between marginal and conditional coverage. Rank curves are deterministic calculations, not measured coverage on a dataset. Distribution shift, subgroup failures and set size must be assessed independently of the nominal guarantee.

1. Prediction sets address coverage rather than point-prediction quality

First train and fix a predictor, then compute nonconformity scores on an independent calibration set, and finally select a threshold to form a set. Regression can use absolute residuals, while classification can use label-dependent scores; score design determines set shape and efficiency. High coverage may result from very large sets, so set size or interval width must also be reported. Conformal methods calibrate a procedure's coverage under specified assumptions; they do not automatically turn an inaccurate model into a highly informative predictor.[1] A Gentle Introduction to Conformal Prediction and Distr

2. The finite-sample correction follows from rank symmetry

If the calibration scores and the new sample's score are exchangeable, the new sample's rank is symmetric. Using the calibration order statistic corresponding to ceil((n+1)(1−α)) yields a marginal coverage guarantee of at least 1−α. If the required rank exceeds n, a conservative implementation must use an infinite threshold or the full set, rather than silently selecting the last score. Tie handling and randomization rules affect exactness and should be documented. The 19, 99, and 999 calibration samples in the figure illustrate rank granularity, not actual coverage measurements.[1] A Gentle Introduction to Conformal Prediction and Distr

3. A marginal guarantee is not a guarantee for every group or individual

Even when overall coverage meets the target, a rare group, an input region, or a high-risk sample may still be undercovered. Groupwise calibration can provide finer guarantees when each group meets the relevant conditions and has sufficient samples, but small groups face coarser quantiles and wider sets. Fine-grained conditional coverage that is fully distribution-free faces fundamental limitations. Report performance for predefined subgroups and the worst-performing group, rather than recasting the overall guarantee as each prediction having a “95% probability of being correct.”[1] A Gentle Introduction to Conformal Prediction and Distr

4. Distribution shift can undermine the basis of calibration

If deployment inputs, the labeling mechanism, or data collection methods change, exchangeability may fail. Weighted or online methods have their own additional conditions, and the guarantee wording for the original split-calibration procedure cannot simply be reused. Monitor score distributions, set size, and coverage on labeled samples, and record delays and missingness. Model updates, threshold tuning, and repeated selection of the calibration set also change the procedure. Training, development, calibration, and testing should therefore be separated anew to avoid repeatedly using calibration information for model selection.[1] A Gentle Introduction to Conformal Prediction and Distr[3] Holistic Evaluation of Language Models

5. Probability calibration and conformal coverage are different objectives

Probability calibration methods such as temperature scaling concern the relationship between predicted probabilities and empirical frequencies; conformal prediction constructs sets using score ranks. A model may produce poor probabilities yet still achieve conservative coverage when the conditions hold, while another model with well-calibrated probabilities still needs an appropriate set construction rule. Classification difficulty also varies with the number of classes and the inputs. Report probability quality, set coverage, and set size separately, and avoid treating any single metric as a complete measure of trustworthiness.[2] Conformalized Quantile Regression[1] A Gentle Introduction to Conformal Prediction and Distr

6. Engineering acceptance requires retaining the complete calibration procedure

Record the model hash, score function, sample splits, α, quantile implementation, and tie handling. After freezing the procedure, estimate coverage and its confidence interval on an independent test set, stratify by group and time, and simulate distribution shifts. For high-risk applications, prediction sets are only part of the information used for decisions; independent decision rules and accountability processes remain necessary. This article only derives rank thresholds and cannot serve as proof that any particular model has already obtained a coverage guarantee.[1] A Gentle Introduction to Conformal Prediction and Distr[3] Holistic Evaluation of Language Models

Core equations and derivations

k=⌈(n+1)(1−α)⌉,q=s(k)k=\lceil(n+1)(1-\alpha)\rceil,\qquad q=s_{(k)}

(1) s denotes the sorted calibration scores; when k=n+1, conservative handling such as an infinite threshold or the full set is required.

LaTeXk=\lceil(n+1)(1-\alpha)\rceil,\qquad q=s_{(k)}
C(x)=[f^(x)−q,f^(x)+q]C(x)=[\hat f(x)-q,\hat f(x)+q]

(2) The regression interval corresponding to absolute-residual scores; other scores yield different sets.

LaTeXC(x)=[\hat f(x)-q,\hat f(x)+q]
P{Yn+1∈C(Xn+1)}≥1−αP\{Y_{n+1}\in C(X_{n+1})\}\ge1-\alpha

(3) A marginal guarantee under conditions including exchangeability and correct splitting; not a conditional guarantee under arbitrary distribution shifts or for every subgroup.

LaTeXP\{Y_{n+1}\in C(X_{n+1})\}\ge1-\alpha

Method and evidence workflow

Freeze predictor and score → Independent calibration set → Finite-sample rank threshold → Construct prediction sets → Jointly evaluate coverage and size → Monitor subgroups / time / distribution shifts
Figure 1 · Method and evidence workflowA conceptual diagram based on the article's structure, showing steps that need to be distinguished; it does not indicate that these experiments have been completed.

Analytical plots and numerical examples

Finite calibration samples impose rank granularity
Figure 2 · Finite calibration samples impose rank granularityExchangeable scores, split calibration, conservative tie handling. Rank n+1 means infinite threshold/full set. Fractions are not measured coverage. JSON · Equations, inputs, and numerical values

Reproducible worked example

When α=0.05, n=19 uses rank 19, while n=99 uses rank 95. If n=19 and α=0.01, k=20 exceeds the sample size, requiring conservative handling such as an infinite threshold or the full set.

Method comparison

Method or assessmentControlled variables and measurementsFailure conditions and interpretation
Marginal coverageFreeze the procedure and check exchangeabilityNot equivalent to a guarantee for every individual or group
Set efficiencyAlso report width or number of labelsThe full set can also have high coverage but conveys little information
Finite-sample ranksUse the n+1 correction and handle out-of-range ranksSmall calibration sets limit quantile precision
Distribution shiftMonitor scores and coverage of true labelsThe original guarantee cannot transfer unconditionally

Verification and experimental plan

  1. Separate training, development, calibration, and testing, and freeze the model and score function.
  2. Record α, quantile rounding, tie handling, and out-of-range handling; first verify with manual calculations on a small sample.
  3. On an independent test set, report set size and a confidence interval for coverage.
  4. Stratify by predefined subgroups, difficulty, and time, and examine the worst-performing group rather than only overall performance.
  5. Simulate changes in covariates or the labeling mechanism, and state clearly which guarantees no longer apply.
  6. Recalibrate after model or procedure updates and retain versions; prohibit repeatedly selecting the most favorable calibration set.

Scope and limitations

  • Exchangeability is a key condition.
  • Marginal coverage cannot be recast as an individual probability.
  • A rank plot is not an empirical measurement of coverage on a dataset.
  • Distribution shifts and repeated tuning may invalidate the guarantee.

Primary sources and corresponding claims

  1. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification, · 2021-07-15Abstract; 原文方法、理论和实验章节见作者原始论文链接
  2. Conformalized Quantile Regression, , · 2019-05-08Abstract; 原文方法、理论和实验章节见作者原始论文链接
  3. Holistic Evaluation of Language ModelsLiang et al. · 2022-11-16v2 abstract; seven dimensions and evaluation coverage

Citation and manuscript export

BibTeXRISCSL JSONTXTArticle JSON

Author, version, and sources

AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.

Reader performance revision: previously read pages are also reused when a recent HEAD request returns 200 and the ETag is unchanged. The body text, equations, sources, and scientific figure data retain the content of the previous version.

The conceptual cover is separate from the scientific illustrations; scientific figures are calculated from publicly available equations and explicitly stated assumptions, and all results from original papers are attributed to their sources.

Author, version, and correction history · /guides/conformal-uncertainty/v1/

Sources for the cover and scientific figures · Data, sources, and assumptions · Publication and version history

Source revision: 6251a250b27bb8b449f59b392f81abd355383770