What an Unlearning Test Can Actually Certify

1. Forgetting is a claim about a comparison

A model that refuses to answer a question has not necessarily removed the training influence of the corresponding record. A model whose output changes after a deletion request has not necessarily become equivalent to retraining without that record. Unlearning claims require a reference procedure and a specified set of observable outputs.

This review separates exact retraining equivalence, approximate distributional guarantees, finite behavioral tests, and operational deletion of stored artifacts. A scalar ridge-regression construction shows how a sufficient-statistic update can exactly match retraining when the model and objective are fixed. A second construction demonstrates that a predictor can agree with a reference on every selected probe while differing elsewhere. Together they expose the difference between a positive test result and a general certificate. The analysis is not legal advice or an audit of any deployed language model. Its purpose is to define what an experiment establishes, which training artifacts remain in scope, and how a reproducible deletion protocol can avoid claiming more than its evidence supports.

2. The primary literature spans several guarantees

Cao and Yang study unlearning through suitable summation-form computations. Bourtoule and colleagues develop a sharded, isolated, sliced training approach intended to reduce retraining cost. Guo and colleagues formalize certified removal and study mechanisms for linear classifiers.

Golatkar, Achille, and Soatto examine selective forgetting in neural weights. Neel, Roth, and Sharifi-Malvajerdi study gradient-based deletion for convex models and sequences of updates. Chen and colleagues investigate privacy risks created by access to models before and after unlearning. These scopes motivate different reference distributions, computational assumptions, and adversary observations. They do not establish a single interchangeable definition of forgetting. We use the inspected original abstracts to identify these distinctions, without reproducing their experimental scores or auditing their proofs. The finite ridge example below is our own exact calculation, and the probe counterexample is a general logical limitation of finite behavioral evidence.

Sources: [1], [2], [3], [4], [5], [6]

3. Specify the learning algorithm, not only the dataset

Retraining equivalence is relative to a learning procedure A applied to the retained dataset. A includes initialization, preprocessing, feature selection, hyperparameter selection, stopping rules, and randomness. If deletion changes a normalization statistic or a selected architecture, retraining only the final layer does not necessarily reproduce A on the retained data.

For randomized training, the reference is generally a distribution of models or released behaviors rather than one privileged seed. Comparing an unlearned model to a single retrained model can confuse ordinary seed variability with deletion error. A report should state whether seeds are coupled and what metric is used. Parameter distance may be misleading when networks have symmetries, while finite output distance may miss hidden differences. Neither metric is automatically wrong, but each supports a different statement. A scientifically useful guarantee should name the entire procedure and observation space before selecting a convenient test. Otherwise the evaluation can be made easier after the desired result is known.

4. Exact deletion in a scalar ridge model

Consider scalar examples (x_i,y_i) and the objective one half of the summed squared residuals plus lambda theta squared divided by two, with lambda positive. Its unique optimum is theta=S_xy/(S_xx+lambda), where S_xy is the sum of x_i y_i and S_xx the sum of x_i squared. Removing one example subtracts its two contributions and recomputes the ratio.

This is exact because those statistics are sufficient for the declared objective. The regularization convention is essential: using a mean loss while keeping lambda fixed changes the relationship when sample count changes. Our construction uses a summed loss with fixed lambda=1, so the deletion formula is unambiguous. The dataset has x=(1,2,3,4) and y=(1,2,2,5), giving theta=31/31=1 before deletion. Deleting a record can change the exact retrained coefficient. In this dataset, deleting record one or record two leaves it equal to one; deleting record three or record four changes it. This calculation is a small transparent example of an algorithm with tractable deletion, not evidence that a large nonlinear network can be unlearned by subtracting one generic vector.

5. A naive subtraction fails because the denominator changes

One tempting approximation subtracts the removed cross-product from the numerator while retaining the old denominator. That procedure forgets the removed contribution to curvature. In the scalar model, the exact retained-data denominator is S_xx-x_j squared+lambda. The approximate and exact coefficients therefore differ except in special cases.

The first figure compares exact retraining with this deliberately naive update for each removed record. It reports the exact and approximate parameter values rather than a vague forgetting score. The example also allows direct evaluation of retained-data loss, so a method that appears close in parameter space can be assessed against its actual optimization objective. We do not claim the naive update represents a named published algorithm. It is a negative control demonstrating how a plausible deletion story can miss part of the computational dependency. Real pipelines contain many analogous dependencies: normalization, vocabulary construction, cached features, checkpoint selection, and optimizer state.

6. Normalization can make the reference move

Suppose a model is trained after centering features by the training mean. Deleting a record changes that mean, so every retained transformed example changes. Subtracting only the removed record's contribution from statistics computed in the old coordinates does not reproduce retraining in the new coordinates unless the transformation update is also included.

The same issue appears with learned tokenizers, feature encoders, and data-dependent augmentation policies. A declared deletion scope may intentionally keep a public or frozen representation unchanged. That is a valid narrower reference if stated clearly. It becomes misleading when described as full retraining from the original raw data. A dependency inventory should distinguish fixed public components, privately learned components, and post-training caches. The scientific comparison is then explicit: unlearning may match a frozen-feature retraining baseline without matching a complete end-to-end retraining baseline. Reporting both where feasible exposes the approximation instead of hiding it behind one model-level label.

7. Finite probes cannot identify an arbitrary function

Let the reference predictor be f_0(x)=0 and a candidate be f_a(x)=a x(x-1). On the probe set {0,1}, both return zero for every a. At x=1/2, however, the candidate returns -a/4. Agreement on those two probes therefore places no bound on disagreement at the midpoint unless the hypothesis class or coefficient magnitude is restricted.

The second figure plots several candidate functions, all perfectly matching the chosen probes. This is a logical construction, not a proposed attack on a live model. It shows why passing a finite prompt suite cannot establish universal absence of information in weights or outputs. A richer probe suite can improve detection power, but a universal claim requires additional structure, assumptions, or a formal guarantee. The test should be described in terms of its coverage and detectable alternatives. A zero observed difference is evidence about the tested inputs, not automatically evidence of equality over all possible inputs.

8. Distributional tests need power and a reference sample

When models are randomized, a two-sample test can compare outputs from unlearning and retraining procedures. Its result depends on sample size, chosen statistic, and the alternatives it can detect. Failure to reject equality is not proof of equality, especially when the test has low power.

A useful evaluation includes positive controls that introduce a known residual influence and negative controls consisting of two independent retraining batches. If the test cannot distinguish a deliberately compromised model from retraining, its reassuring result on the proposed method is weak. Pre-register the probe distribution and decision threshold rather than selecting them after seeing outcomes. Report uncertainty and the cost of obtaining reference models. Parameter-level and output-level tests may disagree because they observe different objects. The goal is not to choose whichever test passes, but to explain the scope of each conclusion and identify important alternatives left unresolved.

9. Membership inference is a diagnostic, not a definition

A membership attack asks whether a record was used in training under a specified adversarial setting. A weak attack may fail even when influence remains. Conversely, a retrained model can still reveal information about a deleted record through correlated retained examples or public knowledge. Thus attack success alone does not define whether an unlearning algorithm matches its declared retraining reference.

The correct comparison includes attacks on the original, unlearned, and independently retrained models under matched observation budgets. Access to before-and-after models can create additional signals, so the adversary's view must be specified. A deletion evaluation should not quietly grant the attack less information on the candidate than on the baseline. Nor should it claim that retraining erases all knowledge of a fact if other authorized data still support that fact. The operational request, statistical reference, and privacy objective need to be distinguished. Without that separation, a model can be penalized for legitimate retained knowledge or credited for merely hiding an answer.

10. Repeated requests introduce state and composition

A one-request unlearning result does not automatically extend to an arbitrary sequence of adaptive deletion requests. Approximation errors can accumulate, stored state can become stale, and the request sequence can depend on previous outputs. The data structure supporting deletion is part of the mechanism.

The scalar sufficient-statistic model is easy to test sequentially: after each removal, recompute statistics from the remaining examples and compare them with incremental subtraction. With exact arithmetic they agree; floating-point implementations can accumulate numerical differences that should be bounded or periodically corrected. In deep learning, repeated fine-tuning can drift in more complex ways. A useful benchmark reports the number of requests, their sizes, ordering, overlap, and whether requests were chosen adaptively. It also states when full retraining is triggered. A practical system may offer a hybrid guarantee with periodic resets, but those resets and their costs must be included in the published method.

11. Artifacts outside the final weights remain relevant

Training influence may persist in optimizer checkpoints, cached embeddings, feature stores, generated datasets, retrieval indexes, and exported model copies. A method that updates one current checkpoint has a narrower scope than an operational deletion workflow covering all authorized artifacts.

This is a systems boundary rather than a new mathematical theorem. A deletion manifest should identify which artifacts were derived from the record, which are updated, and which remain under a separately stated retention policy. Historical audit records can preserve the fact that a deletion action occurred without retaining the deleted content itself, subject to the applicable requirements. Access control and backup handling also matter. These operational tasks should not be mislabeled as proof that a neural model has forgotten a concept, but neither should a model-level result imply they are complete. The final scientific and engineering report should present both scopes so that users can understand what changed.

12. Utility preservation needs a stable target

An unlearning method can force the model to perform poorly on deleted examples by damaging a broad region of its behavior. That may reduce an attack score while also destroying useful retained knowledge. A meaningful evaluation measures retained-task utility and targeted residual influence separately.

The retained evaluation set must be independently specified. If it contains near duplicates of the deleted set, high retained performance may be expected and does not by itself refute deletion relative to retraining. If it excludes every related concept, the evaluation may hide collateral damage. Define relevant neighborhoods, source groups, and tasks before testing, and report tradeoffs rather than compressing them into one score. The exact ridge example illustrates this by allowing direct loss evaluation on all retained records. The function counterexample illustrates the complementary risk: excellent behavior on a narrow test suite can coexist with arbitrary behavior elsewhere. Together they motivate broader, clearly scoped tests.

13. A reproducible verification protocol

Start by fixing the reference algorithm, raw-data preprocessing, randomness policy, and observable outputs. Preserve a versioned deletion request identifying exact records and dependencies. Produce independently retrained baselines where feasible, including multiple seeds when the algorithm is randomized. Evaluate the candidate against both the baseline distribution and known positive-control failures.

Test single deletions, batches, overlapping requests, and long sequences. Report runtime, storage overhead, retained utility, and attack observations under matched budgets. Include model copies and derived artifacts in the operational inventory. If a formal certificate is used, verify that its assumptions match the implemented loss, regularizer, sensitivity, and numerical operations. Do not label a behavioral prompt suite as that certificate. A useful report should state what would cause rejection, what was actually measured, and which alternatives remain indistinguishable. This protocol aims to make a claim falsifiable, not to guarantee that every model family admits efficient exact deletion.

14. Executable checks and their limits

The accompanying program compares incremental ridge deletion against fresh sufficient statistics for every nonempty retained subset of the four-example dataset. It checks the first-order optimality condition and verifies that the deliberately naive denominator update fails for at least one deletion. Sequential removals are tested in multiple orders.

The second construction checks exact agreement at the two probes and nonzero midpoint differences for every nonzero coefficient. It also verifies the plotted function values directly. These tests establish the arithmetic and logical counterexample only. They do not train or unlearn a neural network, implement a privacy attack, or certify a legal deletion obligation. Their purpose is to expose two important verification requirements: update every dependency in the reference computation, and distinguish finite observational agreement from a universal guarantee. Those requirements remain relevant even when the practical algorithm is much more sophisticated than the examples used here.

15. A precise claim is more useful than a reassuring label

Unlearning research benefits from stating exactly which distribution, behavior, or computational dependency is being matched. A narrow but verified result can be scientifically valuable. An expansive claim supported only by a few successful prompts is difficult to interpret and easy to misuse.

Open questions include scalable reference generation, tests with power against meaningful residual influence, compositional guarantees for adaptive requests, and efficient tracking of derived artifacts. Different applications may reasonably choose different scopes, but those choices should be explicit. The finite constructions show that exact deletion is possible for suitable sufficient-statistic models and that finite probes alone cannot certify arbitrary functional equality. Neither result settles modern foundation-model unlearning. They provide a disciplined starting point for evaluating it: define the reference, preserve the assumptions, measure the relevant alternatives, and avoid turning an observed absence of evidence into a universal evidence of absence.

16. The reference must preserve the regularization convention

The ridge construction uses a sum of losses with a fixed penalty coefficient. If instead the objective is the average loss plus a fixed penalty, multiplying by the retained sample count yields a sum objective with a penalty that changes after deletion. The exact coefficient is then S_xy divided by S_xx+n lambda. Removing a record changes both S_xx and the factor n. Reusing the earlier formula would compare against the wrong retraining procedure.

This distinction is small enough to be checked exactly yet easy to overlook in a larger implementation. Many libraries expose a parameter called regularization strength while using different normalizations internally. A deletion mechanism that assumes one convention can appear approximately correct on a large dataset because one removed row barely changes n, then fail for a small retained set or a large deletion batch. The scientific specification should therefore state the full objective, not just a named optimizer and a penalty parameter.

The program includes both conventions as separate exact controls. It computes the retained-data optimum directly for every subset and verifies that the formula appropriate to each objective satisfies its own first-order condition. It also checks a case where applying the sum-loss formula to the mean-loss objective gives a different answer. This is a targeted check of reference correctness, not an additional claim about neural unlearning.

Operationally, the reference manifest should bind the loss reduction, preprocessing, regularizer, and dataset-size interpretation to the model version. That information permits another researcher to reconstruct what never trained on the removed records actually means. Without it, disagreement between a deletion result and a retraining baseline may reflect a changed objective rather than residual data influence, and agreement may be accidental.

Core equations and definitions

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Freeze the reference algorithm and normalization.
• Compare multiple retraining seeds where relevant.
• Use positive-control failures in tests.
• Track sequential requests and derived artifacts.

Limitations

• No neural unlearning or privacy attack is performed.
• Finite probes cannot certify arbitrary functions.
• Operational and legal scopes require separate evaluation.

Open research questions

• How can reference models be generated efficiently?
• Which tests have power against meaningful residual influence?
• How do adaptive deletions compose?

Evidence and its boundaries

Summation-form computations can support efficient removal.

[1] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

SISA limits training dependencies to reduce unlearning cost.

[2] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Certified removal compares a removed-data model with a never-trained reference.

[3] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Selective forgetting concerns information in weights, not only visible output suppression.

[4] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Official CVF primary abstract and bibliographic metadata inspected through indexed web search on 2026-10-28; direct page access returned HTTP 403. No full proof or benchmark replication.

Convex deletion methods study sequences of updates and computational guarantees.

[5] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Before-and-after model access can create additional privacy signals.

[6] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.