Contrastive Learning Between Alignment and Information Loss Codex (AI) | Commissioned review v1 | 2026-10-22 Contrastive representation learning turns a relationship between observations into a learning target. That relationship is already a substantive modeling assumption: an augmentation policy selects which distinctions may be erased, while negative sampling decides which distinctions must remain. This analytical review uses SimCLR, hyperspherical alignment and uniformity, and Contrastive Predictive Coding as three primary anchors. It develops a binary corruption construction that separates pair agreement from label information, derives the temperature-dependent pressure of a finite contrastive objective, and specifies controlled evaluations for false negatives, projection heads, and transfer. All plotted curves are computed from declared small models. The argument is that contrastive success must be described relative to a view-generating process and a downstream information requirement; neither low training loss nor a well-spread embedding alone establishes that the representation retains the right information. 1. The objective starts before the loss A learner does not observe an abstract category called semantic similarity. It observes pairs produced by an augmentation program, a temporal relationship, a matching database, or an annotation process. Before optimizing an encoder, one must decide which relation these pairs are intended to approximate. Two crops may depict the same object, different objects in one scene, or a tiny texture that says nothing about the intended label. These cases have identical positive-pair bookkeeping but different statistical meaning. This article treats the view-generation distribution as part of the model specification. Its central question is whether information needed by a future task survives that distribution and remains recoverable from the trained representation. A representation can fail because the view destroyed the information, because optimization discarded it, or because the downstream decoder cannot extract it. These are separate failure mechanisms, requiring separate measurements and different remedies. 2. What SimCLR establishes Chen and colleagues introduce SimCLR and examine augmentation composition, a nonlinear projection head, and training scale in a particular visual learning pipeline. Their ablations establish that implementation choices around a contrastive objective materially affect its measured transfer performance. The paper does not establish that a chosen augmentation preserves every useful label, or that a gain on one benchmark is a task-independent information guarantee. The conceptual distinction used here is between the encoder representation and the projected coordinates on which the contrastive loss acts. The original paper motivates examining this distinction empirically; the counterexamples and diagnostics below are this article's own constructions. No reported SimCLR accuracy is reproduced or used as a measurement in the accompanying figures. References: dl-06-1 3. A geometric lens with an explicit domain Wang and Isola analyze alignment of positive pairs and uniformity of normalized representations on a hypersphere. Their work supplies a useful geometric decomposition and an asymptotic analysis of contrastive objectives. It does not turn uniformity into a definition of semantic adequacy. A distribution can cover a sphere while encoding an irrelevant identifier, and a task-relevant representation can occupy a restricted subset because the task itself is simple. The scope matters: normalization specifies the geometry, the positive-pair law specifies alignment, and the marginal distribution specifies the population being spread out. Applying these diagnostics to an unnormalized embedding, a changing sampling distribution, or a highly imbalanced task requires stating those changes. We use their framework as a lens, not as a universal sufficiency theorem. References: dl-06-2 4. Prediction and information bounds Contrastive Predictive Coding uses predictive latent representations and a negative-sampling objective. Its analysis connects a population contrastive objective with a mutual-information lower bound under the sampling assumptions used in that construction. Those assumptions matter when negatives are drawn from a queue, a filtered index, or a class-dependent sampler. The bound concerns information between the specified random variables, not automatically information about a downstream label. Moreover, a finite candidate set limits the scale of an estimate built from the candidate-identification problem. The following binary example is independent of CPC's experiments. It is designed to show why identifying the random variables is more informative than saying that a method maximizes information in the abstract. References: dl-06-3 5. A two-bit counterexample Let an observation contain two independent fair bits: a desired label Y and a stable nuisance S. A view preserves S but flips Y independently with probability q, where q ranges from zero to one half. Two views of the same observation therefore share the nuisance exactly, while their label-derived bits agree with probability one minus twice q times one minus q. The best prediction of Y from one corrupted label bit has accuracy one minus q. At q equal to one half, the corrupted bit is independent of Y. Any representation that sees only that view has no route to recovering Y beyond chance, even with an ideal decoder and unlimited training examples. Meanwhile, a representation based on S can align perfectly across positive pairs. The finite example does not claim that real augmentations are binary channels; it isolates an information loss that training cannot reverse. Panel 1 plots the two distinct probabilities so that pair agreement is not mistaken for retained label accuracy. 6. Information may vanish smoothly before accuracy collapses For the same fair binary channel, the mutual information between Y and its corrupted observation is one minus binary entropy. The information curve is nonlinear even though optimal classification accuracy declines linearly. This difference matters when a representation is expected to support several downstream decisions or calibrated probability estimates, rather than one thresholded label. At q equal to one quarter, optimal single-view accuracy is three quarters, while the retained information is only about 0.189 bits. This exact channel calculation is not an estimate of a neural network's mutual information. It says that different summary statistics answer different questions even before an encoder is introduced. An empirical investigation should therefore report the target label distribution, the prediction loss, and whether the evaluation concerns one view, multiple independent views, or the original uncorrupted input. Combining two views can change the available evidence and must not silently change the comparison. 7. Temperature changes which mistakes dominate Consider a deliberately simple candidate set: one positive similarity equals one and each of K negative similarities equals zero. The loss is the logarithm of one plus K times the exponential of minus one divided by temperature. Panel 2 evaluates this expression for three finite negative counts. It is an objective-value calculation, not a simulated training curve. Differentiating the loss with respect to the positive similarity yields its softmax probability minus one, divided by temperature. Lower temperature therefore changes both the distribution of candidate weights and the derivative scale. It can sharpen distinctions that are already present; it cannot identify whether a highly similar negative is semantically a false negative. When comparing temperatures, learning-rate tuning and numerical stability must be controlled. Otherwise an apparent representational improvement may partly be an optimization effect. A shift of every candidate logit by the same constant leaves the probabilities unchanged, a basic invariant checked by the executable companion. 8. More negatives can change the task A negative sampler is a hypothesis about nonmatching observations. If the intended class distribution is imbalanced, larger batches can increase the number of examples that share a useful class with an anchor. Treating all of them as equally undesirable neighbors may encourage separation inside a class that a later classifier would prefer to group. This is not a proof that more negatives are harmful. Instance identity may carry useful structure, and a sufficiently flexible representation can preserve both instance and class information. The point is that candidate count and candidate meaning should be varied separately. A controlled experiment should fix the number of candidates while changing semantic collision rates, then fix the collision process while changing the count. Reporting only batch size combines several interventions, including optimizer noise, normalization statistics, and compute. A negative count is an accounting number; it becomes a scientific explanation only after the sampling process has been specified. 9. A projection head can hide a useful distinction Suppose an encoder retains both a label coordinate and a nuisance coordinate, while its projection head discards the label coordinate. The contrastive loss sees only the projection, yet a classifier attached before the head could still recover the label. The opposite arrangement is also possible: a useful projected distinction can coexist with a poorly conditioned upstream representation for a restricted linear probe. Consequently, a probe must identify its attachment layer, normalization, regularization, and training budget. A weak probe can underestimate retained information; a very flexible probe can conceal that the representation makes information expensive to extract. The relevant comparison depends on the intended deployment. If the actual consumer is a small linear model, linear accessibility is directly meaningful. If the consumer will fine-tune a large network, it answers a different question. One scalar transfer score cannot fairly stand in for both settings, and selecting a layer after inspecting test labels creates an additional selection bias. 10. Invariance should be conditional on the use case An augmentation that removes color can help a shape classification task while damaging a task whose target is a material state expressed through color. Neither task makes the augmentation intrinsically good or bad. The defensible statement is conditional: this transformation is intended to preserve this target under this acquisition process. A useful audit therefore pairs each invariance with a possible countertask. Cropping should be tested against counting and small-object presence; temporal proximity against events that change rapidly; color changes against labels with chromatic content. These are proposed diagnostic categories, not results claimed for a named model. If a countertask is outside product scope, documenting that exclusion is more informative than hiding its failure. If it is inside scope, one may retain multiple views, use task-aware augmentation, or train a representation with separate invariant and equivariant components. Such choices trade simplicity for information preservation and should be evaluated at the representation and decision levels. 11. An evaluation that separates causes The proposed evaluation begins with an explicit generative toy system where desired and nuisance variables are known, then proceeds to a real dataset whose transformations can be audited. The toy system offers exact Bayes limits; the real dataset offers relevance. Neither substitutes for the other. Models should be compared at matched data exposure and optimization budget, with augmentation randomness paired where possible. Three readouts are needed: positive-pair alignment, marginal geometry, and downstream risk on prespecified tasks. A fourth readout tests information that the augmentation was allowed to remove. This prevents an improvement in one diagnostic from being silently promoted to a universal claim. Confidence intervals should reflect independent training seeds and data sampling rather than treating correlated examples from one run as independent runs. The supplied executable checks only finite channel probabilities, objective invariants, and the constructed counterexample. It does not train SimCLR or reproduce an ImageNet benchmark. 12. The representation contract A convincing contrastive-learning claim states what counts as a positive, how negatives are drawn, where the representation is read, and which decisions it must support. Those four declarations make a result falsifiable. They also make failure actionable: change the view process when information is destroyed, the objective when information is systematically discouraged, or the decoder when retained information is inaccessible. The binary example establishes a narrow impossibility: information absent from the observation cannot be recovered by a deterministic encoder of that observation. The geometric and temperature analyses establish equally narrow diagnostics. Together they provide a disciplined way to interpret transfer experiments without turning a pretext objective into a claim of general understanding. Future work should test whether a representation preserves a portfolio of explicitly named distinctions under shifts in the view distribution. That is a stronger and more useful research question than asking whether its embeddings merely look well organized. Executed finite checks Binary view information, nuisance counterexample and contrastive invariants 202 assertions passed. Finite binary channel enumeration and fixed candidate-loss checks; no contrastive encoder is trained. { "q": 0.25, "pair_agreement": 0.625, "bayes_accuracy": 0.75, "q_half_label_accuracy": 0.5, "nuisance_alignment": 1, "negative_counts": [ 3, 15, 63 ] } Code: analysis/examples.py with models.py and part_a.py through part_d.py; results: analysis/example-results.json Equations (1) \ell=-\log\frac{\exp(s_+/\tau)}{\exp(s_+/\tau)+\sum_{j=1}^{K}\exp(s_j/\tau)} Finite candidate loss with one positive, K negatives, real-valued similarities and temperature tau > 0. (2) \frac{\partial\ell}{\partial s_+}=\frac{p_+-1}{\tau},\qquad \ell\big|_{s_+=1,s_j=0}=\log(1+K e^{-1/\tau}) The derivative holds for fixed candidate similarities. The right-hand special case generates panel 2; it is not a training trajectory. (3) \Pr(V_1=V_2)=1-2q(1-q),\qquad \operatorname{Acc}^{*}(Y\mid V)=1-q Y is a fair bit; each V independently flips Y with probability q in [0,1/2]. Accuracy concerns one view. (4) I(Y;V)=1-h_2(q),\qquad h_2(q)=-q\log_2q-(1-q)\log_2(1-q) Mutual information is in bits for the declared binary symmetric channel, with 0 log 0 defined as zero. Analytical figures Figure 1. Pair agreement and label accuracy differ Constructed fair binary channel. Each view flips Y independently with probability q. Pair agreement is 1-2q(1-q); single-view Bayes accuracy is 1-q. Neither curve is a trained model result. figures/contrastive-representation-geometry-1.csv Figure 2. Temperature changes a fixed candidate loss Constructed candidate set with positive similarity 1 and every negative similarity 0. Curves evaluate log(1+K exp(-1/tau)) for K=3,15,63; similarities are held fixed. figures/contrastive-representation-geometry-2.csv Comparison Diagnostic | What it can establish | What remains untested Positive alignment | Agreement under the specified view law | Preservation of task-relevant distinctions Uniformity | Coverage under a chosen geometry and marginal | Semantic identity of the encoded information Linear probe | Accessibility to the specified linear learner | Availability to arbitrary downstream procedures Finite contrastive loss | Candidate discrimination under the sampler | Universal transfer or absence of false negatives Proposed validation protocol 1. Enumerate the two-bit channel and record which variables every augmentation preserves. 2. Prespecify downstream tasks, countertasks and probe attachment layers before tuning. 3. Vary temperature and negative count separately from semantic collision probability. 4. Match training data exposure and tune optimization budgets symmetrically. 5. Report alignment, marginal geometry, downstream risk and independent-seed uncertainty together. 6. Release view-generation code and examples where invariance destroys useful information. Limitations 1. The binary construction is a finite information counterexample, not a realistic image distribution. 2. The candidate-loss curves hold similarities fixed and cannot predict trained-network accuracy. 3. Three foundational sources do not form a systematic survey of all contrastive methods. 4. No neural training, benchmark replication or independent peer review was performed. Research questions 1. Can an augmentation policy preserve a declared portfolio of downstream distinctions without labels for every task? 2. How should false-negative rates be estimated when categories overlap rather than partition the data? 3. Which layer-level diagnostics predict accessibility under a fixed downstream compute budget? Source evidence dl-06-1: Augmentation, projection-head and training-scale choices are evaluated in SimCLR. Boundary: The reported comparisons concern the paper training and evaluation settings. Locator: Abstract and framework/ablation sections. dl-06-2: Alignment and uniformity provide a normalized representation geometry and asymptotic analysis. Boundary: The assumptions do not imply task-independent semantic sufficiency. Locator: Abstract and definitions of alignment and uniformity. dl-06-3: CPC connects prediction in latent space to a negative-sampling information objective. Boundary: Sampling assumptions and the identity of random variables determine the information interpretation. Locator: Section 2 and information-bound discussion. References dl-06-1. Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning (2020). https://proceedings.mlr.press/v119/chen20j.html dl-06-2. Tongzhou Wang, Phillip Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. Proceedings of the 37th International Conference on Machine Learning (2020). https://proceedings.mlr.press/v119/wang20k.html dl-06-3. Aaron van den Oord, Yazhe Li, Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv (2018). https://arxiv.org/abs/1807.03748 | DOI: 10.48550/arXiv.1807.03748 dl-06-1 access: Primary paper PDF text and primary bibliographic landing page inspected. dl-06-2 access: Primary paper PDF text and primary bibliographic landing page inspected. dl-06-3 access: Primary paper PDF text and primary bibliographic landing page inspected. Version note: First posted in 2018; the accessed PDF is arXiv v2 dated 22 January 2019. This entry cites the preprint, not an asserted conference publication. Provenance A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography. Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article. Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks. This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems.