Transferability of Multimodal Representations: Contrastive Objectives, Information Bottlenecks, and Cross-Distribution Failures

1. ViT organizes images into token sequences

ViT divides an image into patches, maps them to vectors, and then processes them using a Transformer. Important conditions in the paper include large-scale pretraining and subsequent transfer; its findings do not mean that “a pure Transformer necessarily outperforms convolution on any small dataset.” Architectural comparisons should hold pretraining data, augmentation methods, and computational budgets constant; otherwise, differences in data investment may be misrepresented as the contribution of architecture alone.[1] An Image is Worth 16x16 Words: Transformers for Image R

2. CLIP changes the supervision interface through text

CLIP learns contrastive representations from paired image–text data, bringing matching images and texts closer together and distinguishing them from mismatched combinations. The paper reports a training dataset of 4 hundred million image–text pairs. At inference time, candidate classes can be written as text, and similarity can be used for zero-shot classification. Here, “zero-shot” means that the corresponding training was not performed on the downstream classification training set; it does not mean that the model has never encountered related concepts.[2] Learning Transferable Visual Models From Natural Langua

3. Similarity scores depend on the candidate set

If the candidate texts contain only two classes, the model is forced to choose between those two descriptions; this does not establish that the image necessarily belongs to either class. Prompt wording, class granularity, and multi-object scenes affect how the model can be used. This review recommends retaining the candidate texts, image preprocessing, and rejection rules together, and evaluating recognition, counting, relations, and text reading separately; a single classification score should not stand in for all visual tasks.作者分析与复核建议 / Author analysis

4. Perceiver processes inputs through a latent bottleneck

Perceiver uses asymmetric attention to aggregate high-dimensional inputs into a smaller set of latent variables, then processes information within that latent set. It addresses the organizational relationship between input scale and internal computation, a different dimension from CLIP's image–text contrastive objective. Introducing a bottleneck raises questions of information selection: whether important details are retained must be tested under specific tasks and resolution conditions.[3] Perceiver: General Perception with Iterative Attention

5. The analytical relationship between resolution and patching

For a square image with side length H and patches with side length P, where H is divisible by P, the number of image patches is$\left(\frac{H}{P}\right)^2$. Taking$P=16$, a side length of 224 pixels produces 196 patches, while 448 produces 784 patches; counting only all pairings between image patches gives 38416 and 614656, respectively, a 16-fold increase. This is a separate counting example and does not correspond to the temperature curves in Figure 2 of this version; it excludes the classification token, batches, and channels, and is not a measurement of actual FLOPs or GPU memory.[1] An Image is Worth 16x16 Words: Transformers for Image R

6. Input budgets should be chosen according to the task

Reading small text and judging broad outlines impose different requirements on information retention. This review recommends first identifying the minimum visual cues required by the task, then comparing resizing, cropping, and patching strategies. If a high-resolution image is reduced until its text is unreadable, blaming the reasoning module for an incorrect table calculation conflates input loss with reasoning error. Evaluation should preserve the correspondence between the original image and the model's actual input.作者分析与复核建议 / Author analysis

7. Cross-modal evaluation must trace the entire pipeline

A system may use an image encoder, a projector, a language model, and external OCR. The version and training boundaries of each module should be specified, with records of which information comes from the original image and which comes from tools. If an answer contains attributes absent from the image, one should investigate whether they were introduced by perception, textual priors, or generation; errors can only be localized if module inputs and outputs are retained.作者分析与复核建议 / Author analysis

Aligning representations does not imply aligning objects individually

An objective that matches an entire image with a sentence can encourage conceptual correspondence without necessarily providing pixel-level localization, object binding, or complete spatial relations. The evidence needed to answer “the red object is to the left of the blue object” differs from that needed to judge whether an image is related to the word “street.” Thus, when citing image–text alignment research, one should specify which transfer task has been demonstrated, avoiding the extension of similarity capabilities to all forms of visual reasoning.作者分析与复核建议 / Author analysis

8. Contrastive learning actually optimizes relative discrimination

For an image and a set of candidate texts, the training objective increases the normalized probability of the correct pairing relative to other candidates. It constrains relative scores within the batch or candidate set, rather than the full truthfulness of natural-language descriptions. If negative samples differ mainly in object category, the model can complete the task using coarse category cues without correctly encoding every spatial relation. It is therefore necessary to examine negative-sample construction, duplicate pairings, and semantic neighbors, rather than infer fine-grained compositional ability solely from declining training loss.[2] Learning Transferable Visual Models From Natural Langua

9. Temperature changes which samples dominate the gradient

At lower temperatures, similarity differences are amplified, and high-scoring negative samples matter more to the normalization denominator and the gradient; higher temperatures produce a smoother distribution. A simple numerical example fixes the positive similarity at 0.8 and 31 negative similarities at 0.2, illustrating how probability changes with temperature. These values are pedagogical settings, not CLIP measurements. During actual training, similarities, batch size, the negative-sample distribution, and temperature change together; this single-variable curve cannot explain the training dynamics of the entire model.[2] Learning Transferable Visual Models From Natural Langua

10. Compressed representations may not match the information required by the task

Compressing an entire image into a finite vector is suitable for comparing overall semantics, but may weaken distinctions involving precise positions, local text, and multiple similar objects. Patch partitioning and positional representations in vision Transformers provide a structured interface; Perceiver processes inputs using a smaller latent set, addressing a different organization of computation. Whether representation compression loses information needed for a task should be tested through localization, counting, or dense prediction. Promptable segmentation models such as SAM use still different objectives and outputs and cannot be compared solely through image–text retrieval performance.[1] An Image is Worth 16x16 Words: Transformers for Image R[3] Perceiver: General Perception with Iterative Attention[4] Segment Anything

11. Compositional generalization requires systematic counterfactual controls

If red objects almost always occur with a particular type of background during training, changing backgrounds or swapping attributes at test time may expose shortcuts. Paired tasks can be constructed in which objects remain unchanged while color, quantity, and spatial relations are varied separately, with text length and prompts controlled. Retrieval results should also retain the actual candidate set: a correct retrieval among easy candidates does not imply that similar relational descriptions can still be distinguished. Results for different languages and visual domains should also be reported separately to prevent aggregate means from concealing systematic failures.[2] Learning Transferable Visual Models From Natural Langua[3] Perceiver: General Perception with Iterative Attention

Core equations and derivations

Method and evidence workflow

Analytical figures and numerical examples

Recomputable worked example

With the positive-sample similarity fixed at 0.8 and all 31 negative-sample similarities fixed at 0.2, the positive-sample probability is approximately 0.929 at temperature 0.1 and approximately 0.056 at temperature 1. Changes in normalized scores do not demonstrate changes in representations or actual recognition capabilities.

Method comparison

Verification and experimental plan

1. Fix the versions of images, texts, and candidate sets, and record licenses, duplicate pairings, splits, and prompting methods.
2. Evaluate retrieval, classification, local localization, counting, and relational judgments separately, avoiding the use of a single task's performance to represent all multimodal understanding.
3. Change only one attribute on the same objects, construct counterfactual controls for backgrounds and positions, and have humans verify the consistency of test items.
4. Sweep candidate counts and the difficulty of similar negative samples, saving the candidate texts and image regions associated with each error.
5. When comparing computational schemes, fix the task interface, input resolution, and quality target, and report input compression and changes in model objectives separately.
6. Report performance separately by language, culture, and visual domain, disclose failures and indeterminate items, and do not treat the average score as a guarantee across all domains.

Scope and limitations

• Fixed-score curves are neither training dynamics nor real-model results.
• Correlations in paired data do not guarantee causal understanding of semantics.
• Different multimodal tasks require different outputs and ground truths.
• No new visual or linguistic cross-domain benchmarks are introduced.

Mapping primary sources to claims

1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleDosovitskiy et al. · 2020-10-22v2; architecture and scaling experiments
2. Learning Transferable Visual Models From Natural Language SupervisionRadford et al. · 2021-02-26v1; contrastive training and zero-shot transfer
3. Perceiver: General Perception with Iterative AttentionJaegle et al. · 2021-03-04v2 abstract; latent bottleneck architecture
4. Segment Anything, , , , , , , , , , , · 2023-04-05Abstract; 原文方法、理论和实验章节见作者原始论文链接