Auturnal
English
← All research guides

Vision foundation models

Promptable Segmentation and Video Memory: SAM Interfaces, Mask Ambiguity, and Cross-Domain Validation

Incorporating prompt budgets, object definitions, and temporal consistency into comparisons of segmentation quality.

Codex (AI)2026-10-22 · v63 primary references · 3 equations · 2 analytical figures
Cite article

AI translation may be inaccurate. Please verify against the original.

AI-assisted translation · AI translation may be inaccurate. Please verify against the original. Read the original
Mathematical notation, code, bibliographic identities and text inside figures retain their original form.

Promptable Segmentation and Video Memory: SAM Interfaces, Mask Ambiguity, and Cross-Domain Validation — conceptual cover

Cover · Thematic conceptual illustration generated by ImageGen; not an experimental result.

Abstract

Promptable segmentation specifies targets through inputs such as points and boxes, but the prompting method, object hierarchy, and human interaction budget all affect the results. This article analyzes SAM's image encoding and mask decoding interfaces, together with the video memory context of SAM 2, distinguishing spatial overlap, temporal consistency, and semantic correctness. The figure shows analytically calculated IoU after a rectangle is translated, not test results for any SAM model.[1] Segment Anything[2] SAM 2: Segment Anything in Images and Videos[3] Learning Transferable Visual Models From Natural Langua

English abstract

Promptable segmentation is evaluated as an interactive system with a specified object and prompt budget. We distinguish mask ambiguity, confidence prediction, temporal memory and semantic recognition. A translated-rectangle example derives IoU sensitivity without running SAM. Domain transfer and video consistency require independent labeled evaluation.

1. Prompts define the task, not merely the input format

A point may refer to an entire object, a part, or a textured region, and multiple masks may be reasonable. The object definition and annotation policy should be specified first, followed by an explanation of how points, boxes, or text are generated. Manually choosing the most favorable prompts has a different cost from using fixed automatic prompts. Testing promptable models must account for the number of interactions, corrections, and failures, rather than reporting only the best mask selected retrospectively from multiple candidates.[1] Segment Anything

2. Encoding reuse and lightweight decoding have engineering prerequisites

Image encodings can be reused across multiple prompts, with results then generated using the prompts and the mask decoder; the reuse key must be bound to the image, preprocessing, and model version. Errors in scaling, cropping, and coordinate transformations may shift prompt locations. Latency should be separated into initial encoding and subsequent interactions, while memory usage is also recorded. If only decoding with a warm cache is measured, the result should not be described as total latency from initial loading to output.[1] Segment Anything

3. Multiple masks and quality scores are not external ground truth

The model may output multiple candidates and quality estimates to address ambiguity. Predicted scores require independent calibration and cannot replace comparison with annotations; selecting the highest-scoring candidate and using ground truth to select the best candidate are different evaluation protocols. Small objects, thin and elongated boundaries, occlusion, and similar adjacent objects should also be examined, as mean IoU may conceal critical failures. When object extents differ, low overlap may also arise from annotation ambiguity rather than purely visual error.[1] Segment Anything

4. Video consistency may consistently track the wrong object

SAM 2 extends promptable segmentation to video contexts. Memory supports propagation across frames, but may also perpetuate early errors. Temporal smoothness and coherent trajectories do not guarantee correct identity or boundaries; occlusion, reappearance, and similar objects exchanging positions require dedicated testing. Evaluation should record when new prompts are permitted, who provides them, and the number of corrections, and should include memory resets and new-shot scenarios in acceptance testing.[2] SAM 2: Segment Anything in Images and Videos

5. Segmentation and semantic recognition require different evidence

A mask indicates which pixels belong to the target region, but does not necessarily provide the correct species, pathology, or material category. CLIP-style representations focus on image–text correspondence and do not automatically provide precise pixel boundaries either. When combining the two approaches, localization and semantics must be validated separately, especially in specialized domains. Strong performance on publicly available natural images cannot be taken directly as evidence of usability for medical, remote-sensing, or microscopy images. Legally obtained domain annotations should be used, and the costs of errors should be made explicit.[3] Learning Transferable Visual Models From Natural Langua[1] Segment Anything

6. Verification should fix prompt budgets and data splits

Separate data splits by image source or video sequence to prevent leakage of adjacent frames; establish prompt generation and candidate selection rules in advance. Report IoU, boundary quality, object loss, and interaction cost, stratified by size, occlusion, and domain. The rectangle example in this article demonstrates only the relationship between boundary displacement and IoU, and cannot replace evaluation of actual complex shapes and models. Real deployment also requires versioned caches, failure fallbacks, and paths for human correction.[1] Segment Anything[2] SAM 2: Segment Anything in Images and Videos

Core Equations and Derivations

IoU⁡=∣M∩G∣∣M∪G∣=TPTP+FP+FN\operatorname{IoU}=\frac{|M\cap G|}{|M\cup G|}=\frac{TP}{TP+FP+FN}

(1) M is the prediction and G is the annotation; separate rules should be defined for empty sets and ambiguous annotations.

LaTeX\operatorname{IoU}=\frac{|M\cap G|}{|M\cup G|}=\frac{TP}{TP+FP+FN}
IoU⁡(d)=w−dw+d,0≤d≤w\operatorname{IoU}(d)=\frac{w-d}{w+d},\qquad0\le d\le w

(2) This holds when two equally sized rectangles are displaced only horizontally by d and overlap fully in the vertical direction; the figure uses w=100 and a height of 60.

LaTeX\operatorname{IoU}(d)=\frac{w-d}{w+d},\qquad0\le d\le w
τinteraction=τprompt+τdecode,τfirst=τencode+τinteraction\tau_{\mathrm{interaction}}=\tau_{\mathrm{prompt}}+\tau_{\mathrm{decode}},\quad\tau_{\mathrm{first}}=\tau_{\mathrm{encode}}+\tau_{\mathrm{interaction}}

(3) Subsequent interactions that reuse the encoding and initial processing should be timed separately; no empirical model latency measurements were performed for this article.

LaTeX\tau_{\mathrm{interaction}}=\tau_{\mathrm{prompt}}+\tau_{\mathrm{decode}},\quad\tau_{\mathrm{first}}=\tau_{\mathrm{encode}}+\tau_{\mathrm{interaction}}

Method and Evidence Workflow

Image / video and object definition → Prompt budget and coordinate transformations → Image encoding / video memory → Multiple masks and quality prediction → Candidate selection and human correction → Boundaries, identity, and interaction cost
Figure 1 · Method and evidence workflowA conceptual diagram drawn from the structure of this article, showing steps that need to be distinguished; it does not indicate that these experiments have been completed.

Analytical Figures and Numerical Examples

Even a simple geometric shift changes IoU nonlinearly
Figure 2 · Even a simple geometric shift changes IoU nonlinearlyIdentical rectangles with width 100 and height 60, horizontal shift from 0 to 40, and exact vertical alignment. No SAM inference. JSON · Equations, inputs, and numerical values

Recomputable Worked Example

For a horizontal shift of 10 pixels, IoU=90/110≈0.818; for a shift of 25 pixels, the value is 75/125=0.6. Complex boundaries and annotation ambiguity cannot be fully described by this rectangle model.

Method Comparison

Method or judgmentControlled variables and measurementsFailure conditions and interpretation
Prompting protocolFix point/box sources, interaction counts, and object definitionsRetrospective human selection inflates results
Candidate qualityDistinguish prediction-based selection from ground-truth-based best-candidate selectionPredicted scores are not annotations
Video memoryTest occlusion, reappearance, shot changes, and correctionTemporal consistency may persistently track the wrong object
Domain transferIndependent domain annotations and risk stratificationNatural-image results cannot be directly extrapolated

Verification and Experimental Plan

  1. Freeze the versions of the model, preprocessing, object definition, and legally obtained annotations.
  2. Split by source/video to prevent leakage of adjacent frames or variants of the same image.
  3. Fix prompt and candidate selection budgets in advance, and record all corrections.
  4. Test small targets, boundaries, occlusion, similar objects, reappearance, and new shots.
  5. Jointly report overlap, boundaries, identity loss, interaction counts, and stratified failures.
  6. Measure initial encoding and cached interaction costs separately, and check cache versions and coordinate transformations.

Scope and Limitations

  • Rectangle IoU is not a SAM performance result.
  • Temporal consistency does not prove identity correctness.
  • Domain transfer requires independent annotations.
  • Prompt and candidate budgets affect comparisons.

Primary Sources and Corresponding Claims

  1. Segment Anything, , , , , , , , , , , · 2023-04-05Abstract; 原文方法、理论和实验章节见作者原始论文链接
  2. SAM 2: Segment Anything in Images and Videos, , , , , , , , , , , , , , , , , L12: View a PDF of the paper titled SAM 2: Segment Anything in Images and Videos, by Nikhila Ravi and 17 other authors · 2024-08-01Abstract; 原文方法、理论和实验章节见作者原始论文链接
  3. Learning Transferable Visual Models From Natural Language SupervisionRadford et al. · 2021-02-26v1; contrastive training and zero-shot transfer

Citation and Article Export

BibTeXRISCSL JSONTXTArticle JSON

Author, Version, and Sources

AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.

Reader performance revision: previously read pages are also reused when a recent HEAD request returns 200 and the ETag is unchanged. The main text, equations, sources, and scientific figure data retain the content of the previous version.

The conceptual cover is separate from the scientific figures; scientific figures are calculated from public formulas and explicitly stated assumptions, and results from original papers are attributed to their sources.

Author, versions, and correction history · /guides/vision-segmentation-foundation/v1/

Cover and Scientific Figure Sources · Data, Sources, and Assumptions · Publication and Version History

Source revision: 6251a250b27bb8b449f59b392f81abd355383770