When a Sensor Is Missing: Multimodal Learning and Identifiability

1. A missing sensor changes the inference problem

Multimodal models combine observations that may contain complementary and redundant information. When one modality disappears, the remaining inputs do not necessarily determine what was lost. A model can still make a prediction, but its ability to output a plausible value should not be confused with identifying the missing measurement or preserving the uncertainty of the complete-data problem. The relevant question is which information remains available under the actual missingness mechanism.

This review separates three issues: architecture compatibility with arbitrary modality subsets, statistical identification of a target from those subsets, and calibration under a changed availability process. A Gaussian sensor construction gives an exact conditional-variance floor. A binary selection construction shows that identical observed complete cases can coexist with different outcomes among missing cases. Both are author-created finite models, not new clinical, speech, or vision benchmarks. They demonstrate why robustness to random masking is only one part of a deployment claim and why imputing a detailed missing modality cannot manufacture evidence that was never observed.

2. What the selected primary models establish

Suzuki, Nakayama, and Matsuo study joint multimodal learning with deep generative models. Wu and Goodman use product-of-experts inference for combinations of missing modalities. Shi and colleagues study mixture-of-experts multimodal variational models and distinguish joint generation, cross-generation, and useful shared representations. These primary records motivate different posterior aggregation strategies rather than a single universal recipe.

Havaei and colleagues combine available modality embeddings for image segmentation. Hazarika and colleagues distinguish modality-invariant and modality-specific representations. Daunhawer and colleagues identify limitations of a studied family of mixture-based multimodal variational objectives. We use the inspected author abstracts for these scope-level statements, without independently reproducing their experiments or transferring their reported performance to new settings. The analysis below focuses on conditional information and missingness assumptions that apply before selecting a particular architecture. An implementation may handle every subset syntactically while remaining poorly calibrated for rare or systematically missing subsets. The scientific question is therefore broader than whether the forward pass accepts a mask.

Sources: [1], [2], [3], [4], [5], [6]

3. Redundancy and complementarity have different consequences

If two sensors measure nearly the same underlying signal with independent noise, combining them can reduce uncertainty. If they measure different target-relevant factors, losing one may remove information that the other never contained. A shared representation objective can encourage agreement, but agreement alone does not reveal which modality-specific information is necessary for the task.

The distinction should be reflected in evaluation. Cross-modal reconstruction may be easy when modalities are highly redundant and fundamentally ambiguous when they contain private factors. A model can produce a typical missing view by drawing on the training distribution, yet the generated details may not correspond to the particular instance. The correct output may therefore be a distribution or an explicit indication of uncertainty rather than a single confident reconstruction. A useful benchmark includes targets that depend on shared factors, private factors, and their interactions. Otherwise a method optimized for common information can appear universally robust while silently discarding exactly the features that a downstream application needs when a sensor fails.

4. A Gaussian model gives an irreducible error

Let independent standard-normal variables S and P represent a shared and a private factor. Suppose modality X observes S exactly, modality Z observes P exactly, and the target is Y=S+alpha P. With both modalities, Y is determined in this noiseless construction. With only X, the optimal squared-error predictor is X and the conditional mean-squared error is alpha squared. No additional training data or larger network can reduce that Bayes error without additional information about P.

Figure 1 varies alpha and compares complete-data error with the missing-private-modality error. The curve is an information bound for this declared distribution, not an empirical model-performance curve. It shows why a method can be excellent when alpha is small and fail when the target depends strongly on the private factor. A benchmark averaging across tasks with different alpha values can hide that dependence. The executable verifies the conditional calculation with finite symmetric moment constructions and checks the alpha=0 control, where the missing modality is irrelevant to the target. Real modalities are rarely this cleanly separated, but the distinction between missing information and poor optimization remains.

5. Noisy observations yield precision arithmetic

Consider a latent scalar S with unit prior variance, observed through two conditionally independent Gaussian sensors with noise variances v1 and v2. With both sensors, posterior variance is 1/(1+1/v1+1/v2). With only the first, it is 1/(1+1/v1). The gain from a second sensor depends on its precision, not merely on the number of modalities. A nearly useless sensor adds little information, while a precise sensor can dominate.

This calculation provides a control for posterior aggregation. Product-like precision addition is appropriate only under the declared conditional independence and calibrated noise model. If two encoders report correlated evidence as independent, the combined posterior can become overconfident. Conversely, a mixture representation may preserve alternative explanations rather than simply sharpen one. These are different inferential roles. A neural aggregation rule should be evaluated against the assumptions it implicitly makes, with correlated-noise controls and missing-subset calibration. The scalar example does not prove that a particular variational architecture is correct or incorrect; it supplies an exact baseline on which an implementation's uncertainty behavior can be checked.

6. Duplicate evidence should not count twice

Two channels can contain the same measurement under different encodings. Treating them as independent observations doubles apparent evidence without adding information. In the Gaussian control, if the second sensor is an exact copy of the first, the correct posterior variance equals the single-sensor variance, not the independent-sensor formula. For noise variance v, the absolute variance gap is v/[(v+1)(v+2)]; it approaches zero for both extremely precise and extremely noisy measurements. Relative overconfidence grows with precision: the correct variance divided by the falsely independent variance is (v+2)/(v+1), approaching two as v tends to zero. Absolute and relative errors therefore have different trends.

A practical multimodal dataset can contain similar dependencies through shared preprocessing, common acquisition hardware, or labels embedded in multiple channels. The model may learn to exploit them, but uncertainty estimates should not pretend that each channel is an independent confirmation. An evaluation can create a duplicate-channel negative control and compare it with a genuinely independent noisy measurement. It should also distinguish independent conditional noise from marginal correlation caused by the shared latent signal. Correlated observations are not automatically redundant, and independent-looking embeddings are not proof of independent evidence. The relevant dependency is part of the generative and measurement model, which should be stated before assigning a probabilistic meaning to a fused representation.

7. Missingness itself can depend on the target

Random masking assumes that a modality is removed independently of the underlying instance, or at least according to a specified observable rule. Real sensor failure, acquisition decisions, and user behavior may depend on the target or on unobserved factors. A model evaluated under random dropout can therefore face a different conditional distribution when deployed.

Let M indicate whether a modality is observed. The target distribution among missing cases is P(Y given X,M=0), which need not equal P(Y given X,M=1). Conditioning on the mask may be essential, but observing the mask alone does not identify the missing-case distribution from complete cases. A study should describe why each modality is absent, whether the mechanism changes across sites or time, and whether outcomes remain observed for missing cases. These questions determine which quantities are estimable. They also prevent a misleading interpretation of robustness: maintaining accuracy under synthetic independent masking is evidence for that intervention on the data, not a guarantee under every real process that produces missing inputs.

8. A selection example exposes nonidentifiability

Consider a binary target Y with prevalence p. Let a be the probability that the modality is observed when Y=1 and b the corresponding probability when Y=0. The observed-case positive rate is pa divided by pa+(1-p)b. Different triples (p,a,b) can yield the same observed-case rate and the same overall observation rate while implying different positive rates among missing cases.

Our construction fixes the observed fraction at one half and the positive rate among observed cases at one half. Then pa=1/4 and (1-p)b=1/4. Any p between one quarter and three quarters yields valid a and b. The missing-case positive rate becomes 2p-1/2, ranging from zero to one. Figure 2 plots this family. Complete-case data alone cannot select one point on the curve. The result changes if outcomes are observed for missing cases, if the selection mechanism is known, or if justified structural restrictions are added. Those are additional sources of information, not benefits obtained merely by training a more flexible imputer.

9. An imputed modality is a prediction, not an observation

A generative model can sample a plausible missing modality conditional on the available inputs. That sample is useful for some downstream procedures, but it remains model-generated evidence. Feeding it into a second predictor as if it were an independent sensor can hide uncertainty and amplify model assumptions. Multiple imputations may represent uncertainty better, yet their validity depends on whether the conditional distribution is correctly specified and whether the missingness mechanism is addressed.

In the shared/private Gaussian example, generating P from its prior produces realistic private factors but does not identify the actual P of the instance. A downstream target prediction should integrate over that uncertainty rather than treat one sampled value as truth. In a visual task, a sharp synthesized image can be especially misleading because perceptual detail suggests certainty. Evaluation should compare conditional coverage, downstream risk, and sensitivity to imputation choices, not just visual realism. The review's finite models do not assess image quality. They show why realism and instance-specific correctness are different claims that require different evidence.

10. Modality dropout changes the training objective

Training with randomly selected modality subsets optimizes an average over the chosen subset distribution. If common subsets receive most of the probability mass, rare but operationally important failures may contribute little to the objective. Uniform sampling over subsets is not necessarily uniform over real deployment events, and with many modalities the number of subsets grows rapidly.

The subset policy should therefore be explicit. A deployment-oriented objective might weight subsets by their observed frequency, by their operational cost, or by a robustness requirement. These choices can conflict. Report per-subset performance and the weighting used to form any aggregate score. A model that improves the average may still degrade a critical missing-sensor case. Selection should also account for how much training data exist for each subset and whether artificial masking preserves the actual acquisition process. The proper comparison is between complete procedures with declared subset distributions, rather than between architecture names under an unspecified random-mask routine.

11. Shared representations can suppress useful private factors

A loss encouraging modality agreement may reduce nuisance variation, but it can also penalize information present in only one modality. If the target depends on that private information, perfect alignment can conflict with prediction. A representation should not be judged solely by how similar its modalities become. The downstream task determines which differences should remain.

A useful experiment varies the target's dependence on shared and private factors while holding the observation process fixed. Compare a shared-only representation, a shared-plus-private representation, and a task-specific baseline. Evaluate both complete and missing subsets. In the Gaussian construction, increasing alpha makes private information progressively more important, yielding an exact reference for this tradeoff. A neural model's performance can then be interpreted relative to the information available rather than to an unattainable complete-data score. This does not prescribe one architecture for all applications. It specifies a test that can reveal whether the objective is discarding signal relevant to the stated task.

12. Calibration must be conditional on availability

A model can be calibrated on the overall population while overconfident on a rare missing-modality subgroup. Aggregate reliability diagrams can hide this if most observations contain all sensors. Uncertainty evaluation should therefore condition on the availability pattern and, where relevant, on data quality, site, or time. The number of samples in each subgroup must be reported.

For regression, examine interval coverage and width under each subset. For classification, compare predicted probabilities with observed frequencies and evaluate a proper scoring rule. A confidence score derived from agreement among modalities is unavailable or altered when a modality disappears, so its fallback behavior must be defined. Out-of-distribution missingness may require abstention or escalation rather than an imputed certainty. These are proposed evaluation requirements, not claims about the safety of any clinical system. The finite Gaussian posterior is an exact reference only under its assumptions. Real calibration requires representative outcomes and should not be inferred from the appearance of a posterior variance produced by a neural encoder.

13. A deployment study should replay acquisition failures

A realistic study distinguishes missing at training time, missing at inference time, corrupted-but-present inputs, delayed inputs, and changes in the reason for missingness. These conditions produce different risks. A mask that says a modality exists does not guarantee that it is aligned, correctly calibrated, or informative. Data quality metadata can matter as much as availability.

Replay actual acquisition patterns when such records are available, with suitable privacy and governance. Include independent random masking only as a controlled comparison. Freeze preprocessing and imputation models before evaluation, and preserve the exact subset and quality flags used for every prediction. Compare with simple available-sensor baselines so that a complex fusion model must demonstrate a benefit beyond a robust fallback. Report latency and resource use when generating missing modalities, since an accurate imputation that misses the decision deadline may not be useful. The study should retain failures and unsupported subsets rather than quietly exclude them from the reported aggregate.

14. The executable checks preserve the information boundary

The accompanying standard-library program computes the private-factor error floor, independent-sensor posterior variances, the duplicate-sensor control, and the binary selection family. It verifies probability normalization, the fixed observed-case distribution, and the changing missing-case prevalence. Figure points come directly from these declared equations. No private observations, trained generative model, or measured sensor data are used.

These checks establish counterexamples and exact identities for the stated finite models. They do not estimate the prevalence of missingness problems in a real application or prove that every multimodal model is miscalibrated. Their usefulness is diagnostic: they distinguish a missing-information limitation from a failure of an estimator to use available information. A system that violates an exact control deserves investigation, while a system that passes still needs domain-specific evaluation. The next empirical step is to identify which assumptions about shared factors, conditional independence, and acquisition mechanisms are defensible for the actual modalities and outcomes of interest.

15. Alignment errors are not equivalent to missing inputs

A modality can be present but paired with the wrong instance or time. A video frame shifted relative to audio, a sensor timestamp offset, or an incorrectly matched record can create misleading agreement or disagreement. A model trained only with missing-modality masks may have no reason to recognize this condition. Treating the corrupted channel as observed can be worse than omitting it.

An evaluation should include controlled misalignment as a separate axis. Shift one stream by known offsets while leaving the others fixed, and compare performance with both the correctly aligned case and an explicit missing-channel fallback. Record whether the model's uncertainty increases or whether fusion amplifies the inconsistency. The target itself must be aligned to the intended decision time. A high-quality feature extractor cannot repair an incorrect pairing by assumption.

This issue also affects cross-generation metrics. A synthesized modality can appear compatible with the available context while failing to match the particular held-out measurement because that measurement was misaligned. Conversely, a model may learn a dataset-specific offset that disappears in deployment. Data provenance should therefore include pairing rules, synchronization tolerances, and preprocessing versions. The Gaussian and binary constructions assume correct instance alignment; their bounds do not quantify synchronization errors. Adding this explicit limitation makes the experimental contract stronger by identifying a common failure mode that lies outside the missingness model rather than silently absorbing it into unexplained prediction error.

16. Robustness means knowing what is no longer knowable

A model that accepts any modality subset has solved an interface problem. A model that remains accurate and calibrated under a specified missingness process has solved a harder statistical problem. A model that recognizes when the available observations cannot identify the target addresses a different and equally important requirement. These achievements should be distinguished in claims and evaluation.

The two constructions show why complete-data performance cannot be inherited automatically by a missing-data predictor. Private factors create irreducible uncertainty, and selection mechanisms can leave missing-case outcomes unidentified even when complete-case statistics look stable. Future work should connect subset-aware training with availability-conditioned calibration, explicit acquisition models, and principled abstention. Generative completion can remain a useful tool within that framework, provided synthesized detail is not treated as an additional measurement. The goal is not to make every missing sensor appear present. It is to produce an honest and useful decision from the information that actually remains.

Core equations and definitions

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Separate absent, delayed, corrupted and misaligned channels.
• Report per-subset metrics and subset weights.
• Test duplicate evidence and private-factor controls.
• Evaluate calibration conditional on availability.
• Replay realistic acquisition mechanisms with robust available-sensor baselines.

Limitations

• No clinical, imaging, speech or sensor deployment is evaluated.
• The Gaussian example assumes exact shared/private separation.
• The selection nonidentifiability assumes outcomes for missing cases are unavailable.
• Primary abstract scope only; source empirical claims are not reproduced.

Open research questions

• Which missingness mechanisms can be identified from available outcomes?
• How should fusion detect duplicate or correlated evidence?
• When should a model abstain rather than generate a missing modality?

Evidence and its boundaries

Joint variational models connect inference across modalities.

[1] — Scope-level source statement only; no independent reproduction, priority ranking or theorem-proof audit. Inspected: Primary author abstract and bibliographic record.

Product-of-experts inference supports combinations of missing modalities.

[2] — Scope-level source statement only; no independent reproduction, priority ranking or theorem-proof audit. Inspected: Primary author abstract and bibliographic record.

Mixture inference emphasizes joint and cross-generation properties.

[3] — Scope-level source statement only; no independent reproduction, priority ranking or theorem-proof audit. Inspected: Primary author abstract and bibliographic record.

Modality embeddings can be combined over the available subset.

[4] — Scope-level source statement only; no independent reproduction, priority ranking or theorem-proof audit. Inspected: Primary author abstract and bibliographic record.

Shared and modality-specific representation subspaces serve different roles.

[5] — Scope-level source statement only; no independent reproduction, priority ranking or theorem-proof audit. Inspected: Primary author abstract and bibliographic record.

Subsampling imposes limitations on the studied family of mixture-based multimodal variational objectives.

[6] — Scope-level source statement only; no independent reproduction, priority ranking or theorem-proof audit. Inspected: Primary author abstract and bibliographic record.