The Bits the Model Must Actually Send: Learned Image Compression and Entropy-Model Mismatch

1. A latent array is not yet a compressed file

A neural encoder can produce a compact array without providing a complete compression system. A decoder must know how to interpret the array, and a bitstream must carry enough information to reconstruct that interpretation. Quantized symbols, probability tables, image dimensions, side information and model compatibility all affect the result. Ignoring any required component can make a reported rate artificially small.

This article focuses on that accounting boundary. We distinguish source entropy, model cross-entropy, training-time rate estimates and actual coded bytes. The quantities are related under stated assumptions, but they are not interchangeable. We also separate distortion of a particular source image from the perceptual plausibility of a reconstruction. The finite examples use a binary source because its probabilities can be written exactly and every expected code-cost term can be inspected. They do not simulate natural-image quality or benchmark a learned codec. Their purpose is to expose which bits and assumptions a compression claim must include.

2. Joint optimization does not remove quantization

Ballé, Laparra and Simoncelli describe nonlinear analysis and synthesis transforms with quantization and joint rate-distortion optimization, using a continuous proxy during training. This makes the transform and coding model learnable together. The training relaxation is useful for optimization, but deployment still requires discrete symbols and a specified reconstruction operation.

A comparison should identify whether its rate is calculated from noisy continuous latents, rounded symbols, an ideal probability model or the final byte stream. A mismatch can arise even when the neural weights are unchanged. The distortion term also depends on the actual reconstruction path and precision. A useful implementation check runs the complete encoder and decoder on a small fixed image set, compares decoded outputs with the evaluation script's outputs and verifies that every required bit is counted. The existence of an end-to-end objective does not make that engineering check redundant. It establishes a training framework, while the executable codec supplies the evidence for a deployment rate-distortion point.

Sources: [1]

3. An entropy model pays for its mistakes

Let a binary source emit one with probability p and let an ideal coding model assign probability q to that event. The expected negative log probability is minus p log base two q minus one minus p times log base two of one minus q. When q equals p, this is the binary entropy. When the coding model is mismatched, the excess expected cost is the finite-distribution Kullback-Leibler divergence.

For p equal to 0.9, the entropy is approximately 0.468996 bits per symbol. A uniform model has cross-entropy one bit, while q equal to 0.8 gives approximately 0.521928 bits. A badly reversed model with q equal to 0.1 costs approximately 3.004936 bits in this ideal account. These are expected information quantities, not literal lengths of a finite arithmetic-coded file. They show why a visually good transform can still compress poorly if its symbol probabilities are modeled inaccurately. Conversely, a low training estimate is uninformative if the decoder uses different probabilities.

4. Finite files include overhead and rounding

An entropy calculation can produce a fractional number of bits per symbol, while a stored file has an integer length and may include headers, termination information, padding and integrity metadata. The gap is especially visible for small images or blocks. A report should distinguish payload bits, side-information bits and fixed per-file overhead instead of silently charging them to different experiments.

The binary example gives a simple consistency test. A claimed uniform binary payload cannot have an expected ideal cost below one bit per independent symbol without additional exploitable structure or a changed source model. A measured file can be larger because of overhead. It can also differ from the ideal formula because the probability table is quantized or the sequence is finite. None of these effects invalidates entropy modeling; they define the connection between the model and a real coding protocol. Rate estimation should be checked against actual files over a range of sizes, including the small cases in which fixed costs are not negligible.

5. Side information can help and still cost too much

A scale hyperprior transmits information used to predict properties of the main latent representation. The primary source explicitly relates this mechanism to side information. It can improve conditional probability estimates, but its bits are not free. The total rate must include both the main symbols and the auxiliary information needed to decode them.

In a constructed binary budget, compare a uniform model costing one ideal bit per symbol with a perfectly matched p=0.9 model that requires sixteen extra side-information bits for the block. The second account costs approximately 0.468996 times the block length plus sixteen. It becomes smaller than the uniform account only above about 30.13 symbols, so thirty-one is the first integer length beyond the ideal crossover. This is not a measurement of a hyperprior codec. It illustrates the condition under which better conditional modeling pays for its metadata. Real systems must use their actual side-information representation, probability precision and finite-bitstream costs.

Sources: [2]

6. Conditional context has a decoding schedule

Minnen, Ballé and Toderici combine autoregressive and hierarchical priors to exploit latent dependencies. The source highlights complementary information sources and the computational cost of context modeling. Better conditional probabilities can reduce the bitstream while introducing a dependency schedule that limits parallel decoding.

This creates a systems tradeoff that a rate-distortion curve alone cannot resolve. If the probability of a symbol depends on previously decoded neighbors, the decoder must obtain those neighbors in the required order or use an equivalent supported computation. A highly parallel transform does not automatically make entropy decoding equally parallel. Evaluation should measure encoding and decoding separately, including probability-model execution and transfer between devices. A codec suitable for offline archives may accept a dependency chain that is unattractive for interactive use. Neither choice is intrinsically superior; the application contract determines which resource matters. The report should preserve that contract rather than calling every smaller file an unqualified system improvement.

Sources: [3]

7. Discrete likelihoods must match the coder's symbols

Cheng and colleagues study discretized Gaussian mixture likelihoods and attention modules for learned compression. The distinction between a continuous density and a probability mass assigned to a quantized symbol is central. A density value at a point is not automatically the probability that a rounded latent lands in a quantization bin.

A practical entropy model needs a rule for converting its parameters into symbol probabilities, including tails, finite precision and any minimum-probability handling. The encoder and decoder must apply compatible rules. A numerical discrepancy in a probability table can be more serious than a small estimation error: it can desynchronize a coding process and prevent correct decoding. Testing should therefore include extreme latent values, small predicted scales and symbols near support boundaries. The goal is not to infer that a flexible likelihood family solves every calibration problem. It is to verify that the learned probability model, its discretization and the actual entropy coder describe the same executable distribution.

Sources: [4]

8. Perceptual quality and source evidence are different

High-Fidelity Generative Image Compression studies adversarial and perceptual objectives alongside quantitative and human evaluation. This is a useful reminder that reconstruction quality has more than one interpretation. A reconstruction can look convincing while changing a source-specific texture, character or small object. A squared-error metric can penalize a harmless shift while missing a more important semantic change.

The appropriate objective depends on use. A creative preview may prioritize plausible appearance. A scientific figure, archival document or image used as evidence may require preservation of specific details. An evaluation should identify protected content and test it directly, rather than assuming that a generic perceptual preference guarantees fidelity. The finite binary model does not answer which visual changes are acceptable. It supplies only a transparent coding-cost argument. Real perceptual assessment requires suitable images, sampling procedures and reported uncertainty. A strong codec report can present several quality measures without pretending that they collapse into one universal notion of correctness.

Sources: [5]

9. Variable rate is an interface to validate

The recurrent compression work of Toderici and colleagues provides multiple deployment rates from one trained architecture. This offers a different interface from maintaining a separate checkpoint for every operating point. The source supports that architectural possibility, but every operating point still needs a complete bitstream and reconstruction evaluation.

A progressive representation should specify which prefixes are decodable, how missing refinement stages are interpreted and whether truncation at arbitrary byte positions is supported. These are format and implementation properties, not automatic consequences of recurrent computation. A test suite should decode supported prefixes, reject malformed truncations clearly and confirm that the displayed rate includes all prefix metadata. It should also examine whether quality improves monotonically under the metric actually promised to users; an extra refinement stage need not improve every conceivable perceptual criterion. Reporting only selected favorable rates can conceal unstable regions of the interface. The rate-control contract should make those limits visible.

Sources: [6]

10. Distribution shift can change the rate before the picture

A probability model trained on one image population may assign poor probabilities to latents from another. The transform can still reconstruct the new images acceptably while the bitstream expands because the entropy model is mismatched. Conversely, an image can produce a compact latent sequence while losing an important unfamiliar detail. Rate and distortion under shift should therefore be measured separately.

Useful stress cases include different resolutions, noise characteristics, synthetic diagrams, text-heavy images and unusual color statistics, chosen according to the intended application. A benchmark should preserve all attempted inputs and report failures to encode or decode. If a codec adapts its probability model to a new image, the adaptation cost and any transmitted update belong in the total account. An oracle model that is fitted using information unavailable to the decoder cannot be used as a deployment-rate estimate. The binary q-versus-p example isolates this mismatch mathematically, while actual image tests are needed to determine how large it is for a trained system.

11. Calibration is about probabilities, not labels

In compression, a useful probability estimate is one that gives an efficient and synchronized coding distribution for the symbols that actually occur. A classifier-style statement that the most likely symbol is usually correct is insufficient. Assigning probability 0.51 or 0.99 to the same observed symbol gives the same top prediction but a different code-cost contribution, especially when the prediction is wrong.

A diagnostic should inspect negative log probabilities, tail behavior and the agreement between estimated and realized rate. Aggregate calibration bins can help but may hide conditional dependencies. A model can have plausible marginal frequencies while missing structure that a context model could exploit. Conversely, a complex model may improve average rate by a small amount while increasing latency and failure sensitivity. The proposed evaluation treats entropy-model quality as one component of a full codec contract. It should be possible to identify whether a regression comes from changed latents, changed probabilities, changed coding precision or changed metadata, rather than merely observing that the final file became larger.

12. Rate-distortion optimization has a chosen scale

A common objective combines an expected rate term with a weighted distortion term. The tradeoff parameter is meaningful only relative to the units and normalization of those terms. Changing image resolution, pixel scaling, color treatment or the averaging convention can change its interpretation. Two checkpoints trained with the same displayed coefficient need not occupy comparable operating points if their objective definitions differ.

An experiment should state whether rate is measured per pixel, per channel or per image, and how distortion is normalized. It should report actual achieved operating points rather than relying on the training coefficient as a proxy. Comparing interpolated rate-distortion curves also requires common ranges and careful treatment of crossing curves. A method can be better in one region and worse in another. The binary construction uses bits per source symbol and an explicitly separate side-information budget, so its units are unambiguous. That clarity should be preserved when moving to images, where several normalizations can look similar while answering different questions.

13. A learned codec needs a compatibility record

The decoder must know the transform and probability-model versions required by the bitstream. A change in network weights or numerical probability conversion can alter reconstruction or make the file undecodable. Successful loading of tensors with matching shapes does not establish compatibility with an older encoded image.

A durable artifact should bind the format version, model identity, quantization rule, probability precision and required side information. Test vectors should include encoded bytes and expected decoded outputs under a declared numerical tolerance or exactness criterion. A maintenance update should run both new-file and old-file decoding checks. If compatibility is intentionally broken, the format should identify that change clearly rather than silently attempting a new model. These requirements are not unique to learned compression, but neural components can make the dependency boundary less obvious. A compact checkpoint is part of a codec's machinery, and its storage or distribution cost should be accounted for according to the deployment scenario.

14. Efficiency is a multi-resource comparison

A smaller bitstream can reduce storage or transmission cost while requiring more encoding compute, decoding latency or model memory. A perceptually stronger reconstruction can increase distortion under a literal pixel metric. A more accurate entropy model can reduce payload but add side information and sequential work. These are tradeoffs to expose rather than reasons to force every system into a single ranking.

We propose reporting actual bytes, several application-relevant quality measures, encoding and decoding latency, peak memory and failure counts. The benchmark should identify hardware, threading, precision, batch behavior and whether model loading is included. Warm-cache and cold-start cases may serve different users. An offline conversion service and an embedded viewer need different summaries. The finite side-information example captures only one accounting tradeoff, so it cannot predict which codec is best on real hardware. Its role is to ensure that a mathematical improvement in one term is not promoted into an unmeasured claim about the full system.

15. The calculations have narrow executable checks

The first check evaluates binary entropy and cross-entropy for p=0.9 across several coding probabilities, then verifies that the difference equals the finite Kullback-Leibler divergence. It should reject invalid probabilities and confirm equality at q=p. The second check adds the fixed sixteen-bit side-information cost and finds the integer crossover by direct enumeration, not just by printing the derived threshold.

The resulting figures show expected ideal information budgets, not integer arithmetic-coder outputs. Negative tests should omit side information, use natural logarithms without converting units or confuse probability density with symbol mass. These mistakes should be detected by explicit unit and normalization checks. A separate real-codec study would need to produce and decode actual files, but no such study is claimed here. The constructions validate the article's arithmetic and clarify what an entropy-model estimate can support before an empirical rate-distortion result is available.

16. Compression claims should terminate in a decoder

The selected sources show how learned transforms, quantization, side information, conditional priors and perceptual objectives can be combined. Their common engineering endpoint is a decoder receiving a complete and compatible representation. A claim about compression is strongest when another implementation can count the bytes, decode the file and inspect the reconstruction under the stated quality contract.

Our binary examples separate source uncertainty from model mismatch and expose a side-information crossover. They do not rank image codecs or estimate natural-image entropy. Future work should make complete bitstream accounting, model-version compatibility and protected-detail tests routine alongside rate-distortion curves. This would clarify which gains come from a better representation, which come from probability modeling and which come from accepting a different reconstruction objective. A learned codec then becomes easier to maintain and compare because its claims are tied to the bits it actually sends and the information its decoder is allowed to change.

Core equations and definitions

(1) Binary-source entropy in ideal expected bits per symbol.

(2) Expected ideal cost under a mismatched binary coding model; not a finite-file length.

(3) Accounting identity for every component required by the chosen bitstream and decoder.

(4) Constructed ideal crossover for a matched binary model with sixteen side-information bits versus a uniform model.

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Freeze image preprocessing, transforms, quantization and decoder versions.
• Distinguish relaxed rate estimates, rounded-symbol estimates and actual bytes.
• Count main, hyperprior and format information together.
• Verify probability normalization and encoder/decoder table agreement.
• Produce and decode reference bitstreams for a real codec benchmark.
• Report protected-detail fidelity separately from perceptual plausibility.
• Measure rate and distortion under deployment-relevant shifts.
• Record encoding/decoding latency and memory independently.
• Test supported progressive prefixes and malformed input handling.
• Retain compatibility vectors across model updates.

Limitations

• Binary examples are ideal expected-information calculations, not actual image-codec rates.
• Sixteen side-information bits are a stipulated toy cost.
• No cited rate-distortion curve, human preference study or hardware result is reproduced.
• Perceptual quality does not establish exact source-detail preservation.
• The article does not implement or audit an entropy coder.
• The primary-source selection is focused rather than exhaustive.

Open research questions

• Which rate-estimation discrepancies best predict decoder incompatibility before deployment?
• How should learned codecs expose protected-detail and perceptual objectives as separate operating contracts?
• Can per-image adaptation amortize its metadata and compute costs reliably across small and shifted inputs?

Evidence and its boundaries

A nonlinear transform, quantizer and synthesis transform can be optimized jointly using a continuous training proxy.

[1] — A relaxed training objective is not identical to every deployed quantized bitstream. Inspected: Primary abstract on nonlinear transforms, quantization and joint rate-distortion optimization.

A scale hyperprior supplies side information used to model dependencies in quantized latents.

[2] — The side information has a coding cost and must be included in total rate. Inspected: Primary abstract on side information and hierarchical latent probability modeling.

Conditional context and hierarchical side information can exploit different latent dependencies.

[3] — Compression gains must be evaluated with decoding dependency and computation costs. Inspected: Primary abstract on arithmetic-coded latents and combined autoregressive/hierarchical probability models.

A more flexible entropy model can change rate estimates and rate-distortion optimization.

[4] — A continuous density must be related to discrete symbol probabilities; reported codec comparisons are not reproduced. Inspected: Primary abstract on discretized mixture likelihoods, rate estimation and attention.

Generative compression studies perceptual reconstruction objectives in addition to conventional distortion.

[5] — Perceptual preference does not establish exact preservation of every source detail. Inspected: Primary abstract on adversarial/perceptual compression and human evaluation.

A recurrent learned codec can provide multiple rates from one trained architecture.

[6] — Rate flexibility still requires measuring the complete coded representation and reconstruction behavior. Inspected: Primary abstract on recurrent encoding, binarization and variable deployment rates.

References

1. End-to-end Optimized Image Compression
   Johannes Ballé, Valero Laparra, Eero P. Simoncelli. ICLR 2017 (2017). https://arxiv.org/abs/1611.01704v3
   Source scope

   Primary abstract on nonlinear transforms, quantization and joint rate-distortion optimization.

2. Variational Image Compression with a Scale Hyperprior
   Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, Nick Johnston. ICLR 2018 (2018). https://arxiv.org/abs/1802.01436v2
   Source scope

   Primary abstract on side information and hierarchical latent probability modeling.

3. Joint Autoregressive and Hierarchical Priors for Learned Image Compression
   David Minnen, Johannes Ballé, George Toderici. NeurIPS 2018 (2018). https://arxiv.org/abs/1809.02736
   Source scope

   Primary abstract on arithmetic-coded latents and combined autoregressive/hierarchical probability models.

4. Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules
   Zhengxue Cheng, Heming Sun, Masaru Takeuchi, Jiro Katto. CVPR 2020 (2020). https://arxiv.org/abs/2001.01568v3
   Source scope

   Primary abstract on discretized mixture likelihoods, rate estimation and attention.

5. High-Fidelity Generative Image Compression
   Fabian Mentzer, George Toderici, Michael Tschannen, Eirikur Agustsson. NeurIPS 2020 (2020). https://arxiv.org/abs/2006.09965v3
   Source scope

   Primary abstract on adversarial/perceptual compression and human evaluation.

6. Full Resolution Image Compression with Recurrent Neural Networks
   George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, Michele Covell. CVPR 2017 (2017). https://arxiv.org/abs/1608.05148v2
   Source scope

   Primary abstract on recurrent encoding, binarization and variable deployment rates.