The Error Ledger of Low-Bit Inference: Quantization Scales, Calibration, and Actual Memory Savings

1. A quantization format involves more than bit width

Uniform affine quantization maps real numbers to a finite integer range, then reconstructs them using a scale and a zero point. An actual format must also specify the signed range, grouping direction, group size, scale precision, rounding rules, and clipping method. Finer grouping generally adapts better to local value ranges, while requiring more metadata. The same 4-bit label can therefore correspond to different effective byte counts and errors. Weight packing layouts and hardware kernels also affect whether computation can proceed directly; a small compressed file does not prove that runtime dequantization buffers are unnecessary.[1] GPTQ: Accurate Post-Training Quantization for Generativ[2] AWQ: Activation-aware Weight Quantization for LLM Compr

2. Weight error is amplified through input covariance

In a linear layer, a weight perturbation ΔW produces an output error ΔWx, whose expected squared magnitude depends on the second-order statistics of the inputs. Weight errors in certain directions may be small yet produce substantial output changes because inputs have large magnitudes in those directions. GPTQ uses information associated with calibration inputs to approximately address the layer reconstruction problem; AWQ emphasizes activation-aware weight processing. Neither proves that errors are small for all inputs; instead, they make approximations within a given calibration context. Evaluation must include lengths, tasks, and languages outside the calibration set to check for statistical mismatch.[1] GPTQ: Accurate Post-Training Quantization for Generativ[2] AWQ: Activation-aware Weight Quantization for LLM Compr

3. Activation outliers change the difficulty of quantization

A few large activation values may force the quantization range to widen, reducing resolution for most ordinary values. SmoothQuant transfers quantization difficulty between weights and activations through equivalent rescaling of adjacent linear operations; the equivalent transformation itself does not eliminate requantization error. Scaling parameters and calibration procedures should be recorded, and long inputs and atypical distributions should be examined. Low-bit storage of weights alone and low-bit computation of activations as well are different approaches; quality conclusions for the former cannot establish those for the latter.[3] SmoothQuant: Accurate and Efficient Post-Training Quant

4. Memory savings require complete, itemized accounting

Raw weight storage can be calculated exactly from the parameter count and bit width, but runtime memory also includes scales, zero points, unquantized layers, workspace, activations, and the KV cache. Long contexts may make KV dominant, so total device memory need not continue to decrease proportionally when weight bit width is reduced. The 7B parameters in the plot are merely a counting example of 7×10^9 scalars; GB and GiB are reported separately to avoid unit confusion. Any conclusion about peak memory still requires measurement on actual devices and under actual batch conditions.[1] GPTQ: Accurate Post-Training Quantization for Generativ[2] AWQ: Activation-aware Weight Quantization for LLM Compr[3] SmoothQuant: Accurate and Efficient Post-Training Quant

5. Calibration and validation must be strictly separated

Calibration samples are used to select scales or approximate input statistics, whereas validation samples are used to estimate quality on unseen distributions. Continually adjusting group sizes and outlier thresholds based on the test set already constitutes using that set for development. We recommend freezing an independent calibration set, specifying task and length strata in advance, and saving failure examples and floating-point baseline outputs. For safety or factuality tasks, rare errors should also be examined rather than comparing only a single mean perplexity. The final report after screening multiple quantization configurations should state the search budget.[1] GPTQ: Accurate Post-Training Quantization for Generativ[2] AWQ: Activation-aware Weight Quantization for LLM Compr

6. Speed gains depend on executable kernels

Low-bit storage may ease bandwidth pressure, but it may also increase the costs of unpacking, dequantization, and format conversion. Small-batch decoding, large-batch prefill, and different hardware have different bottlenecks, so speed rankings may change. Experiments should separate loading, prefill, token-by-token decoding, and end-to-end latency, and record batch size, length, precision, and kernel version. The byte accounting in this article establishes only a lower bound on storage, while the uniform-noise approximation explains only resolution; neither can replace device measurements or task-quality checks.[1] GPTQ: Accurate Post-Training Quantization for Generativ[3] SmoothQuant: Accurate and Efficient Post-Training Quant

Core equations and derivations

Method and evidence workflow

Analytical plots and numerical examples

Recomputable worked example

7×10^9 weights ideally packed at 4 bits occupy 3.5 GB, approximately 3.260 GiB. Group scales, unquantized layers, KV, and workspace are not yet included, so this cannot be called the complete device-memory requirement for deployment.

Method comparison

Verification and experimental plan

1. Freeze the floating-point baseline, tokenizer, and independent calibration/validation split; record sources and hashes.
2. Fully record weight, activation, and KV precisions and grouping parameters, and list all layers retained in floating point.
3. Account for actual file bytes, scale metadata, dequantization workspace, and peak device memory.
4. Measure layer output error, perplexity, task success, and rare failures separately; do not substitute one metric for all of them.
5. Test calibration transfer across different lengths, languages, and input distributions, and retain the worst-performing strata.
6. Independently repeat prefill/decoding timing on fixed kernels and hardware, and report gains subject to quality constraints.

Scope and limitations

• There are no measurements of actual model quantization or device speedups.
• The uniform-error approximation does not apply to arbitrary weight distributions.
• Raw weight bytes exclude KV and workspace.
• Quality outside the calibration set requires separate validation.

Primary sources and corresponding claims

1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, , , · 2022-10-31Abstract; 原文方法、理论和实验章节见作者原始论文链接
2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, , , , , , , , , · 2023-06-01Abstract; 原文方法、理论和实验章节见作者原始论文链接
3. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, , , , , · 2022-11-18Abstract; 原文方法、理论和实验章节见作者原始论文链接