ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization

The Core Problem and Solution

Quantizing LLM weights to 3 or 4 bits requires rounding continuous values to discrete integers. Standard round-to-nearest (RTN) has a hidden weakness: weights landing near the midpoint between two quantization levels create ambiguity about which direction to round. ReRound resolves this by using a diffusion model to "suggest" better rounding directions for these ambiguous weights, then selects the overall quantization that best preserves the weight matrix's spectral properties.

How ReRound Works

The method operates in three phases. First, the authors train a conditional diffusion model—a generative model that learns to produce continuous reconstructions of quantized weights—on the pretrained LLM. This diffusion model serves as an oracle for what good low-bit weight values should look like.

Second, ReRound introduces a tolerance metric to identify "midpoint ambiguous" weights. Instead of rounding all weights the same way, it splits them into two groups: weights close to quantization interval midpoints (within a tolerance band) are rounded using the diffusion model's reconstruction as guidance, while weights near interval boundaries use standard RTN. By varying the tolerance parameter, the method generates multiple candidate quantized weight matrices.

Third, it selects among candidates using a spectral criterion: the chosen quantization is the one whose de-quantized (reconstructed continuous) weight matrix has leading singular values closest to the original full-precision weights. This spectral matching acts as a validation mechanism, effectively asking "which quantization preserves the matrix structure best?" The tolerance parameter that produced the winning candidate becomes ReRound's final setting.

Critically, this entire process is offline—no calibration data or additional compute during inference.

Why This Matters

Low-bit quantization (3–4 bits) is essential for deploying large models on resource-constrained devices, but it introduces severe information loss. Most existing methods either require calibration data (reducing practical usability) or apply uniform rounding rules that ignore local geometry around midpoints.

ReRound's insight—that midpoint ambiguity is a real problem and that diffusion-guided reconstruction can resolve it—is novel. By combining learned reconstruction with spectral validation, the method avoids both calibration overhead and the brittleness of purely rule-based approaches.

The empirical results underscore practical value: ReRound consistently outperforms calibration-free baselines (standard RTN, naive quantization schemes) on smaller LLMs at 3-bit and 4-bit precision and remains competitive with calibration-dependent methods like learned quantization, despite requiring no calibration data. This makes it a strong candidate for deployment scenarios where calibration is infeasible.

The focus on smaller LLMs (likely 1B–13B parameters) is strategic; these models are common in edge and on-device settings where quantization matters most, and midpoint ambiguity effects are more pronounced at lower precisions.