KronQ: LLM Quantization via Kronecker-Factored Hessian

The Problem: Existing Quantization Methods Ignore Output-Side Information

Compressing large language models by reducing their weights to lower precision (quantization) is essential for deployment, but it degrades performance. Current second-order post-training quantization methods like GPTQ treat all output channels as equally important when deciding which weights to keep at higher precision. This assumption is wrong. The authors show that you also need to consider how sensitive each output dimension is to quantization errors—information captured by gradient covariance, which existing methods discard entirely.

How KronQ Works

KronQ builds on the Kronecker-factored Hessian, a mathematical approximation that decomposes the curvature of the loss function into a product of two matrices: one depending on input activations and one on gradients (output information). Crucially, this means the reconstruction loss depends on both sides of the weight matrix, not just inputs.

The method operates at two levels:

Bidirectional incoherence processing. Traditional PTQ rotates weights randomly on the input side to reduce their variance before quantization, making them easier to compress. KronQ extends this to the output dimension using gradient covariance information. By applying complementary rotations on both sides, weights become more uniform in magnitude across both dimensions, yielding better quantization precision.

Mixed-precision allocation. Not all layers compress equally—some are more sensitive to errors. KronQ derives a sensitivity metric from the Hessian traces (a measure of curvature) to decide which layers or channels need to stay at higher precision. Channels with high gradients and high input variance are marked as sensitive and preserved.

Why This Matters

The empirical results are striking. On LLaMA-3-70B at 2-bit weight-only quantization—an extremely aggressive compression setting—GPTQ and a recent variant (GPTAQ) completely break down, producing perplexity above 2000 on WikiText-2 (a standard benchmark where lower is better). KronQ achieves 7.93 perplexity, a functional model. This isn't a marginal improvement; it's the difference between a usable and unusable system.

The insight is simple but overlooked: output sensitivity matters as much as input statistics. By incorporating gradient information already computed during calibration, KronQ extracts more signal from the same data without requiring retraining. This makes extreme quantization practical for very large models where memory and latency are critical constraints.

The method is grounded in solid linear algebra—the Kronecker factorization is a well-understood approximation—yet yields outsized practical gains, particularly in the regime where existing methods fail entirely.