Knowledge note

GPTQ

A second-order post-training weight-quantization algorithm that compensates rounding errors using calibration activations and scales the procedure through shared column ordering and block updates.

Updated September 16, 2026Confidence: High

GPTQ

GPTQ is a second-order post-training weight-quantization algorithm that rounds weights to a low-bit grid while adjusting the remaining weights to preserve each layer’s outputs on calibration data.

Why It Exists

Round-to-nearest scales to huge models but can destroy accuracy below 8 bits, while earlier second-order methods repeat expensive greedy bookkeeping for each weight row. GPTQ keeps error compensation while reorganizing it into shared, GPU-friendly work.

How It Works

For each linear layer, GPTQ records representative input activations and uses H = 2XXᵀ as the Hessian of the layer-output reconstruction loss. After quantizing a column, inverse-Hessian information determines how to adjust the columns that remain in full precision.

All output rows follow one column order, 128-column blocks accumulate corrections before a global matrix update, and a damped Cholesky formulation supplies stable correction rows. The result is weight-only quantization that the paper applies to 175B-parameter models in a few hours.

Tradeoffs

GPTQ needs calibration data and an offline quantization pass. Its original form does not quantize activations or reduce the multiplication count; deployment speed depends on packed-weight kernels and on inference being limited by weight memory traffic.

Common Confusions

  • GPTQ vs. round-to-nearest: Both use a fixed low-bit grid, but GPTQ compensates rounding error in the still-unquantized weights.
  • GPTQ vs. quantization-aware training: GPTQ changes a pretrained model without retraining; QAT simulates quantization during training or fine-tuning.
  • GPTQ vs. a numeric format: GPTQ is a calibration and rounding algorithm, not a bit encoding such as INT4 or FP4.

Where It Appears

  • GPTQ: Second-Order Weight Quantization at LLM Scale — Source-defining explanation of the algorithm, deployment kernel, and OPT/BLOOM results.
  • FlatQuant — Uses GPTQ as a weight-rounding solver and comparison baseline, while showing that learned flattening can make RTN competitive.
  • QuaRot — Uses GPTQ as its default 4-bit weight quantizer after rotating away activation outliers.
  • HiFloat4 (HiF4) — Adapts GPTQ-style rounding to the HiF4 hierarchy as HiGPTQ.
  • Post-Training Quantization — The broader one-shot model-compression family containing GPTQ.
  • GEMM — The matrix multiplication whose memory and arithmetic behavior determines whether compressed weights improve runtime.