Knowledge note
Post-Training Quantization
Model quantization applied to an already-trained network using a small calibration set or weight statistics instead of full retraining.
Post-Training Quantization
Post-Training Quantization is model quantization performed after training, using pretrained weights and usually a small calibration set instead of retraining the model end to end.
Why It Exists
Training or fine-tuning billion-parameter models with simulated low-precision arithmetic can be prohibitively expensive. PTQ offers a shorter deployment path: start with an existing checkpoint and transform it into a smaller or faster representation.
How It Works
A PTQ method chooses a numeric grid, scale granularity, and which tensors to quantize. Simple methods derive ranges from weights or calibration activations and round to the nearest grid point. More accurate methods correct rounding errors, learn clipping or scaling parameters, rotate or transform distributions, or keep sensitive values at higher precision.
PTQ may be weight-only, activation-only, or weight-and-activation quantization. Calibration is normally much smaller than training, but it should expose representative tensor distributions.
Tradeoffs
PTQ is inexpensive relative to retraining, yet it has less freedom to recover from quantization error. Very low bit widths, activation outliers, calibration-domain mismatch, and missing optimized kernels can erase its accuracy or latency benefits.
Common Confusions
- PTQ vs. quantization-aware training: PTQ starts from a finished model and avoids full retraining; QAT exposes model parameters to simulated quantization during optimization.
- Weight-only vs. W4A4: Weight-only PTQ leaves activations in higher precision; W4A4 quantizes both weights and activations to 4 bits.
- Compression vs. speed: Fewer bits always reduce storage, but latency improves only when hardware and kernels exploit the representation for the target workload.
Where It Appears
- GPTQ — Uses activation-derived second-order correction to make 3–4-bit weight-only PTQ practical at 175B scale.
- FlatQuant — Learns affine transformations, scaling, and clipping for accurate weight-and-activation PTQ.
- QuaRot — Rotates activations to remove outliers so plain 4-bit weight-and-activation PTQ works end to end.
- HiFloat4 (HiF4) — Compares direct-cast HiF4 with NVFP4 and adapts GPTQ as HiGPTQ for further accuracy recovery.