deploybase

FP16 vs FP32 vs INT8: GPU Precision Formats for AI

Deploybase · January 7, 2025 · GPU Comparison

Contents

Fp16 vs Fp32 vs Int8 AI: Precision Fundamentals

Fp16 vs Fp32 vs Int8 AI is the focus of this guide. When developers pick a precision format, developers're trading off memory, speed, and accuracy. FP32 (32-bit float) is the default-it stores 7 significant digits. FP16 (16-bit float) cuts that to 4 digits but runs 2-4x faster. INT8 (8-bit integer) runs fastest but only handles integers from -128 to 127.

This matters. A 70B parameter model takes 280GB at FP32, 140GB at FP16, and 70GB at INT8. The smaller the number, the cheaper the GPU developers need.

FP32 Standard Precision

FP32 is the baseline. Every framework supports it. Every GPU accelerates it. Developers get full precision-sign, exponent, mantissa-with 7 significant digits per number.

Trade-off: it's hungry. 70B models need 280GB of memory. That's way more than any single GPU. Developers need distributed training (tensor parallelism splits matrices across multiple GPUs) or a cluster. A100 80GB GPUs work but aren't ideal alone.

FP16 Half Precision

FP16 cuts memory in half. A 70B model now fits in 140GB instead of 280GB. Speed bumps up 2-4x because the hardware (tensor cores) is optimized for it. A100 GPUs especially love FP16.

Catch: gradients can underflow at FP16 during training. Loss scaling fixes this-developers scale gradients up during training, scale them back after. Works fine. Most frameworks (PyTorch, TensorFlow) handle it automatically. Mixed precision-keeping weights at FP32 but doing forward/backward in FP16-is the standard practice for training.

INT8 Integer Quantization

INT8 is integers only. A 70B model fits in 70GB-25% of FP32. Speed jumps 8-16x on hardware that likes integer math. But INT8 training doesn't work. Quantization errors compound through backprop.

Use it for inference. Post-training quantization is easy: train at FP32, convert to INT8, done. Developers lose 1-3% accuracy depending on how well developers calibrate. Quantization-aware training (simulating quantization during training) recovers some of that, but adds complexity. For most inference workloads, post-training quantization is fine.

Mixed Precision Training

Keep weights at FP32, run forward/backward in FP16. Developers get FP16 speed with FP32 stability. Loss scaling handles underflow: scale gradients up, train, scale back down. It's automatic in PyTorch (AMP) and TensorFlow. Basically free performance.

Memory Impact Analysis

Math is simple:

  • FP32: 70B × 4 bytes = 280GB
  • FP16: 70B × 2 bytes = 140GB
  • INT8: 70B × 1 byte = 70GB

An A100 80GB barely fits FP16 (developers still need room for activations and optimizer states). INT8 is the only option for single-GPU inference on H100 (80GB). RTX 4090 (24GB) requires INT4 quantization for 7B-13B models but cannot fit 70B models even at INT8.

Inference Performance Gains

FP16 runs 2-4x faster. INT8 runs 8-16x faster. Both are bandwidth-limited, not compute-limited-moving data on and off the GPU is the bottleneck. Smaller precision = less data moved = faster overall.

Training Accuracy Comparison

FP32 is the baseline. FP16 with loss scaling matches FP32 accuracy. INT8 training doesn't work-errors compound during backprop. Use INT8 only for inference via post-training quantization.

Quantization Methods

Post-training: Train at FP32, convert to INT8. Use representative calibration data to find the right quantization ranges. No retraining. Simple.

Quantization-aware training: Simulate quantization during training. Models learn to cope with lower precision. More work, better accuracy.

Dynamic quantization: Compute min/max values on the fly. No calibration needed. Accuracy varies depending on data distribution.

Hardware Acceleration Support

A100s love FP16-tensor cores give developers 5x speedup. H100s add FP8 (even faster, more calibration needed). Consumer RTX GPUs support FP16 but with less fanfare (2-3x improvement). Fewer tensor cores.

Cost Impact on Inference

Precision changes how many inferences developers can squeeze from one GPU. FP16 = 2-4x more throughput. INT8 = 8-16x more throughput. Cost per inference drops the same way. Lower precision = cheaper per-inference cost (same GPU, more work).

When to Use Each

FP32: Developers need the accuracy or stability. Training is the first choice.

FP16: Training large models or inference with reasonable accuracy needs. Standard pick.

INT8: Inference only. Tight budget. Single-GPU deployment.

Real-World Examples

Fine-tune Llama 70B: FP16 needs 140GB. Two A100 GPUs fit it. FP32 needs 280GB (developers're buying twice as much hardware).

Run inference locally: RTX 4090 (24GB) handles 7B-13B models in INT8 or INT4. For 70B models, an A100 80GB with INT8 (70GB) is required. One-time hardware cost can beat cloud fees at sufficient inference volume.

API inference: OpenAI GPT-4o Mini uses quantization internally. Developers pay less per token because their infrastructure costs less.

FAQ

What precision should I use for training? FP32 for baseline. Mixed precision (FP32 + FP16) for speed. FP16 solely if memory-constrained with careful implementation.

Is FP16 accuracy acceptable? Yes, with proper loss scaling. Mixed precision matches FP32 accuracy. Pure FP16 requires careful tuning.

Can I use INT8 for training? Not directly. Quantization-aware training works but adds complexity. Post-training quantization simplifies deployment.

How much faster is INT8 versus FP32? Depends on hardware. Theoretical 8x speedup. Actual speedup varies 4-12x based on bandwidth.

Which precision for Llama models? FP16 for training on multi-GPU. INT8 for inference on consumer hardware. Mixed precision for speed/accuracy balance.

GPU Pricing Guide - Hardware cost comparison. A100 GPU Pricing - Premium precision support. H100 GPU Pricing - Latest precision options. Model Distillation - Alternative compression approach. Tensor Parallelism - Distributed training method.

Sources

NVIDIA precision documentation Framework mixed precision guides Quantization research papers Industry deployment standards (March 2026)