---
title: "If you're still training in FP16, you're leaving half your GPU on the table."
url: https://stacklist.com/card/faffd203-636d-465e-8ac4-f0c65b38ebcf
source_url: "https://www.linkedin.com/posts/paoloperrone_if-youre-still-training-in-fp16-youre-share-7480370913650176000-CIDz/?utm_source=share&utm_medium=member_ios&rcm=ACoAAAI21ZsBNnZPaKuTab7nquKLCveUW7o-1DE"
stack: https://stacklist.com/stack/4aae218c-38d7-4c05-b7ae-f2db0029a2e8
summary: "FP8 floating-point training offers 2x memory savings and throughput improvements over FP16 by using two specialized formats (E4M3 for weights and E5M2 for gradients) that NVIDIA's Transformer Engine switches between automatically. The post discusses the evolution from FP32 to FP8 and hints at emerging NVFP4 technology that further optimizes precision reduction while preserving network accuracy."
tags: "fp8-training, gpu-optimization, mixed-precision, transformer-engine, neural-networks, model-compression, deep-learning"
key_entities: "Paolo Perrone (person), Michał Piszczek (person), Curtis Burkhalter (person), NVIDIA (organization), Transformer Engine (technology), H100 Tensor Cores (technology), Hopper (technology), Blackwell (technology), NVFP4 (technology), FP8 training (concept), mixed-precision training (concept), E4M3 format (concept), E5M2 format (concept)"
classification: "analysis"
content_hash: "sha256:2e2f9af8375a6ea480351eab34379348fe67b1a2751145e29401d3e6c401908b"
acp_version: "0.2"
token_counts_approximate: 579
visibility: public
agent_accessible: true
status: "final"
---

# If you're still training in FP16, you're leaving half your GPU on the table.

Paolo Perrone 2h Report this post If you're still training in FP16, you're leaving half your GPU on the table. 8-bit floating point. Half the bits of FP16. A quarter of FP32. Two formats: E4M3 (4 exponent bits, 3 mantissa bits) Higher precision, smaller range. Best for: forward pass, inference, weights. E5M2 (5 exponent bits, 2 mantissa bits) Lower precision, larger range. Best for: backward pass, gradients. Why two? Gradients spike during training. They need room to grow. E5M2 handles that. Weights and activations stay stable. They need accuracy. E4M3 handles that. Switch formats mid-pass. That's the trick. What you get: → 2x memory savings vs FP16 → 2x throughput on H100 Tensor Cores → Near-identical accuracy when done right → Models that needed 80GB now fit in 40GB NVIDIA's Transformer Engine handles the format switching automatically. Mixed FP8 training is production-ready on Hopper. FP32 → FP16 → BF16 → FP8. Each step doubled throughput. This is the next one. 💾 Bookmark this before your next training run. 7 2 Comments Like Comment Share Copy LinkedIn Facebook X Michał Piszczek 26m Report this comment FP8 is already old news. The more interesting question is what happens when training moves to NVFP4. Unlike classic INT4 or FP4, NVFP4 is not “just 4 bits”. NVIDIA pairs 4-bit floating-point values with block-level scaling, stochastic rounding and Transformer Engine, so tensors are constantly rescaled to preserve dynamic range where it actually matters. The result is counterintuitive: • ~2× lower memory than FP8 • ~2× higher math density • accuracy surprisingly close to FP8/BF16 on many transformer workloads • much larger models fit on the same hardware That’s why Blackwell wasn’t designed around FP8… it was designed around NVFP4. FP32 → FP16 → BF16 → FP8 was about reducing precision. NVFP4 is different. It’s about reducing precision only where the network can afford it while preserving information statistically. If this trend holds, FP8 may become the new “legacy precision” much sooner than most people expect Like Reply 1 Reaction 2 Reactions Curtis Burkhalter, Ph.D. 1h Report this comment Great explanation of the different formats and why you would use the different formats Paolo ! Like Reply 1 Reaction See more comments To view or add a comment, sign in