What the Loss Curve Misses: FP4 Pretraining Slows Rare Facts
A study investigating low-precision pretraining finds that NVFP4 requires significantly more exposures than BF16 to learn rare facts effectively.
A study investigating low-precision pretraining finds that NVFP4 requires significantly more exposures than BF16 to learn rare facts effectively.