Research and ACM SRC Posters Archive

GIFT: Geometry-Informed Low-Precision Gradient Communication for LLM Pretraining


Poster Type: Research Posters

Author: Jieying Wang (Rutgers University–New Brunswick), Shuyuan Fan (Rutgers University–New Brunswick), Mingkai Zheng (Rutgers University–New Brunswick), Fangru Linghu (Rutgers University–New Brunswick), Zhao Zhang (Rutgers University–New Brunswick)

Abstract: Gradient communication is a major scaling bottleneck in large language model (LLM) pretraining. While FP8 can substantially reduce communication cost, direct quantization in Euclidean coordinates can distort highly anisotropic gradients. We present GIFT, a geometry-informed gradient communication method that transforms gradients into more isotropic coordinates before low-precision quantization. To make this practical, GIFT uses a one-sided, low-rank geometry approximation and applies it selectively to layers most sensitive to quantization error, while leaving the optimizer, training recipe, communication collective, and FP8 format unchanged. Across Llama-style models with approximately 600M and 2B parameters, GIFT achieves downstream performance comparable to Muon training with FP32 gradients. On 64 NVIDIA GH200 Superchips, GIFT reduces end-to-end training step time by 10.87% for Llama-600M and 17.05% for Llama-2B relative to FP32 gradient communication.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF


Back to Poster Archive Listing