Authors: Bin Ma and Xingjian Ding (University of California, Merced); Tekin Bicer (Argonne National Laboratory); and Pengfei Su and Dong Li (University of California, Merced)
Abstract: Diffusion Transformers (DiTs) are increasingly adopted in scientific computing, yet growing model sizes and resolutions make distributed multi-GPU inference essential. Ulysses sequence parallelism scales DiT inference but introduces frequent all-to-all collectives that dominate latency. Overlapping these with computation is difficult due to tight data dependencies, large messages, and asymmetric interconnect bandwidths.
We introduce CoCoDiff, a distributed DiT inference engine exploiting two observations: (1) V requires only linear projection while Q/K need additional normalization and RoPE, enabling overlap of V's communication with Q/K computation; (2) adjacent denoising steps produce similar tensors, yielding temporal redundancy. CoCoDiff introduces three mechanisms: Tile-Aware Parallel All-to-all (TAPA) decomposes collectives into topology-aligned phases; V-First scheduling hides V's communication behind Q/K computation; and V-Major selective communication transmits only active projections on slow interconnects. On Aurora with four DiT models across 1–8 nodes (up to 96 Intel GPU tiles), CoCoDiff achieves an average speedup of 3.6×, peaking at 8.4×.
Presentation: file