Workshops Archive

Past the FLOPS Era: Memory-Bandwidth Disaggregation as a Design-Space Coordinate for LLM Accelerators


Workshop: 6th International Workshop on RESource DISaggregation in High Performance Computing (RESDIS)

Authors: Bambi DeLaRosa, Katya Giannios, and Khalid Ahmad (Micron Technology)

Abstract: LLM accelerator design inherited FLOPS-era HPC assumptions that today’s production serving no longer satisfies, foremost one high-bandwidth pool sized for weights. Key/value cache bandwidth demand overtakes weights at deployment batch sizes. We characterize accelerators by two bandwidth scalars (βw ,βs) and workloads by state-to-weight ratio ρ(n) and weight reuse r. Per decode latency splits across the two pools, and a local-minimum theorem proves the shared-pool corner strictly dominated past b·ρ(n) = 1, with the penalty at most 2×. The two-tier gain at equal cost rises from 1.6% at the balance point to 34% at 512K context, while one compromise design recovers 3.9%. Only 0.11% of the feasible designs in a 1,270,129-point space come within 10% of optimal. The contribution is a coordinate system in which memory-centric codesign can be reasoned about in hardware terms, not only at the serving layer: how the optimum migrates with context, and the trade-offs induced.


Back to 6th International Workshop on RESource DISaggregation in High Performance Computing (RESDIS) Archive Listing

Back to Full Workshop Archive Listing