Poster Type: ACM SRC Posters
Author: Irene Simo Munoz (Cornell University)
Abstract: LM decoding is bounded by the memory wall: weights and KV cache move every token, and the low-arithmetic-intensity matrix–vector products of decode leave throughput set by bandwidth rather than compute. Bank-level PIM attacks this by keeping weights resident and computing in place, but its lockstep execution cannot skip zeros. We present the first end-to-end sparsity-aware inference framework for bank-level PIM. A column-union cost model shows that lockstep cost is set by the union of columns (or KV tokens) a bank-group must visit, not the nonzero count. Activation-aware weight pruning aligns surviving columns across rows and matrices, while a group-aligned adaptation of query-aware KV selection collapses the attention scan. Against a measured NVIDIA H200 at batch-1, this work cuts per-token latency by 4–7× and, beyond moderate context, undercuts the GPU’s most energy-efficient batched configuration (crossing near 10K, around 6× lower at 128K), while preserving accuracy and extending on-board context.
Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF