Poster Type: Research Posters
Author: Sean Tai (Queensborough Community College), Mohammad Atif (Brookhaven National Laboratory), Sankha Dutta (Brookhaven National Laboratory), Meifeng Lin (Brookhaven National Laboratory)
Abstract: Large language model inference is increasingly limited not only by raw compute capability, but also by memory bandwidth, runtime behavior, and GPU power efficiency. We study how LLM inference performance and energy use vary across hardware platforms and inference frameworks. Experiments are conducted on NVIDIA's Jetson Thor, and A100 and H200 GPUs using two inference frameworks: vLLM and Hugging Face Transformers. Three open-weight LLMs are evaluated. For each hardware, framework, and model combination, inference metrics are collected. Pareto and histogram analyses are used to study the trade-off between throughput and energy efficiency. To connect inference behavior to hardware limits, BabelStream benchmark's triad kernel is used to measure host-memory and GPU-memory bandwidth. In addition, roofline analysis separates prefill and decode phases to show when performance is compute-bound or memory-bound. These measurements help explain why the Jetson Thor can be energy-efficient, while A100 and H200 provide higher throughput but consume more power.
Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF