HPC Unites Generations

To explore where high-performance computing may be headed, this summer, I invited members of the community to join me for a conversation. Florina Ciorba, Dong Li, Onkar Patil, Alex Knigge, Ruth Hammond, and Yan Guimarães had much to say about the state of HPC. Each brought a different level of experience and a unique perspective on the HPC landscape to this informal roundtable.

Although they span different generations, career stages, and sectors, the participants shared many of the same hopes, concerns, and ideas about the future of HPC. Their answers revealed a surprising amount of common ground about HPC’s role in tackling society’s most complex challenges.

Florina Ciorba

An Associate Professor at the University of Basel, Florina studies performance optimization, from scheduling inside a single node to execution across many nodes or jobs. She has worked in fault tolerance, system failures, monitoring, and energy efficiency. 

Dong Li

Dong is the Co-Founder and Chief Scientist at Yotta Labs and an Associate Professor at the University of California, Merced. He worked at the Department of Energy (DOE)’s Oak Ridge National Laboratory (ORNL) for four years and at Tenstorrent (a startup working on artificial intelligence accelerators) as a Senior Principal Memory Architect. His current project includes working on AI-native clouds. He is a pioneer at building memory tiering for HPC.

Alex Knigge

Alex works at DOE’s Sandia National Laboratories in the Lightweight Distributed Metric Service team and has been doing HPC competitions and internships since her freshman year of college. She plans to pursue a master’s degree at the University of Cambridge. At SC, she’s been a team manager and coach for three years. 

Onkar Patil

Onkar went into an industry position after university. He is currently a Member of Technical Staff at Zyphra, focusing on AI inference, distributed and disaggregated deployment, CPU and GPU optimization, collective communication, and memory management.

Ruth Hammond

Ruth is a third-year graduate student who has completed five internships with DOE National Labs (ORNL, Lawrence Livermore National Laboratory, and Los Alamos National Laboratory) and two with Industry (J&J and IBM Research). She studies AI/machine learning workloads on HPC, as well as evaluation and benchmarking.

Yan Guimarães

Yan is an HPC and distributed systems researcher from Brazil whose work centers on MPI optimization and the Julia ecosystem. His contributions to projects such as Dagger.jl and Reactant.jl help make distributed and hardware-accelerated computing more accessible and efficient.

HPC Is More Than People Think

Breanna: In one sentence, what is the most misunderstood thing about HPC today? 

Ruth: The public mostly fears HPC as a sci-fi superintelligence or confuses it with ‘datacenters’, if they know it at all, not realizing this powerful computational hardware is behind things they rely on daily, like weather forecasting and medicine.

Yan: The biggest misconception is that HPC is primarily about building massive hardware, when the real challenge today is developing the complex software and compilers needed to actually harness these diverse architectures.

Dong: HPC is not only about performance.

Florina: The importance of research in scheduling. Many people believe that since scheduling is an ‘old and hard problem’ it has been studied for many decades and we have all the best solutions in place. That is untrue and unfortunately, only simplistic and often inefficient solutions are deployed in practice.

Why HPC Matter More Than Ever?

Florina: What makes HPC attractive to you, now in the age of AI and other exciting novelties? 

Onkar: My personal analogy about HPC is that it is the Formula One of Computer Science where the frontiers of computational performance and capabilities are constantly pushed by innovation and research. The same breakthroughs are then applied in many different use cases which enhances our daily capabilities. With theoretical and algorithmic advances in AI, now HPC sits front and center in enabling and bringing this new age of information processing to all the people. This brings in its own set of challenges that need to be carefully resolved with HPC innovation so that we sustainably move forward. There couldn’t be a more interesting time to be part of HPC before this.

Dong: To me, HPC is unique, because it combines the knowledge of the whole system stack (from architecture, OS, compiler, and programming models). Also, it gives me a venue to apply HPC to scientific applications from a variety of domains. In the age of AI, I’m happy to see that much old HPC knowledge has been re-visited and played critical roles to make AI efficient and powerful.

Florina: HPC is attractive to me since I first learned about parallel computing in spring/fall 2002 after I started my PhD. Parallel computing was the first course I took as a PhD student, and was taught by my advisor (who made an exception and said will teach it in English vs. Greek – that was at the tech univ of Athens, Greece). I learned about the classical parallel architectures and algorithms, about systolic arrays (Ring a bell today? Ahem, dataflow architectures, Cerebras?) and I witnessed the dream of Exascale computing becoming a reality. And then the boom of AI (classical and generative). I’m wondering: is there anything not exciting about HPC today? The age of AI just makes HPC even more important, because it disseminates the need for HPC beyond the classical computational science community and HPC aficionados.”

Yan: Probably because it was one of the first doors that opened to me, but I do like HPC since I like to get my hands dirty with complicated questions, put my brain to work, and study how computing works – for example, ‘how we can use that power to predict whether a hurricane will happen in the next few days and prepare everyone for it’. I mean, it is an important scientific field where we need to think carefully about what we are doing, and it really attracts me.

HPC and AI: Partnership or Race?

Ruth: Where do you see HPC’s role in the AI race evolving over the next decade? 

Florina: HPC fueled/fuels AI and AI can take HPC to the next level. I think it is a beautiful symbiosis and synergy between two fields that can co-evolve beyond what each can evolve into alone. I think AI shouldn’t be a race in the first place, although it is. HPC should support the right contestants (if AI is a race, namely the academics, not the industry) and not be wastefully used/abused, e.g, for brute force training without algorithmic (yes, even parallel AI algorithms) research/innovation and with disregard for the sustainability impact of this computing power.

Onkar: HPC is now the most important thing that enables AI today and will scale it tomorrow. Current naive scaling practices that the AI industry has adopted have led to many concerns over unnecessary compute and memory HW hoarding, energy crisis and other environmental hazards. HPC should look at the challenge to again efficiently scale itself to the next frontier by innovating newer communication, memory and compute technologies so that we ethically and efficiently handover AI technologies to the next generation.

Dong: HPC has been and will continue to play critical roles for training and inferencing of large foundation AI-models. HPC may interact more often with the edge devices in the near future to make AI more approachable by people.

Yan: That is a question that always gets me thinking. Since AI is the ‘new’ technology of this era, I am always wondering how we can optimize AI models to use less computation. I believe we already have extremely powerful computers, so the answer cannot be only to get more computational power or faster communication. I think AI models need to be optimized in how they work and how they use computation, memory, and communication.

What Should HPC Focus On Next?

Florina: What other definitions do you see for the ‘P’ in HPC than performance? 

Alex: Progress. HPC is enabling progress in ways that weren’t even imagined a few decades (even years) ago. Biomedical sciences, climate research, physics, and so many more fields are getting the chance to be looked at in a new light because of the computational power that HPC produces.

Dong: Portability. With the lower bar of designing domain-specific accelerators, the accelerators are becoming more diverse and common. Ensuring program portability and performance portability across a spectrum of hardware is a key to the success of future HPC.

Yan: Programmability. By that, I mean: ‘Can domain researchers express parallel work correctly without needing to get their hands dirty with communication protocols, scheduling systems, data distribution, synchronization, load balancing, and so on?’ I probably have this idea about HPC because of the project I participated in. One of the main ideas of Dagger.jl is how we can deliver high performance while letting the runtime manage many of the complexities of HPC.

Onkar: It is hard for me to say anything other than Performance. But if I have to, I will say ‘Possibilities’. With HPC, the possibilities of what computational problems you can solve is endless

Alex: What subsection of HPC deserves more attention than it’s currently getting? 

Florina: Algorithmic thinking and making great ideas in the literature runnable in practice. For example, optimal scheduling on parallel machines is notorious for being NP-hard. Algorithmic research in scheduling either reinvents the wheel (again and again) or is discarded as research contributions for there is a belief that scheduling is an old problem, (partially) solved, and not deserving attention. Then, why is there so much resource waste, if we don’t need better scheduling solutions, especially in practice? Why are there still high wait times in the queues, if we have resource waste?”

Dong: I think that the HPC system hasn’t been able to successfully integrate AI components for resource management and scheduling. Those management problems include memory management across nodes, job management across time periods, power management (including power grid control) across the HPC infrastructures.

Onkar: Speaking from an infrastructure point of view, I will say Power and interconnects. The two most energy and cost consuming technologies deserve more attention than what is being given. In compute, memory and storage scaling, these two get sidelined but will play a huge role in scaling and enabling beyond Exascale.

Yan: I believe runtime systems that manage many of these complexities for the user need more attention. They can provide an abstraction layer inside the HPC software stack and let researchers focus more on their scientific questions.

The Biggest Challenges Ahead

Onkar: What scares you, if anything, about HPC? 

Alex: HPC is growing faster than anyone can control right now. With how it’s linked directly to AI, both fields are symbiotically growing each other. The environmental impacts of HPC are highly concerning and are being disregarded in the name of innovation. I fear that this trend will continue and it will be too late for intervention when we need it.

Dong: The power consumption of HPC systems is becoming larger and larger. As there are more HPC data centers, the environmental impact of HPC is often ignored. Additionally, HPC systems tend to use mix-precision computation and change the execution order of computation. How can we verify the computation results of HPC systems? Are the results generated by HPC fully reliable?”

Onkar: HPC is quickly moving from being a scientific stream to becoming a commercial stream. But the years of research and experience that the scientists have had in carefully developing this field till now could be easily ignored. For example the environmental impacts of running exascale computers. HPC as a field getting generalized to any other Computer Science field is scarier than any other field getting generalized.

Florina: That we don’t consider the impact on real life our field and community bears. We need to think, every time we write a piece of code and put it out there (online), that this code may be used for some other purpose than we intended it. I used to think that we, computer scientists, are in a ‘comfortable’ position of not impacting real lives directly, like medical doctors do. I’m not so sure about that, anymore.

Advice for the Next Generation

Yan: As a young engineer looking to grow, what foundational topics or books, especially around core mechanics like communication protocols, would you recommend I study, and how should early-career software engineers adapt their learning as the HPC landscape shifts under the massive costs and demands of AI?

Florina: Fundamental principles do not change, no matter how fast everything else changes. So long as we are using a von-Neumann based computer, we will always be governed by the ‘laws’ and ‘rules’ of a stored program (in memory) computer, with clear computation, communication, and data storage and movement steps, concurrent or not, distributed or not, synchronized or not. With a non-von-Neumann computer, we will have to see what other concepts data, computing, communication map to and if we need to adopt new terms/concepts/etc.

Dong: I would recommend learning computer architecture, parallel programming, and AI algorithms. Learning computer architecture helps to identify and resolve performance problems in parallel systems. Those problems may not be easily identified by AI (at least by current AI). Parallel programming is needed, even though we are using vibe coding, because we need fundamental programming knowledge to verify the correctness of the code generated by AI.

Onkar: The computer science books that were relevant 30 years ago are still relevant today. The basics of HPC wouldn’t change. I would always point students to reading the same textbooks to grasp the core HPC principles. But to see how they are applied in today’s world I would recommend reading top conference papers after understanding the core concepts.

Florina: I subscribe to Dong Li’s opinion, and would add that not only ‘sequential’ computer architecture is a critical fundamental, but also parallel computer architectures, which are built atop sequential computer architectures, but where the complexities and intricacies compound the challenge of parallel programming (on shared/distributed memory, with/without accelerators).

Expanding Access to HPC

Ruth: If you had unlimited resources, what is one thing you would do to make HPC more accessible to underrepresented communities or early-career researchers? 

Florina: I would donate a small HPC system to each such community and educate its members how to use it. Hackathons, blue sky ideas competitions, fairs, and of course, start early (not only at university). Donations and offering education for free, breaks so many barriers (mental, institutional, practical).

Onkar: What is currently stopping HPC from being a seamless, daily tool for you, and if it were easily deployable at the push of a button, how would it change what you work on?

Florina: HPC is complex. Rocket science is complex. I don’t think we want rocket science at the push of a button. Not sure we want HPC to work like that too (it can always be used for not noble purposes too). Goes back to the other question of what scares me and the missing realization of our own responsibility around HPC.

Looking Toward the Future of HPC

Dong: With the rapid rise of diverse hardware accelerators, will we ever reach a point where a program can run on any chip without having to be completely rewritten for each one? 

Florina: This is an issue of portability, software- and performance-wise, and relates to what I see as an important ‘p’ in HPC. High level programming languages/models that have the right level of abstraction exist (OpenMP, OpenACC, CUDA->HIP, Intel OneAPI, Kokkos, Raja, etc.). The challenge is that only few of these are standard and many of these started as research efforts. I would like to propose that ‘no new HW architecture should ever be allowed to be sold if it does not offer a standard-based programming model’. This would solve the problem posted by Dong. And certainly slow down the pace of innovation for HW manufacturers and HW adopters. Would that be so wrong? To go fast in an inefficient direction or slow in the right (more efficient) direction?”

Yan: ‘Any chip’ is an ambitious aim, and ‘ever’ is also ambitious, so when thinking about it that way, I really do not know. However, I think that in the next decades, it is totally possible to have programs that can run on several of the most widely used chips without having to be completely rewritten. That does not mean they will automatically have optimal performance on every chip, but it can greatly reduce the amount of backend-specific work required from the programmer. Reactant.jl, a project that I am also involved in, is already working toward this at the compiler level.

Onkar: I think it will soon be a reality with the new age AI coding tools. It is much easier to write an assembly cross architecture transpiler that efficiently and effectively replicates the instructions from one architecture to the other. We will soon be moving to a pseudocode/algorithmic design process which will be translated into native code for any architecture efficiently.

Dong: Vibe coding will be very helpful to quickly develop a code base customized to a specific accelerator, assuming that the training data to build the AI model for vibe coding is sufficient enough. Given the emergence of the vibe coding, it would be interesting to see how we evolve those high-level, architecture-transparent programming models, such as OpenCL. If we can develop low-level code quickly using vibe coding, is there a strong need for a high-level programming model?”

Dong: As supercomputers move toward more modular, “disaggregated” designs where components are physically separated, how do you think standard communication tools like MPI will need to evolve? 

Florina: HPC is a tradeoff space. Modular designs are very nice and useful. What we gain in separation of concerns through modularization we lose in performance through the need to communicate between more (and potentially farther) modules.

Onkar: Disaggregation and modularization will have to be closely accounted for in Communication technologies so that programs can keep track of where its process data will exist and where it needs to move. We will see that MPI could evolve into a more tightly coupled paradigm for disaggregated memory architectures.

Dong: With the support of hardware architecture and system software, the disaggregation can be hidden from the standard tools and libraries. For example, the CXL-based memory disaggregation can be exposed to the MPI library as a traditional NUMA node. This will enable the portability of those tools and libraries across the disaggregated systems.