Sean Lam: Uniting a Computing Coalition for Climate Research

Sean Lam of the Western Australian Climate Science Initiative presented the project’s 150-year, 4 km climate projection dataset as part of the SC Illuminations program.

When Sean Lam took the stage in the SCinet Theater at SC25 last November, he filled one of 24 slots that HPC Illuminations Pavilion (now known as SC Illuminations) reserves each year to showcase high-performance computing research that conventional conferences may overlook. Lam used the time to tell the international HPC community how a two-person team at a state government agency in Western Australia (WA) delivered the most detailed statewide climate projections produced for WA to date by building a partnership that no single institution could have produced alone. Looking at the numbers, it is clear to see why the partnership was necessary. Commercially procuring equivalent compute and storage resources would have required an investment estimated at around $20 million, but the team’s budget was $7 million. The only choice was either to streamline the workflow or find a way to build a coalition that could close the gap. So the team started reaching out.

As a Principal Climate Modeler in the Climate and Sustainability Portfolio at WA’s Department of Water and Environmental Regulation (DWER), Lam leads the modeling work for the state’s Climate Science Initiative (CSI) within DWER. He coordinates the technical partnership, model design, computational strategy, and stakeholder engagement required to deliver the project. SC26’s theme, “HPC Unites,” reflects what the SC Illuminations program is designed to do: spotlight research in which the work itself depends on coordination and community building in areas that might not otherwise be noticed. Lam’s path to the SC Illuminations stage runs through a region that most global climate models cannot clearly see. 

“WA is basically one-third of the landmass of Australia, and it is a really big task,” he said of the project. “It’s not just a capital city. It covers from tropical climate to subtropical.” Sharing the scope of the problem, Lam explained that since 1910, WA has warmed by approximately 1.3°C and has experienced some of Australia’s most significant climate shifts, particularly in the southwest. The problem, he said, is that global climate models operate at larger grid scales of 100 to 250 kilometers and do not provide sufficient granularity to distinguish the future climate of Perth from that of Bunbury, 175 km (109 miles) south of it. Many publicly available Australian climate projection products which include WA are produced at resolutions of approximately 20 km, which remain too coarse for many regional adaptation applications.

Quite a lot of the climate data is focused on the east coast of Australia, because there are more research institutes on that side,” Lam said. “Most of the time, their data doesn’t really represent the Western Australia climate. So, the Western Australian Climate Adaptation Fund funded this project in 2021.

A still from a hailstorm simulation produced by the Western Australian Climate Science Initiative. High-resolution regional modeling of extreme weather events like this one supports infrastructure and adaptation planning across Western Australia. Image courtesy of Sean Lam.

Like many people who find themselves working in HPC, Lam noted he did not start out thinking he would be operating supercomputing systems. His postgraduate research was in atmospheric chemistry modeling on desktop machines, and he moved through consultancy doing dispersion modeling that ran comfortably on desktop hardware. The shift came when state and federal government work brought him into climate model simulations that demanded a different scale of compute. 

“I got a taste of where the world is moving,” Lam said. “To run this kind of model, you have to use a supercomputer. That’s how I gained more exposure and knowledge through this sort of modeling, slowly moved into the field, and started working with HPC systems. I was learning new things every day.” 

He is plain about how he arrived: “I’m one of the ones who never thought I’d be working directly with supercomputers, actually. My experience with supercomputers is just from the last few years.”

For Lam, leading the modeling meant working with those around him to build a partnership around the work. Kelly Barnes, DWER’s senior manager, structured the project and secured the funding. Murdoch University brought regional climate modeling expertise through Associate Professor Jatin Kala and senior postdoctoral researcher Julia Andrys. The Pawsey Supercomputing Research Centre provided compute on its Setonix system with roughly 150 million Service Units allocated over the three-year project. The New South Wales government contributed model input data and expertise through its NARCliM 2.0 program. 

Setonix at the Pawsey Supercomputing Research Centre. The Western Australian Climate Science Initiative’s projection workload runs on Setonix under what project lead Sean Lam describes as one of the largest allocations Pawsey has supported in Western Australia. Image courtesy of Pawsey Supercomputing Research Centre.

---

About the Setonix Supercomputer

The Setonix system is housed at Pawsey Centre, and is the most powerful research computer in the Sourthern Hemisphere. It is built on the same architecture as Frontier at Oak Ridge National Laboratory, as well as Lumi at CSC Finland.

• Manufacturer: HPE Cray
• Model: EX Supercomputer
• 1584 Dual 2.45GHz AMD EPYC 7763 “Milan” 64-Core CPU Nodes with 256 GB RAM
• 16 Dual 2.45GHz AMD EPYC 7763 “Milan” 64-Core CPU Nodes with 1 TB RAM
• 154 Single AMD EPYC 7A53 “Trento” 64-Core GPU Nodes with eight AMD Instinct MI250X GPUs and 256 GB RAM
• 38 Single AMD EPYC 7A53 “Trento” 64-Core GPU Nodes with eight AMD Instinct MI250X GPUs and 512 GB RAM
• Eleven Data mover nodes   
• Thirty-One Visualization nodes
• Nine Login nodes 
• Connected by HPE’s Slingshot interconnect (200Gb/sec)
• Lustre file systems /scratch (14 PB [3 SSD, 11 HDD]), /software 
• NFS /home   

Source: Pawsey Supercomputing Research Centre

“We started with a two-person team,” he said, “But through the collaborations and working with partners, we managed to deliver something that maybe a much bigger team could deliver.”  

Pawsey staff carried much of the technical load. Setonix runs on AMD architecture, and the team’s partners had been working on Intel, so the climate model required recompilation and validation in the new environment. Maciej Cytowski, who leads Pawsey’s scientific services team, worked through the issues that surfaced. 

“Pawsey actually wrote up instructions, basically a standard procedure of how this is done,” Lam said. Sean Fleming, a senior data specialist at Pawsey, suggested the approach that ultimately cut required storage in half. 

Lam explained that the most important optimization was a workflow change. Regional climate modeling typically uses a technique called “dynamical downscaling,” in which simulations are run at successively finer resolutions, allowing the computational heavy lifting to be focused where it is most needed. Rather than performing the entire downscaling chain from global climate models, i.e., downscale from coarse-resolution global climate model output (~100–150 km) to 20 km and then to 4 km, the team leveraged NARCliM 2.0’s 20 km regional climate simulations as a starting point and downscaled directly to 4 km resolution using one-way nesting.The revised workflow reduced computational requirements by approximately 45% and cut storage requirements roughly in half . 

Without the partnership, we might have delivered only a fraction of the coverage required to represent Western Australia’s diverse climate regions, Lam added.

He was also clear about why the coalition worked: “It’s a great partnership, but it’s not a standard partnership. A lot of people went above and beyond just the standard paperwork to help us. If you have a common goal and aim to do something good for the field, people are willing to help, but you can’t push them too much. That’s just how we roll.” 

That pattern, where multiple institutions coordinate around an outcome that no single party could deliver, underpins SC26’s “HPC Unites” theme. His presentation as part of SC25 Illuminations in the SCinet Theater put Lam’s work in front of an audience beyond his usual channels. Aside from the opportunity to share his team’s work, he says one of the most concrete outcomes from the experience came from a conversation on the exhibition floor with the team from Taiwan’s National Center for High-performance Computing, which has been running a regional climate projection program for nearly two decades at a scale considerably larger than the CSI’s current allocation. 

“They’ve been doing that for more than 20 years,” Lam said. “They basically run every global climate model, like 30 of them, and we only run five. Everything is done really well: the science, data storage, and communication. I had a good chat with them, and I learned quite a lot from their project. If we want success in this project, this is like the gold standard.” 

The CSI is now finishing the south-west domain of the projection set and moving toward WA’s northwest domain. Lam is also thinking about what the project’s outputs might enable beyond the immediate planning use case. Beyond adaptation planning, the resulting dataset may also provide a valuable training resource for emerging AI-based climate modeling approaches. High-resolution, physics-based regional simulations remain scarce globally, and the 150-year dataset produced through CSI could help support the development and validation of next-generation machine-learning climate models.. 

“This is a substantial dataset that researchers can build upon,” Lam noted. “We have 150 years of climate data based on carefully selected ensemble of high-performing climate models. I think this is a good opportunity for the HPC community.”  

Looking back at SC25, Lam was direct about the experience. “It can be intimidating, just the sheer amount of work out there and hearing how people are talking about the future,” he said. “But in my opinion, everyone should be working to go to the SC Conference at least once. I think it’s a really good experience.” 

SC Illuminations