Many scientific simulations—like those supporting LLNL’s national security mission—contain systems of linear equations, so application codes often rely on linear solvers to get the job done. Created at LLNL in 1998, the hypre software library provides specialized, scalable solvers for a range of purposes. A recent major release, version 3, includes a new semi-structured algebraic multigrid (AMG) solver along with support for mixed numerical precision at runtime.

“We refactored and rewrote a big portion of the code for this release,” explains project lead Rob Falgout. “Development in hypre has always been driven by support for applications and users, and we’ve tried to think strategically about the future so our developers can spend more time on research and algorithms and less time on fixing problems.”

Structured for Speed

Among hypre’s strengths are its AMG solvers, which accelerate simulations of complex physical phenomena by efficiently solving the sparse linear systems that arise from discretized partial differential equations. These solvers, particularly BoomerAMG, are designed for massively parallel computation.

AMG methods are well-suited for unstructured problems, but hypre doesn’t stop there. The software includes support for structured grids, where the underlying structure of the discretized mesh is known. Version 3 introduces SSAMG, a semi-structured solver that expands existing multigrid solver capabilities to address more types of grids and, in turn, provide more options for users. The “semi” aspect of this new solver is key, as some grids are partially or compositely structured. SSAMG treats grids as structured parts with arbitrary connections, which helps reduce computational overhead. The team’s extensive testing on representative problems has shown SSAMG to outperform BoomerAMG in many cases.

“We’ve wanted to develop this new solver for a long time,” Falgout notes. “In hypre, we have always let users describe the structure in their problems, but it hasn’t been easy to develop solvers that take advantage of that structure. Now we’re providing better solutions in semi-structured settings.”

Figure: The new SSAMG solver accommodates multiple structured grids coupled by unstructured connections (black lines). In this example, each structured grid is coarsened in a different direction as part of the SSAMG algorithm, and coarse-grid information is interpolated back to the fine grid to accelerate the solution process.

Precisely Right

Like most of LLNL’s foundational high performance computing (HPC) software projects, hypre has evolved with the rise of heterogeneous architectures and exascale computing power. Furthermore, application teams are increasingly seeking flexibility with different types of workloads, which often include machine learning pipelines in addition to traditional modeling and simulation. The hypre team has had to consider key facets of the modern HPC environment including graphics processing units (GPUs), memory usage, and precision requirements for floating-point arithmetic. “Users need accuracy but also want to improve speed and performance while reducing the memory footprint and taking advantage of GPUs,” says Falgout. 

Building on its support for multiprecision at compile time, hypre now provides both multi- and mixed-precision computation at runtime. (In multiprecision, a solver uses one precision at a time, whereas mixed-precision uses different precisions simultaneously.) Users can switch from one type of precision to another—for example, from double to single or from single to long double—without recompiling the application code. An upcoming enhancement will apply different precisions to each grid level of AMG solvers.

Looking Ahead

The project’s nearly three-decade run has shown there’s always something new to investigate. For instance, the team is exploring artificial intelligence tools for developing new algorithms as well as for helping hypre choose solver parameters for the user. GPU refinements for mixed-precision solvers are also in progress. And Falgout points out, “Some classes of problems we don’t yet know how to solve effectively with multigrid methods.”

Alongside Falgout, the LLNL team responsible for hypre v3 include Rui Peng Li, Victor Magri, Wayne Mitchell, Daniel Osei-Kuffuor, and Ulrike Meier Yang (now retired). Development in is funded in part by the Department of Energy’s Office of Science and its Scientific Discovery Through Advanced Computing (SciDAC) program, and by the National Nuclear Security Administration’s Advanced Simulation and Computing (ASC) program. Watch Falgout’s FEM@LLNL seminar describing v3 features.

—Holly Auten