The open-source Axom project, supported by the Advanced Simulation and Computing (ASC) program at LLNL, focuses on developing software infrastructure components that can be shared by high performance computing (HPC) applications running on diverse computing platforms.
Initially, Axom development focused on providing fundamental capabilities for MARBL, a next-generation ASC multiphysics code in development at LLNL. As Axom has evolved and grown, it is now used in multiple LLNL ASC applications. Axom is also part of the RADIUSS project, an institutional effort that promotes and funds the integration of software developed in LLNL programs into projects across the Laboratory.
Axom is a library of robust “building block” software components that provide foundational capabilities needed for production simulation codes. Axom components include:
- Inlet. Input file parsing, verification, and input data storage and retrieval supporting file readers for JSON, Lua, and YAML.
- Klee. Format specification for material shape input to multi-material simulations; tools for placing those shapes on meshes.
- Lumberjack. Scalable distributed memory parallel message logging and filtering.
- Mint. Lightweight mesh data model that supports Axom tools.
- Primal. Geometric primitives (point, segment, ray, etc.) and spatial operations.
- Quest. Spatial queries over surfaces and volumetric meshes (signed distance, in-out, point location, etc.).
- Sidre. Hierarchical key-value in-memory data base (meshes, fields, etc.) and parallel file I/O, including integration with the Scalable Checkpoint/Restart (SCR) software.
- Slam. Set-theoretic API for accessing arbitrary subsets of mesh and other simulation data (sets, relations, etc.).
- Slic. Configuration interface for coordinating diagnostic, warning, error messages, and more between integrated application components.
- Spin. Spatial index structures for managing and accelerating spatial searches.
Axom is one of the key modular software projects applied widely in the LLNL ASC program that embodies a new approach to developing applications in LLNL’s Strategic Deterrence (SD) Principal Directorate. The others include RAJA, Umpire, Conduit, and BLT. Specifically, the modular projects emphasize software sharing and reuse and aim to:
- Make applications portable across a wide range of computing platforms while maintaining high performance on individual platforms
- Simplify management and sharing of mesh-based data across applications and their constituent physics packages
- Facilitate collaborative development and sharing of modular software building blocks across applications; relieve the burden of software maintenance effort
- Enable the adoption of common development tools and practices across the LLNL ASC simulation software ecosystem
In traditional WCI physics applications, infrastructure that supports the types of features Axom provides tends to be unique to each application code base and comprises 30% or more of each application’s code. Centralizing these capabilities in a modular library that applications can use selectively makes application development more efficient and helps to insulate simulation projects from key challenges associated with advanced technology HPC systems.
Figure 1 shows the coupling between the Axom datastore component, Sidre, and other software in a multiphysics application. Multiple physics packages operating on different meshes and mesh types store and access their data in Sidre data hierarchies, which conform to the Conduit Mesh Blueprint mesh-aware data schema. Then, other physics packages, libraries, and data processing and manipulation tools (visualization and analysis; mesh and geometry queries) can be developed independently and interoperate and share data easily by programming to a single interface.
Other modular components in Axom provide fundamental capabilities that can be shared across applications. For example, point containment queries, which determine whether a point in space lies inside or outside a region defined by a surface, are important for “shaping” materials onto a simulation mesh. This process is illustrated in Figure 2.
Another example is Axom’s signed distance query that determines the minimum distance from a point to a surface mesh. This is important for computing gradients in immersed boundary applications, for example. The signed distance query is illustrated in Figure 3.
Lastly, Axom’s point location query can be used to determine values of a simulation mesh field at arbitrary points in space. This is a common use case to determine field values at locations associated with “tracer particles” or probes in a simulated experiment. See Figure 4.
The Axom project continually adds new components and capabilities to meet ASC application needs. For example, Axom’s new “Klee” component is being developed to provide general geometry and material specification in order to shape materials onto a simulation mesh. Klee provides volume fractions for each material in a mesh element, for any mesh type supported by the Conduit Mesh Blueprint.
Geometry and material specification are often time-consuming aspects of setting up programmatically important computational simulations. A common use case is that, currently, users must translate geometry and material information from one input format to another to run problems with more than one code. When fully deployed, Klee’s common input format will eliminate the effort needed to translate geometry from one code to another. Figure 5 shows a simple code-agnostic input specification and shaping the material information onto a simulation mesh.
The focus on software sharing and reuse in Axom is rooted in a vision to foster a collaborative computer science ecosystem that broadens the developer base beyond individual, independent WCI application projects. Because Axom is open source, it can be used by applications in the broader HPC community. This approach helps enable LLNL programs to engage researchers, students, and vendors as well as establish a bridge to help deploy innovative research concepts into production applications.