The open-source Axom project, supported by the Advanced Simulation and Computing (ASC) program at LLNL, focuses on developing software infrastructure components that can be shared by high performance computing (HPC) applications running on diverse computing platforms.

Initially, Axom development focused on providing fundamental capabilities for MARBL, a next-generation ASC multiphysics code in development at LLNL. As Axom has evolved and grown, it is now used in multiple LLNL ASC applications. Axom is also part of the RADIUSS project, an institutional effort that promotes and funds the integration of software developed in LLNL programs into projects across the Laboratory.

Axom is a library of robust “building block” software components that provide foundational capabilities needed for production simulation codes. Axom components include:

  • Inlet. Input file parsing, verification, and input data storage and retrieval supporting file readers for JSON, Lua, and YAML.
  • Klee. Format specification for material shape input to multi-material simulations; tools for placing those shapes on meshes.
  • Lumberjack. Scalable distributed memory parallel message logging and filtering.
  • Mint. Lightweight mesh data model that supports Axom tools.
  • Primal. Geometric primitives (point, segment, ray, etc.) and spatial operations.
  • Quest. Spatial queries over surfaces and volumetric meshes (signed distance, in-out, point location, etc.).
  • Sidre. Hierarchical key-value in-memory data base (meshes, fields, etc.) and parallel file I/O, including integration with the Scalable Checkpoint/Restart (SCR) software.
  • Slam. Set-theoretic API for accessing arbitrary subsets of mesh and other simulation data (sets, relations, etc.).
  • Slic. Configuration interface for coordinating diagnostic, warning, error messages, and more between integrated application components.
  • Spin. Spatial index structures for managing and accelerating spatial searches.

Axom is one of the key modular software projects applied widely in the LLNL ASC program that embodies a new approach to developing applications in LLNL’s Strategic Deterrence (SD) Principal Directorate. The others include RAJA, Umpire, Conduit, and BLT. Specifically, the modular projects emphasize software sharing and reuse and aim to:

  • Make applications portable across a wide range of computing platforms while maintaining high performance on individual platforms
  • Simplify management and sharing of mesh-based data across applications and their constituent physics packages
  • Facilitate collaborative development and sharing of modular software building blocks across applications; relieve the burden of software maintenance effort
  • Enable the adoption of common development tools and practices across the LLNL ASC simulation software ecosystem

Modular Features

In traditional WCI physics applications, infrastructure that supports the types of features Axom provides tends to be unique to each application code base and comprises 30% or more of each application’s code. Centralizing these capabilities in a modular library that applications can use selectively makes application development more efficient and helps to insulate simulation projects from key challenges associated with advanced technology HPC systems.

Figure 1 shows the coupling between the Axom datastore component, Sidre, and other software in a multiphysics application. Multiple physics packages operating on different meshes and mesh types store and access their data in Sidre data hierarchies, which conform to the Conduit Mesh Blueprint mesh-aware data schema. Then, other physics packages, libraries, and data processing and manipulation tools (visualization and analysis; mesh and geometry queries) can be developed independently and interoperate and share data easily by programming to a single interface.

Figure 1. Mesh data sharing is a central pillar of our approach and enables the coupling of applications, libraries, and tools. Here, simulation data from two packages employs different mesh types: structured and unstructured. Simulation data is described with a common mesh-aware data schema, defined by the Conduit Mesh Blueprint, and stored in a shared datastore, Axom’s Sidre component. This simplifies the ability of tools, libraries, and other physics packages to access and process the data and to integrate new tools into existing applications.

Other modular components in Axom provide fundamental capabilities that can be shared across applications. For example, point containment queries, which determine whether a point in space lies inside or outside a region defined by a surface, are important for “shaping” materials onto a simulation mesh. This process is illustrated in Figure 2.

Figure 2. Axom’s point containment query supports efficient tests of whether points lie inside or outside a region bounded by a closed surface. On the left, an octree hierarchical data structure is constructed around elements in a surface mesh to accelerate spatial point queries. On the right, points on a simulation mesh can be quickly tested whether they lie inside or outside the region bounded by the surface to “shape in” materials on the mesh.

Another example is Axom’s signed distance query that determines the minimum distance from a point to a surface mesh. This is important for computing gradients in immersed boundary applications, for example. The signed distance query is illustrated in Figure 3.

Figure 3. Axom’s signed distance query is used to define a scalar distance field on a mesh based on an embedded surface. On the left, a bounding volume hierarchy is constructed around a surface mesh to efficiently determine a small subset of candidate surface elements where the minimum distance from a point in space “q” may occur. On the right, points on a simulation mesh are used to construct a scalar signed distance field relative to the surface.

Lastly, Axom’s point location query can be used to determine values of a simulation mesh field at arbitrary points in space. This is a common use case to determine field values at locations associated with “tracer particles” or probes in a simulated experiment. See Figure 4.

Figure 4. Axom’s point location query is used to find field values on a simulation mesh. The query uses a spatial index data structure to quickly find the mesh element containing a given point. Then, a robust iterative solver is used to determine the parametric coordinates of the point in the element, which can then be used to determine the field value at the point.

Ongoing Development

The Axom project continually adds new components and capabilities to meet ASC application needs. For example, Axom’s new “Klee” component is being developed to provide general geometry and material specification in order to shape materials onto a simulation mesh. Klee provides volume fractions for each material in a mesh element, for any mesh type supported by the Conduit Mesh Blueprint.

Geometry and material specification are often time-consuming aspects of setting up programmatically important computational simulations. A common use case is that, currently, users must translate geometry and material information from one input format to another to run problems with more than one code. When fully deployed, Klee’s common input format will eliminate the effort needed to translate geometry from one code to another. Figure 5 shows a simple code-agnostic input specification and shaping the material information onto a simulation mesh.

The focus on software sharing and reuse in Axom is rooted in a vision to foster a collaborative computer science ecosystem that broadens the developer base beyond individual, independent WCI application projects. Because Axom is open source, it can be used by applications in the broader HPC community. This approach helps enable LLNL programs to engage researchers, students, and vendors as well as establish a bridge to help deploy innovative research concepts into production applications.

Figure 5. “Shaping” material and geometry data onto a simulation mesh is an important aspect of simulation set up. Axom’s Klee component provides a way to specify and validate geometry and material information in a code-agnostic way. Then other Axom tools, such as Quest, are used to shape the material data onto a mesh.