A team of CASC researchers recently hosted the second annual workshop for the MFEM user and developer community. The event was designed to promote collaboration, describe the project’s latest features, expand application engagements, and solicit feedback to guide future development. More than 150 participants from dozens of organizations and countries joined the one-day hybrid workshop. Noting that half of the workshop’s attendees were outside the U.S., CASC group leader Aaron Fisher stated, “MFEM is a global community, and mathematics is a language that crosses borders easily.”

The open-source MFEM software library provides advanced finite element discretization methods to many high performance computing (HPC) applications in the Department of Energy (DOE), academia, and industry. Large-scale simulations powered by MFEM impact a wide range of scientific phenomena and processes—electromagnetics, fluid dynamics, additive manufacturing, topology optimization, structural mechanics, biological applications, nuclear energy, and more. The project’s discretization methods enable HPC systems to run these simulations more efficiently, and MFEM has become a trusted part of the DOE’s Exascale Computing Project as well as the Office of Science SciDAC Program’s strategic software portfolio.

“While purely virtual, our 2021 workshop succeeded in bringing the MFEM community together for the first time, fostering many new developments and collaborations,” said MFEM principal investigator Tzanio Kolev. “This year we were pleased to offer a hybrid option for Lab attendees at the new University of California Livermore Collaboration Center.” Alongside Kolev and Fisher, workshop organizers were CASC computational mathematicians Socratis Petrides and Ketan Mittal, and former LLNL Fernbach Fellow Will Pazner, who is now assistant professor of Applied and Computational Mathematics at Portland State University.

Finite Elements, Infinite Possibilities

In the decade-plus since being released as open source, MFEM’s capabilities have grown to include dynamic adaptive mesh refinement, mesh optimization, high-order methods, scalable solvers, GPU support, and more. Today the project has over 120 contributors, 150 GitHub visitors per day, and 180 downloads per day. “Because of its open-source development, MFEM attracts users and developers from all over the world,” Pazner pointed out. “Many contributors find us on GitHub and begin interacting through issues and pull requests.”

Indeed, Ben Zwick from The University of Western Australia (UWA) met the MFEM team through GitHub, and he kicked off the workshop’s first session of technical talks. Zwick described the role of predictive models in treating epilepsy, which affects about 50 million people globally. Brain tissue resection is one treatment, but locating the specific area of seizure onset is a major challenge for surgeons. Electrophysiological source imaging procedures, such as intracranial electroencephalography (EEG), are key diagnostic resources in determining where electrical disturbances occur in the brain.

Figure 1: In the temporal lobe of an epilepsy patient, streamlines (blue) of the electric field are generated by a current dipole source (magenta). The finite element solution uses a regular hexahedral grid implemented in MFEM. Image courtesy of Ben Zwick, winner of the 2022 MFEM workshop visualization contest.

Zwick’s team at UWA’s Intelligent Systems for Medicine Laboratory has developed methods to solve the EEG forward problem that predict the brain’s postoperative geometry after implantation of intracranial EEG electrodes. He noted, “We’re using MFEM to manage finite elements and boundary conditions and [Python wrapper] PyMFEM to approximate the electric dipole source within the brain. Not only is the code very fast, but the development time is also fast and interactive” (see Figure 1). His team is integrating their MFEM-based solution into the open-source 3D Slicer software for biomechanical and electrical modeling, as well as generating meshes directly from diagnostic images.

Christina Migliore, a PhD student at the Massachusetts Institute of Technology, discussed her team’s development of the Stix mini-app, which is built on MFEM. Stix provides a simulation model for ion cyclotron radiofrequency (ICRF) power in fusion devices, such as tokamak reactors. In particular, Stix enables researchers to explore the microscale physics of the narrow sheath region around a reactor’s plasma-facing components, which are susceptible to hot spots and impurities.

Sheaths are tiny compared to the reactor’s size, so they can be approximated as nonlinear boundary conditions. Migliore explained, “Numerically modeling ICRF sheaths will help us understand where they form, which factors influence rectification, and ways to mitigate problems.” With MFEM’s finite elements, Stix can couple a sheath boundary condition to a global wave solver, rendering the solution as a pseudo-2D mesh (see Figure 2). Migliore’s team developed Stix with MFEM team member Mark Stowell over a few years as part of the RF-SciDAC project in the DOE SciDAC program. The RF-SciDAC principal investigator Paul Bonoli acknowledged the extensive development required by the collaboration, stating, “We weren’t using an off-the-shelf product, and we were very happy to be able to engage Mark and others at MFEM.”

Figure 2: Powered by MFEM, the Stix mini-app is able to look at a cross-section of a strap ICRF antenna on a 2D mesh, shown here before adaptive mesh refinement (AMR) steps. The simulation enables identification of damage to the device’s plasma-facing components. Images courtesy of Tom Jenkins (TechX; left) and Christina Migliore (right).

Adolfo Rodriguez, co-founder of OpenSim Technology, introduced a DOE Small Business Innovation Research project that models subsurface wellbores to prevent collapse. “Wellbore stability is a relatively simple problem with a lot of stress and pressure complications at the same time,” he explained. “Our equations solve for mechanical equilibrium linear elasticity, and we use PyMFEM to implement the solution.” OpenSim’s synthetic well model requires a refined mesh around the wellbore itself to accurately capture stress points, then a coarser mesh farther away from that focal point (see Figure 3). The model can compute a second grid of concentric rings, which depict stresses around the cylindrical wellbore and identify failure zones. Rodriguez’s team has implemented performance speedups with parallel processing, and they are working on a web app for engineers to use in the field.

Figure 3: A wireframe representation of the x-direction displacement, obtained by solving the linear-elastic equilibrium equations of a pressurized synthetic cylindrical wellbore under far-field stress boundary conditions.

The workshop also showcased unique applications developed at LLNL, where many code teams rely on MFEM’s finite element discretization, mesh partitioning, and other features. For instance, Jorge-Luis Barrera from the Computational Engineering Division discussed the Livermore Design Optimization (LiDO) code, which solves optimization problems for a wide range of Lab-relevant engineering applications. Leveraging MFEM and the LLNL-developed engineering simulation code Serac, LiDO delivers a powerful suite of design tools that run on HPC systems. Barrera highlighted several design examples that benefit from LiDO’s integration with MFEM, including multi-material geometries, octet truss lattices, and a concrete dam under stress (see Figure 4). LiDO’s graph architecture that seamlessly integrates MFEM features ensures robust topology optimization, as well as shape optimization using nodal coordinates and level set fields as optimization variables.

“It’s always interesting and inspiring to see people using MFEM in new ways that we hadn’t anticipated or didn’t know about. It’s clear that MFEM is one of the premier finite element packages, and has significant mindshare in the research community,” said Pazner, whose graduate students use PyMFEM in combination with Jupyter notebooks to learn finite element concepts. Fisher added, “When we interact with the community this way, we gain insight into how researchers are using MFEM and if they may be struggling. This really helps us improve the experience for our customers inside and outside of the Laboratory.”

Figure 4: MFEM enables the LiDO code to perform gradient-based systematic design optimization, as in this example of stress minimization on a concrete dam. Updating the mesh coordinates generates optimal designs that experience large shape changes compared to their initial configuration. Image courtesy of Jorge-Luis Barrera.

Valuable Discourse

The workshop gave the MFEM team a chance to explain key features to newer or inexperienced users, as well describe enhancements planned for upcoming releases. The event coincided with the version 4.5 release, which includes container and cloud support, improved GPU algorithms, submesh extraction, better meshing and solvers, a new mini-app for automatic differentiation in nonlinear elasticity, and a fractional partial differential equation example. The team is already preparing MFEM for LLNL’s El Capitan and other heterogeneous exascale supercomputers.

Beyond the roadmap, technical talks, and a tutorial, the workshop’s Q&A session and active Slack channel offered participants a forum for discussing implementation issues, making suggestions, and learning more from the team in real time. “We learned valuable feedback from the MFEM users that we will take into consideration in future development,” noted Kolev.

Speakers’ slides and videos are posted to the MFEM website. Winners of the workshops’ visualization contest are featured in an online gallery.

—Holly Auten