Disclaimer: This article is more than two years old. Developments in science and computing happen quickly, and more up-to-date resources on this topic may be available.

Researchers from LLNL’s Center for Applied Scientific Computing hosted a virtual workshop on October 20 for the MFEM user and developer community. The workshop’s goals were to promote collaboration, describe the latest features, expand application engagements, and solicit feedback to guide MFEM’s future development.

MFEM, which stands for Modular Finite Element Methods, is a C++ software library that provides high-order mathematical algorithms for large-scale scientific simulations—such as topology optimization for additive manufacturing or wave propagation for electromagnetics applications. The project’s discretization methods enable high performance computing systems to run these simulations more efficiently. The Department of Energy’s Exascale Computing Project includes MFEM in its strategic software portfolio.

More than 150 researchers from dozens of organizations and countries attended the one-day workshop. “I was surprised by how many people signed up and wanted to present their work. I had no idea of the breadth of MFEM’s reach,” stated Aaron Fisher, who organized the event with Tzanio Kolev, Will Pazner, and Mark Stowell. According to the registration survey, more than half of the participants were new users.

MFEM version 1.0 was released as open source in July 2010 and moved to GitHub in 2017. Today, the software features many linear and nonlinear solvers, analysis of 2D and 3D finite element spaces, advanced meshing capabilities, and parallel scalability for different hardware architectures including heterogeneous CPU/GPU machines like LLNL’s Sierra supercomputer. The GitHub repository averages 35 downloads per day.

Figure 1. Presenting an overview of MFEM, Kolev described adaptive mesh refinement features with animations that show the mesh adjusting to the edge of the blast wave as the latter expands.

“Over the years, we’ve enjoyed a growing community of both internal and external users and contributors, and the Lab’s MFEM team has expanded to more than 15 people. But we hadn’t gathered the community together like this before,” said Kolev, who has led the project since its inception. “Beyond sharing the latest features and our development roadmap, we felt it was important to learn how researchers and application scientists are using MFEM and how we can be of better service to them.”

Engaging with users has many benefits. Fisher pointed out, “People have begun to use MFEM in ways we hadn’t considered. Because we’re open source, we can learn from how others use MFEM, and, in turn, feed this work back into the Lab’s missions. Also, having a product like MFEM used for cutting-edge research around the world raises the Lab’s stature in the research community.”

A Full Agenda

The workshop kicked off with overviews of the project and its recent developments, then featured a dozen presentations highlighting a wide range of application areas that leverage MFEM’s algorithms and capabilities, some of which are described below. Throughout the day, attendees connected on Slack to ask questions, post speakers’ slides, and share publication links.

Members of the MFEM team demonstrated how to take full advantage of specific features. For instance, Vladimir Tomov described the project’s mesh optimization strategies including ways the user can define target elements. He showed how to optimize a mesh’s shape by limiting displacements to preserve a boundary layer and by changing the size of a uniform mesh in a specific region.

For users unfamiliar with MFEM’s solver library, Pazner demonstrated a few ways—in some cases adding just a single line of code—to run scalable solvers for differential equations. These solvers execute hierarchical finite element discretizations for both low- and high-order problems. “Some of these solvers are standard in our field, while others are based on new methods published by Lab researchers,” he said.

Participants also joined breakout sessions to focus discussions on MFEM’s three primary application areas—electromagnetics, fluids, and structural mechanics—with moderation by Stowell, Julian Andrej, and Jamie Bramwell. For example, Stowell explained, “Computational electromagnetics has enabled important advances in microcircuitry, medical imaging, motor design, battery technologies, and much more. MFEM makes finite element modeling of electromagnetics possible and accurate, so engineers and designers can create smaller and more efficient devices than ever before.”

In the weeks leading up to the workshop, the community was invited to enter a contest showcasing MFEM-powered simulations. Pazner noted that only during the judging process did the organizers realize the need for two winners—one for static images and one for animations. “It was too hard to select just one winner,” he said. “We considered each submission’s application, use of MFEM, image resolution, technical difficulty, and visual impact.” Submissions will be featured in the MFEM online gallery.

Figure 2. The winners of the simulation contest represent two very different research applications using MFEM. Left: The electric field generated by electrocardiogram waves of a rabbit’s heart ventricles, rendered by Dennis Ogiermann of Ruhr-University Bochum (Germany). Right: Incompressible fluid flow around a rotating turbine, animated by Tamas Horvath of Oakland University (Michigan).

Application Showcase

As open-source software, MFEM is readily available to researchers outside the Lab, and most of the workshop’s talks were given by external collaborators and users. A trio of undergraduate students from the National University of Colombia presented their work on phase change heat and mass transfer, in which they used MFEM to simulate brinicles—icicles made of briny water that form beneath sea ice. These simulations require finite element solutions to calculate heat and salinity convection-diffusion as well as vortex formation and fluid flow.

“I was impressed with presenters who only recently started using MFEM. The Colombian students picked up MFEM quickly and produced interesting results in one semester. That made me happy,” said Stowell, who has worked on the project for 5 of his 20 years at the Lab. Pazner added, “I was surprised at the scope of presentations. The connection between abstract ideas and real-world applications isn’t always apparent, so seeing these examples at the workshop was encouraging.”

Figure 3. William Dawn from North Carolina State University explained his team’s use of MFEM in neutron transport modeling. The 2D MARVEL reactor model runs on Oak Ridge National Laboratory’s Summit supercomputer, and MFEM’s support for GPU architectures is crucial for rendering the model’s mesh with 3 million vertices and 1.8 billion degrees of freedom. A 3D model, like the one shown here, is in development. Image courtesy of William Dawn.

William Dawn from North Carolina State University described his work with unstructured neutron transport. His team models microreactors, a new class of compact reactor with relatively small electrical output. As part of the Exascale Computing Project, Dawn’s team is modeling the MARVEL reactor, which is planned for construction at Idaho National Laboratory. MFEM satisfies their need for a finite element framework with GPU support and rapid prototyping. With MFEM, the team discretizes a neutron transport equation with six independent variables in space, direction, and energy.

The MARVEL reactor model—including a mesh with 3 million vertices and 1.8 billion degrees of freedom—runs on Oak Ridge National Laboratory’s Summit supercomputer. “MFEM has been a game-changer for our project. Being able to start the day with a new idea for a solver and finish with a new implementation that can run on Summit is huge for us,” Dawn stated.

Mathias Davids from Harvard Medical School demonstrated MFEM’s use in a medical setting. Magnetic resonance imaging (MRI) generates anatomical and physiological images via non-ionizing electromagnetic waves. Davids, who contacted MFEM developers for assistance in 2020, explained how MRI machines use magnetized gradient coils to perform image encoding in MRI. Harvard researchers are investigating MRI-induced peripheral nerve stimulation (PNS), which can cause pain or muscle contractions during the scan and limit the achievable spatial and temporal image resolution.

Figure 4. Harvard Medical School researchers are using MFEM in predictive models that investigate PNS. The results can help optimize gradient coils (shown in red and blue) in MRI machines to reduce PNS effects. Image courtesy of Mathias Davids.

To understand the phenomenon and improve gradient coil design, Davids’ team has created models that predict PNS effects. He said, “MFEM plays a central role in our efforts to develop these PNS models.” The high-spatial-resolution models contain electromagnetic pathways for 2,000 nerve segments in the human body. MFEM provides the solvers to render electric field patterns on hexahedral meshes made up of 80 million mesh elements.

Davids continued, “We have been making good use of MFEM’s partitioning and large-scale parallelization capabilities. Initially, these simulations using commercial software packages took hours to days to run. Using MFEM, we’ve been able to bring this down to 30 to 40 minutes.” By identifying PNS “hot spots” in these models—and after seeing good agreement between predictive data and experiments—Harvard researchers are optimizing coil designs to mitigate PNS effects in next-generation MRI systems.

Beyond the (Next) Workshop

With the next MFEM version scheduled for summer 2022, the project roadmap includes additional meshing functionality, accommodation for physics in new domains, documentation improvements, and expanded GPU support in preparation for LLNL’s El Capitan and other exascale supercomputers. The team also plans to explore machine learning, cloud computing, and design optimization for engineering applications in future versions.

Organizers noted that the workshop revealed opportunities for growth. For instance, presentations from the University of Texas at Austin and the French Atomic Energy Commission highlighted external teams’ efforts to extend MFEM’s capabilities in new use cases. Additionally, Kolev stated, “There was interest in incompressible flow problems that’s disproportionate with our efforts in this area. We have always listened to the needs of application teams and users, so this feedback is quite important.” Stowell added, “We were able to gauge the community’s priorities, which will help us set development priorities in the future.”

With many more presentation abstracts submitted than the one-day agenda could accommodate, the MFEM team looks forward to hosting another user-focused event and expects to adjust the format going forward. In addition, they plan to invite workshop contacts to participate in a seminar series and are already following up on potential collaborations. Videos of the 2021 presentations are available online.

For Fisher, seeing MFEM’s use in the broader scientific community is exciting and motivating. He said, “While we are hammering out code every day, it’s easy to forget that our project has such a broad impact. I’m proud to be part of the team.”

—Holly Auten