A thriving open-source software community is a wonderful thing. Developers and users learn from each other, contribute to features that benefit everyone, and ensure the project remains stable and relevant. For the LLNL-led MFEM project, which has been open source for more than a decade, community collaboration has helped evolve this C++ software library into a robust mathematical engine for scientific simulations.
MFEM, which stands for Modular Finite Element Methods, provides algorithms that break down high-order mathematical equations into discrete operations, improving a simulation’s performance while reducing its computational cost. The software is used in large-scale simulations of scientific phenomena and processes including additive manufacturing, biotechnology, design optimization, electromagnetics, fluid dynamics, nuclear energy, structural mechanics, and more.
“The community is a catalyst for our annual workshops,” stated principal investigator Tzanio Kolev. “We are motivated by the challenges they face and the successes they have.”
The MFEM team hosted a virtual workshop on October 26 to discuss the development roadmap and hear how researchers use the software in their scientific applications. The workshop also included a Q&A session, student lightning talks, and a visualization contest. Registration drew 270 people from more than 130 organizations—research labs, industry, academia—across 34 countries. Along with Kolev, LLNL organizers were Aaron Fisher, Milan Holec, Ketan Mittal, Socratis Petrides, and Vladimir Tomov, joined by Portland State University professor Will Pazner.
Application Spotlight
The workshop featured more than a dozen talks by MFEM collaborators on topics as varied as topology optimization for industrial applications; thermomechanical and fusion energy modeling including tokamak simulations; and electromagnetic and radiofrequency wave propagation. “I’m always surprised at the variety of problems people are solving with MFEM,” said Fisher.
A relatively new use of MFEM comes from the Amazon Web Services (AWS) Center for Quantum Computing. Though a fundamentally different paradigm from classical computing, quantum computing relies on the latter for the hardware design process. AWS senior applied scientist Sebastian Grimberg described how his team simulates quantum computing hardware components in 3D with Palace—PArallel, LArge-scale Computational Electromagnetics—an open-source finite element code designed for full-wave electromagnetics simulations. For example, simulations of superconducting metamaterials or qubit and resonator geometries can inform chip design for quantum computers.
Scalable solvers are crucial to these simulations. “The geometries we’re concerned with have a very wide range of feature sizes. The models become very large very quickly, so we need large-scale 3D simulations to enable our design work,” explained Grimberg, whose team regularly runs models with more than one billion degrees of freedom. Furthermore, large design spaces require running many models in parallel, so high performance computing workflow automation is important to optimizing the design process.
In development since 2021, Palace is built on MFEM and the libCEED library, which was produced for the Exascale Computing Project’s Center for Efficient Exascale Discretizations. Grimberg noted, “The risk of building your own solver is greatly improved by the availability of open-source projects like MFEM. We were able to stand on the shoulders of giants instead of building things up from scratch.” Palace leverages MFEM’s support for NVIDIA and AMD GPUs as well as its operator assembly capabilities.
Student Showcase
New this year, the workshop featured five-minute lightning talks by students. “Successful research reflects the diversity and potential of the community,” said Holec. “It was great to hear the young voices of science and learn how these scientists embrace and apply MFEM to solve diverse problems. ”
Shani Martinez Weissberg, a PhD student at the Computational Mechanics and Experimental Biomechanics Lab at Tel Aviv University, presented her model for analyzing bone strength following laser-induced thermotherapy (LITT). “LITT is used to destroy tumors in soft tissue. Within bone tissue, the surgeon may need to drill a hole into bone and administer high localized temperature to destroy the tumor cells. We wish to expand the use of this therapy to treat bone tumors,” she explained. The study is in a preclinical stage, so Weissberg models rabbit femurs to predict the risk of bone fracture during the invasive therapy.
The key to this analysis is micro-finite elements (µFE). Weissberg extracts bone tissue voxels (3D points on a grid) from micro-computed tomography, performs segmentation of the bone tissue using MIA image processing software, then converts each 40 µm3 voxel into a hexahedral element using Simpleware ScanIP software. A typical load is applied on the head of the femur in the µFE model, which has more than 125 million degrees of freedom and relies on MFEM to solve a linear elasticity problem. “We use MFEM because of its ability to handle a large model,” Weissberg noted. “A human bone is significantly larger than a rabbit bone. This will create a much larger model and emphasizes the importance for MFEM to handle large models.”
Community Growth
MFEM’s origins can be traced to a Texas A&M University project more than two decades ago, and some of the first contributors remain involved, now as part of LLNL’s Center for Applied Scientific Computing (CASC). Kolev reflected, “It has been amazing to see the growth of MFEM and its impact in the Department of Energy and the broader scientific computing community. We never imagined that when we started it more than 20 years ago.”
Released as open source in 2010, MFEM version 4.6 came out this September, bringing the total lines of code to almost 400,000. Kolev stated, “While we have tried to keep MFEM small but powerful over the years, the growth of the user base and the development of new features such as GPU computing have led to a steep upward trajectory of code additions.” According to GitHub data, the software averages 250 downloads or clones daily (91,000 annually) and has been downloaded in 115 countries.
Comparing workshop data from the past three years, the MFEM community is clearly growing. Almost 25% of workshop registrants were students, and 62% of all participants claimed less than one year of experience with MFEM. “We’re seeing tons of people who are new to MFEM and have heard about it from colleagues and collaborators,” noted Fisher.
Beyond education and experience, workshop demographics also revealed that half of participants resided outside the U.S. And most of LLNL’s MFEM team members are originally from other countries. Holec, whose talk summarized a deterministic transport application built into MFEM, pointed out, “Our project’s international flavor is represented in the workshop’s speakers and audience. We had more talks by non-native English speakers than native.” Fisher added, “It’s a good thing we all speak math.”
MFEM’s 2024 development roadmap includes GPU performance and portability enhancements, additional examples for both new and advanced users, and continued work on design optimization and automatic differentiation—all of which will be part of the version 5.0 release. Workshop speakers’ slides and videos are posted to the MFEM website, and visualization contest winners are featured in an online gallery.
—Holly Auten