Thanks to a dynamic developer community and a long history of encouraging open-source software (OSS) development, Lawrence Livermore has reached quadruple-digit GitHub offerings. One thousand—and counting!—LLNL software projects have been publicly released on the GitHub platform since the first one was published in October 2008.

These projects address a wide range of needs: software tailored for unique high performance computing (HPC) scenarios, user documentation, tutorials, automation tools, scientific applications, student projects, codes accompanying research papers, and much more. Some have been in development for decades, while others serve a single ephemeral purpose.

One thousand is an impressive round number, but arguably an arbitrary one. Not all software projects are still useful, and many are no longer actively developed. Nevertheless, says OSS advocate Ian Lee (GitHub username @IanLee1521), “This milestone is significant because it reflects the Lab’s culture of curiosity, collaboration, and transparency.”

Lee, who serves as LLNL’s HPC security architect and Security Operations Team lead, launched an online catalog (software.llnl.gov) in 2015 to provide users and developers with a portal into the rich software ecosystem that underpins much of the Lab’s daily mission-driven work. The catalog aggregates all Lab-developed OSS repositories—colloquially known as repos—hosted on GitHub. This includes all repos under the LLNL GitHub organization plus those from other LLNL-affiliated organizations. (The catalog itself is open source, so its data is publicly accessible. The organization list is housed in a JSON file.)

Release the Codes!

Some of the Lab’s software is necessarily classified, while other proprietary projects are licensable by external parties. Otherwise, software projects are publicly released whenever possible, paving the way for collaboration with research partners from academia, industry, and other government agencies.

For example, the Department of Energy’s (DOE) Exascale Computing Project relied heavily on OSS not only to lower the barriers to development, but also to build a software ecosystem usable by exascale-capable HPC centers—in other words, to help developers develop it and users use it. In 2023, LLNL and Sandia National Laboratories announced the High Performance Software Foundation, a nonprofit venture intended to encourage DOE labs to invest in and release software, ultimately bringing it to the mainstream for other entities trying to solve similar HPC challenges.

Within the Lab, the developer community has established recurring forums where OSS takes center stage—seasonal hackathons and the annual Developer Day event—so teams can work on their own releasable software, learn how to use existing open-source tools, and share wisdom and experiences with each other.

In addition, developers have worked closely with LLNL’s Innovations and Partnerships Office to create a streamlined release process supportive of OSS development. Lee, who spearheaded the release policy efforts, states, “We needed to provide developers with a clear process and set of guidelines for the release of open-source software. This process needed to encompass all the proper reviews important at a national security lab, but also not get in the way of leveraging this powerful resource and opportunity.”

This evolution of the Lab’s OSS portfolio is not without growing pains. Minimally used repos present a low cyber security risk, but heavily developed and more widely adopted repos require close inspection of contributions before they can be merged into the code base. “Security is essential in our mission, and many project teams make use of automated checks, trusted contributor access, and other safeguards to ensure code integrity,” Lee states. Furthermore, the Lab collaborates with the Open Source Security Foundation (OpenSSF) to develop secure build processes and quality assurance methodologies.

Everyone Wins

The open-source paradigm confers many benefits on developers and users, regardless of whether the software is developed at LLNL or elsewhere. The natural feedback loop can lead to new features and bug fixes. Developers, including student interns, can hone their skills and demonstrate their talents for job opportunities. Code released with a scientific paper encourages reproducibility. A project’s public exposure can inspire new applications or commercial potential, and collaborative development can encourage and sustain partnerships. Ultimately, OSS improves the Lab’s computing capabilities and even industry standards.

Computer scientist Vanessa Sochat (@vsoch) is the primary developer for the Flux Operator, a Kubernetes extension for running a “MiniCluster” of the HPC workflow management software Flux. In addition to standing up converged computing (HPC + cloud) environments for Flux, she contributes to dozens of other OSS repos based not just on her areas of expertise, but also depending on what catches her interest. “My favorite language at the moment is Go, so I tend to migrate toward it,” Sochat says, “But of course I work in Python a lot, too.”

Adrien Bernede (@adrienbernede), a former LLNL employee now working abroad, continues to contribute to the Lab’s RADIUSS project, which is a suite of open-source HPC tools and libraries. He develops and maintains continuous integration processes, helping OSS teams save development costs without needing specialized skills. “I think that making the decision to open software to others, like LLNL did, is paving the way to better science,” states Bernede.

Crucial Communities

Some LLNL-affiliated projects have grown into huge communities with their own GitHub organizations and multiple supporting repos, prompting user-focused events and communication channels. Several open-source projects have gained traction because of their teams’ engagement efforts and through participation in community-building activities like the tutorials hosted by LLNL’s High Performance Computing Innovation Center (HPCIC) every August.

Sochat points out, “A lot of community work goes beyond writing code, such as participating in standards groups, giving talks at large venues, or championing our work and vision on social media. Impact has to go beyond a specific employer and touch the larger community.”

For instance, the Spack team regularly holds live tutorials and informal “birds of a feather” sessions where users can meet the developers and ask questions about the award-winning package management tool, which installs the user’s code on almost any type of computing platform or environment. Since its open-source release a decade ago, Spack has received thousands of contributions outside LLNL.

Project lead Todd Gamblin (@tgamblin) notes, “About 150 people work on Spack every month, and we review and merge anywhere from 300 to 600 changes during that period. We have a pretty deep bench of developers.”

Sustained community engagement is partly attributable to project teams who ensure their software remains relevant as users’ needs evolve over time. SUNDIALS, a library of time integrators and nonlinear solvers that released v1.0.0 in 2002, was downloaded more than 140,000 times last year. To showcase their software’s versatility, the SUNDIALS team publishes a list of real-world use cases, including in simulations ranging from power grids and thermodynamic processes to spacecraft trajectories and networks of neurons.

Computational scientist Cody Balos (@balos1), who joined the team in 2018, says “Over the years we have added new features and numerical methods to SUNDIALS that have enabled it to run on the latest and greatest HPC systems, remain state-of-the-art, and work for a wide variety of science applications. All the while, we have improved our software development practices and the design of SUNDIALS to ensure that it continues to be user friendly, fast, and high quality.”

Averaging 91,000 downloads per year, the MFEM project provides high-order finite element discretization algorithms that improve the accuracy and performance of computer simulations. The team’s annual workshop brings together researchers, mathematicians, and developers from nearly every continent.

“It’s been great to see the growth of our open-source community, and we are constantly amazed by the new applications and the creativity of MFEM users,” states project lead Tzanio Kolev (@tzanio).

Respect in Tech

Livermore’s open-source innovation is frequently recognized internationally and at high-profile venues. Almost a dozen OSS projects have won R&D 100 Awards over the past two decades—most recently Variorum, a power management library for any type of HPC hardware, and zfp, which provides data compression for HPC applications without sacrificing accuracy.

Last fall, SUNDIALS won the 2023 SIAM/ACM Prize in Computational Science and Engineering, which recognizes outstanding contributions to the development and use of mathematical and computational tools and methods for solving science and engineering problems. The Spack team won the HPCwire Editor’s Choice Award for the Best High Performance Computing Programming Tool or Technology at SC23, the world’s premier supercomputing conference.

Also at SC23, the open-source SCREAM project snagged the inaugural Gordon Bell Prize for climate modeling. Led by LLNL with collaborators from several national labs, SCREAM is the first global cloud-resolving model to run on an exascale supercomputer and at unprecedented resolution.

“The world is taking notice of what we’re doing,” Lee points out. “We’re able to make an impact in scientific computing, HPC, and software development because of our open-source footprint, contributions, and passionate developers.”

Follow @LLNL_OpenSource on Twitter/X or contact open-source@llnl.gov.

—Holly Auten (@hauten)