Commercial products could not patch our Macs institution wide, so we developed our own solution.
Charles Heizer, MacPatch lead developer
MacPatch system management application and desktop from a developer’s perspective

MacPatch Keeps Thousands of LLNL Computers Running Smoothly

Monday, February 26, 2018

Lawrence Livermore’s Information Technology Solutions Division, in support of the Livermore IT (LivIT) program, oversees enterprise computer system management for more than 6,300 employees, many of whom regularly use multiple desktop or laptop computers. Working for the LivIT program, the Systems Management Solutions Group is responsible for all operating system (OS) platforms used at the Laboratory, with specialized staff dedicated to Mac, PC, and Linux OS.

On average, about 3,200 of the Laboratory’s active computers are Macs. Software developer Charles Heizer knows well the challenges that come with enterprise-level OS management. “Installing third-party patches is a lot of work. Apple and other vendors don’t address patch management well for an enterprise’s needs. In some cases, determining which patch is relevant for a given update makes it difficult to transition from one patch to the next,” he explains. “And, as a national laboratory, we can have more security concerns than a private-sector company does.”

So Heizer began developing a Mac-specific solution to address managing and installing Apple specific patches. Over time, the need for patching third-party software increased as “zero day” exploits were found in a number of software products. It became clear that a standalone solution was the only way to accommodate both centralized system management and support as well as unique program needs. MacPatch was born.

The first generation of MacPatch used SOAP application programming interfaces and ran on Mac OS hardware. As Mac adoption increased across the Laboratory, Heizer explored solutions to address pain points such as heavy memory usage and difficulties in reusing code. The second generation of MacPatch included a range of improvements and was able to handle software installation, not just patching. “Managing Macs in an enterprise environment doesn’t mean all the hardware is Mac based. We added Linux server infrastructure,” states Heizer. “MacPatch can also attach patches to software packages, so users can update software at the time of installation.”

The third generation of MacPatch was released in May 2017 and featured a transition to REST-based web services and Python libraries in a Flask ecosystem. “Version 3.0 was a major overhaul of the backend. We implemented new technologies to solve performance and scaling issues,” explains Heizer. Key objectives for this version were performance and code reuse, as well as moving from lesser-known open-source tools to those with larger communities. As a result, MacPatch is easier to grow and maintain.

Furthermore, the 3.0 release went smoothly—no service interruptions—and reduced network traffic while doubling the number of clients per server. “Initially, I thought the benchmark totals were wrong,” recalls Heizer. “But this version was a huge success in terms of efficiency. The second generation took a lot more effort and servers to keep the lights on. Moving to new technologies helped us realize significant performance gains.” Apple representatives were onsite in June to review Livermore’s Mac environment, giving Heizer’s team an opportunity to demonstrate MacPatch’s capabilities.

This progress has enabled Heizer to focus on the future. “It’s still significant work to manage all the Macs here, but I don’t have to hover over the system anymore. I have more time to support related services around the Laboratory and contribute to mobile device software development,” he states.

Development of MacPatch version 3.1 is under way. In addition to rewriting the administrator console and upgrading various software libraries, the team is changing MacPatch’s design to handle configurations on the server side instead of the client side. “The next release will help Mac administrators better control and centralize their environments,” says Heizer.

Heizer’s team belongs to Apple’s beta testing community, which receives access to the next Mac OS version in advance of public release. “We start testing immediately so we’re ready for deployment and customer support on Day One,” notes Heizer. Software developer Jorge Escobar leads MacPatch’s quality assurance and technical support, and the team completes comprehensive troubleshooting before the rest of the Laboratory population receives the upgrade.

As open-source software since 2013, MacPatch is available on GitHub.