View an LLNL Flickr album of hackathon photos throughout the years.

Hackathons are one of LLNL Computing’s most enduring and beloved traditions. Although some details have changed since the first hackathon, the premise remains the same: Participants have 24 hours to work on any project of their choosing, whether that’s learning a programming language, building a prototype, developing a new skill, or experimenting with a software framework. They are encouraged to explore ideas outside their day-to-day work. Above all, it’s okay to fail.

Originally called Ship-It Day, Computing’s first hackathon took place in the summer of 2012 and encouraged participants to “think it, build it, pitch it, ship it”—in other words, come up with an idea, try to execute on it, and present their findings to the group at the end of the 24-hour period. More recently, presentations generally follow an outline of “what I wanted to do, what happened, what I learned.” In the early years, teams won prizes for innovation. Other giveaways have included buttons and stickers featuring the hackathon logo.

After 10 years and 33 hackathons, Computing is committed to this tradition. “Looking back, we can honestly say that hackathons have become all that we imagined and more,” states Livermore Computing (LC) leader Becky Springmeyer. “Hackathons have become a highly valued tradition among a wide variety of employees. Some bring their whole team. Others use the occasion to work with strangers who become friends by the end of the event. Yet others seize the opportunity to learn or evaluate something new on their own.”

Grassroots Innovation

On the surface, hackathons provide staff with a chance to step away from their desks and meet others in the Computing community. But such interactions with colleagues—along with a change of scenery and plenty of snacks including the ubiquitous popcorn machine—can spark creativity and collaboration. Moreover, hackathon projects often end up affecting real projects.

Underlying all of this is the explicit permission to take risks and to explore rapidly changing technologies and applications that may not be immediately relevant to a hacker’s regular work. Computing associate director Bruce Hendrickson points out, “There are numerous examples of hackathon projects maturing into funded activities, and there are even more examples where employees have honed a skill via a hackathon and then applied it to Lab missions.”

Amar Saini, a data scientist in the Global Security Computing Applications Division (GS-CAD), focuses on machine learning whenever he attends a hackathon. When choosing a project, he first conducts a literature survey to see what others in the field have done. His hackathon projects have included automatic conversion of car color in photos and videos, speech denoising through the use of convolutional neural networks, and artificial intelligence to play games through reinforcement learning. “I like creating something fun with friends and seeing others adopt this culture at the Lab. Having an energetic community for hackathons makes it easier for others to get started and stay motivated,” he states.

Software engineer Joe Eklund has used hackathon time to develop a status dashboard for the applications he supports in the Applications, Simulations, and Quality (ASQ) Division. Adapted from the open-source monitoring tool Uptime Kuma, his dashboard alerts him of any downtime and is viewable by his applications’ users. In another hackathon effort, Eklund wanted to find an easy way to diagnose his connection to Lab resources that would work properly with the Lab’s firewall. Eventually he launched the project on a production server where any employee can test their network connection (such as download and upload speeds) when onsite or from VPN.

“I love testing open-source tools. Often I don’t have the time to try them in my regular job, but during hackathons I take the opportunity to see how new software can help me accomplish my job,” explains Eklund. “Usually the effort starts and stops at the hackathon, but sometimes I end up rolling a deployment into production because of a hackathon success.”

At one hackathon, National Ignition Facility Computing (NIFC) computer scientist Joshua Senecal created a plugin for an image viewer and analysis tool. The plugin read raw data produced by the confocal microscope that inspects NIF target capsules for cleanliness, then split the data into luminance and height components and displayed these to the user. “What I didn’t realize was how important that plugin would be,” he says. “I’ve used it over the years to diagnose issues with the analysis software that processes the confocal data and to explain why the software behaves in a certain way.”

For the development team that supports LLNL’s Environmental Restoration Department, hackathon projects are likely to lead to or influence production applications. Led by Gary Laguna, who attended every hackathon until his retirement in 2020, the team has collaborated on 61 projects, 42 of which have been incorporated into the production code base or improved development operations. ASQ software developer Tyler Jackson points out, “The other 19 projects are generally where we tried a new technology and decided not to pursue it further, which has saved us several times from embarking on the wrong path.”

For instance, the team has built tools that support project management and task tracking in business processes and workflows. Most significantly, they use the Django framework for many new applications after experimenting with it at the 2014 summer hackathon. According to Jackson, the hackathon opportunity is “one of the most valuable experiences Computing offers.” He says, “Larger, day-to-day software projects have a sequence of scoped events that need to be executed. Taking the chance to carve out and explore a project that operates outside of that critical pathway is extraordinarily refreshing.”

Hacking the Hackathon

A few teams have used hackathon data to test new software or to help organizers with logistics. In 2018, Jackson and a colleague wanted to learn the data visualization tool Tableau, so they scraped and parsed the event’s historical Confluence registration pages with Python before processing the data with Excel and Tableau. Their visualizations showed hackers’ participation over time broken down by Computing division, season, year, and frequency.

LC developer Greg Lee, who has attended every hackathon, recalls, “One of the most useful hackathon projects was a script that parses the hackathon Confluence registration pages to count the expected attendees and their preferences for the meals that organizers provide. The script also generates a list of email addresses that can be used to notify attendees.” He runs the script before each hackathon and notes that it has changed over the years to accommodate organizers’ needs. “We learn something new each time,” he says.

Evolution and Adaptation

Computing divisions take turns sponsoring and organizing each hackathon during the seasonal schedule. Spring hackathons are run by volunteers from the NIFC and Enterprise Applications Services divisions. Summer hackathons, often abuzz with interns, are backed by LC and the Center for Applied Scientific Computing, while ASQ and GS-CAD organize the fall hackathons. A fourth hackathon joined the rotation in winter 2020 with sponsorship from LLNL’s Data Science Institute.

Although the COVID-19 pandemic brought many of LLNL’s onsite activities to a halt, EAS and NIFC organizers immediately switched the spring 2020 hackathon to a virtual-only format, which continued for several seasons until the first hybrid hackathon in the summer of 2021. Now hackers can work remotely or together, following the latest COVID-19 protocols. Each event’s kickoff and presentations are broadcast via a videoconferencing platform so everyone, whether onsite or off, can participate.

A key role on each organizing committee is filled by administrators from the sponsoring divisions. Jackson notes, “Planning a hackathon reminds me that the Lab only runs because of the incredible talent and skill of our administrative teams.” Senecal adds, “Based on my experience, we’d never have a successful hackathon without administrative staff support, and I’ve always been profoundly grateful for it. They probably do most of the behind-the-scenes work in pulling these events together.” Administrators handle hackathon-related logistics such as event announcements, supplies management, and meal arrangements.

For example, the summer 2022 hackathon was under a time crunch due to onsite facility availability and summer students’ schedules. LC computer scientist and two-time organizer Tammy Dahlgren states, “Katrina Trujillo and Nicole Rhoden stepped in to manage the ordering, purchasing, and delivery of two meals for the event. Given the tight timeline, it was a major relief to entrust those tasks to them.”

Hackathon…and on and on

Many hackers are repeat hackers, and the same can be said of many organizers. Hendrickson observes, “Hackathons are wildly popular not just because they allow employees to try new things and develop new skills, but also because they are so much fun!”

For Senecal, participating in hackathons and volunteering as an organizer are ways to stay involved in the developer community. He says, “I love hackathons and being part of the group that makes them happen. The atmosphere is relaxed and open, everyone is supportive, nobody judges you if you fail, and there’s a lot of interesting stuff happening.”

Saini returns because of the opportunity to think creatively with colleagues. “Others pull me in because they have awesome ideas,” he says. Springmeyer adds, “In a world where we often rush around with the weight of deadlines and looming activities, hackathons allow us to reconnect with friends and challenge ourselves while sharing food, fun, and good ideas.”

Computing will co-host the fall 2022 hackathon with LLNL’s Office of the Chief Information Officer on October 6–7. In true hackathon spirit, the event will introduce something new: Amazon Web Services will facilitate participants’ use of hands-on machine learning tools and a cloud-based 3D racing simulator.

—Holly Auten