The end of the hackathon provides serendipitous moments. Collaborations happen when others can see what you did.
Joe Ciurej
The crowd at the hackathon kickoff

Dispatches from the Fall Hackathon

Tuesday, December 18, 2018

Computation’s 2018 fall hackathon saw hearty participation and a wide range of projects despite pouring rain and a “perfect storm” of technical glitches. The 24-hour event was held at Lawrence Livermore National Laboratory’s (LLNL’s) High Performance Computing Innovation Center on November 29 and 30.

This hackathon recap is furnished by the six-person web team that manages this website ( and others (such as,, and This hackathon was our first together. As longtime followers of these thrice-yearly events, we were long overdue for participation. After all, Computation’s website always needs some type of development, upgrade, or adjustment.

The external website’s maintenance is no one person’s full-time job. We all wear different hats in Computation, with collective skill sets of front- and backend development, graphic design, communications, writing, photography, and more. This summer we undertook a significant effort to upgrade the website’s accessibility features, and our plan for the hackathon was to tackle related outstanding tasks. (Successful web accessibility means making a site useful and inclusive to all users, particularly those with disabilities. More information can be found on

Figure 1. Michael Ward (left) and Joe Ciurej, our fearless leaders, tackle hackathon organization tasks. (Photo by Randy Wong. Click to enlarge.)

Day 1

12:59pm One minute to go! The snack table is full of Cheez-Its, cookies, and fruit. Small details matter: Each table has a power strip, and chairs are arranged so team members can sit across from each other.

1:00pm And…the building’s WiFi is down. Co-organizers Joe Ciurej and Michael Ward assure us that rebooting is under way. In the meantime, our table has an iPhone hotspot and access to the Lab’s public WiFi network.

1:10pm After an issue with the projector is resolved, Katie Lewis officially kicks off the hackathon. Katie leads Computation’s Applications, Simulation, and Quality Division, which co-hosted the first hackathon way back in the summer of 2012. She urges participants to take advantage of this dedicated time.

“This is an opportunity to look around and find better ways to get the job done. Other people are working on things that might help you solve your problems, too,” she says. “Go forth and innovate.”

1:15pm We are hacking!

Well, we are trying to. WiFi and VPN access remain spotty, so we’re adjusting. Coincidentally, the Lab’s data center also has a power outage right now, probably due to the heavy rain outside. Temporarily without Jira, where we track bugs and development tasks, we’re relying on collective memory to reconstruct our To Do list.

Past hackathon participants often extoll the benefits of just being in the same room with colleagues, especially those they don’t run into every day. Already we are finding this to be true, as almost immediately we start talking to Rob Blake about a unique recruiting opportunity.

3:46pm In the last hour, our team has knocked out a few minor tasks and begun strategizing for the eventual website overhaul. We’ve optimized display quirks like the spacing on accordion menus (see Current Projects). Inspired by the earlier conversation with Rob, we’ve also made “Apply Now” buttons more prominent on the various Careers sub-pages (see Researchers). True, the latter takes us off track from our original hackathon goal, but it was easy and necessary. Why not implement now?

Figure 2. Ahn Quach (left) and Clinton Cohagan concentrate on a project using SAML— security assertion markup language. (Photo by Randy Wong. Click to enlarge.)

3:53pm The popcorn machine needs its first refill. (Fun fact: Computation owns not one but two popcorn machines.)

Computation’s external web presence is a highly customized Drupal 7 site, and it’s one of LLNL’s most visited websites. Most of our visitors use the Chrome browser on their desktops. The website logs nearly 800 pages views a day. Accordingly, one hackathon task is developing a new containerized reverse proxy caching solution to speed up page loading.


5:50pm THE TACO BAR IS HERE. Turns out the delay was one of those life-at-the-Lab situations: The catering truck came to the wrong gate and had to be redirected around the perimeter of our square-mile campus. Even if it derails their own projects, hackathon organizers must adapt to unforeseen circumstances—especially when it comes to feeding the hackers.

6:02pm  Ian Lee approaches our table with a full taco plate. “I have a news article idea for you,” he says, introducing Mark Miller. Sneak preview: Big news is coming in January for the open-source tool VisIt. Stay tuned to this website for more!

Mark’s news may have eventually made its way to us. But spontaneous, convenient interaction allows us to quickly gather pertinent details face to face—a first-time introduction in this case—and establish a timeline for a follow-up interview.

6:31pm Mike Goldman, director of LLNL’s Data Science Institute (DSI), stops by to follow up on a conversation from earlier in the week. We intend to add a new page to the DSI website that provides access to public data sets, and the data curation effort is farther along than it was a few days ago. We kick around a couple of ideas for the future page’s template as well as a maintenance plan.

It’s another example of off-topic, off-the-cuff progress made possible by proximity and opportunity. Relationship building and open communication are hallmarks of every hackathon, as we now see for ourselves.

Figure 3. Kevin Athey gets comfortable building a simple infrastructure to store and track video files. (Photo by Randy Wong. Click to enlarge.)

6:45pm We deploy today’s tested changes to production.

7:26pm About 20 people are still here working and chatting. Cookie supply is down about 75 percent. The rain has finally stopped.

8:15pm With one exception, our team is packing up for the night. We’ll be back bright and early, coffee in hand.

Day 2

8:29am Reliable WiFi today. The room is filling. Now that Jira is back up, it’s time to take stock of our sprint tasks. We’re more than halfway done with what we’d planned to accomplish, but we don’t expect to finish everything.

Major hackathon hazard: We keep coming up with new ideas and therefore new tasks!

9:05am Some of the changes we’re implementing are visible to our users, such as unique links on the home page. Multiple links phrased as “Read More” will confuse visually impaired users who rely on screen readers for site navigation, so these links are now differentiated—e.g., “Read More about Software” and “Read More about People.” Other changes are behind the scenes, making our Drupal instance more effective for our workflow and more adaptable to future accessibility enhancements.

10:17am We update the rollover color on tertiary menus, update offsite links on a page that describes parallel software development tools, and replace a handful of leaderboard images.

Michael is smiling even as co-organizer duties divert his attention. He’s new to LLNL but nevertheless jumped into the hackathon with both feet. “Compared to managing robotics competitions for teenagers, this is easy,” he states. His background is in geospatial analysis, graph theory, and image processing using machine learning. Today he’s been working on deep learning and neural network techniques to potentially apply to his “day job” in the Global Security Computing Applications Division.

10:40am We check in with co-organizer Joe, who is expanding on another team’s work from a previous hackathon. He explains, “The idea is to produce a Beamer template that makes it easy to generate a presentation. If it’s easy, hopefully more people will do it.”

Presentations are about more than just making teams accountable for their hackathon time. “The end of the hackathon provides serendipitous moments. Collaborations happen when others can see what you did,” says Joe.

Figure 4. Walt Nissen is a ubiquitous presence at hackathons. (Photo by Randy Wong. Click to enlarge.)

11:19am Pizza arrives. Our production website now includes several redirects from old pages to new URLs.

1:05pm Many teams are gearing up for presentations. Participation in this culminating session—showing others what you’ve accomplished or where you failed—has waned in recent hackathons, so now there are incentives. In addition to the handy template Joe completed, presenters receive an LLNL-branded hackathon sticker inspired by Gary Laguna and designed by Michael.

1:30pm Presentations begin. Everyone has five minutes to explain what they attempted, how far they got, and any lessons learned.

1:45pm We’ve got just enough time for another push to production with changes validated since yesterday.

1:46pm Cyrus Harrison talks about using more CPUs to conduct visualization analysis on LLNL’s Sierra supercomputer. His hackathon project follows a demo he gave at the recent SC18 conference, and today he shows a small performance increase while running in-situ visualizations in parallel with physics codes.

1:59pm Our turn to present! Before we know it, Joe’s giving the one-minute signal.

2:11pm Keith Healy speaks on behalf of a hackathon team that explored using Jupyter Notebooks for interactive software performance analysis. He says the idea came from an earlier hackathon project led by Walt Nissen. Today’s demo shows a plot-on-request feature that allows an analyst to choose the desired performance test, platform, and compiler to generate visualizations of what they want, saving time and avoiding headaches.

One team describes a project management tool they built using Kendo, Django, and other open-source software. Another team shows a text-to-speech machine learning prototype. Machine learning crops up again in a presentation about optimizing job scheduling on the Lab’s high performance computing systems.

Figure 5. Ilona Beltsar presents her team’s improvements to the Windchill context utility. (Photo by Randy Wong. Click to enlarge.)

3:12pm After 18 presentations, Cyrus returns to the podium for closing remarks. He recalls his own hackathon participation since 2013, giving an example of a real product born during that year’s event.

Michael reminds everyone to fill out the participation survey, which provides crucial information to the next hackathon’s organizing committee. “Survey data helped us make the taco decision,” he notes.

He and Joe also acknowledge Computation’s administrative team who helped with hackathon logistics: Nicole Armbruster, Jamie Goodale, Florann Mahler, Leanne Randall, and Katrina Trujillo.

3:18pm That’s a wrap! We’ll see you at the spring 2019 hackathon.