Sponsored by the Computing Directorate, Developer Day is a chance for LLNL software developers to share ideas, learn about the latest technology, discuss best practices, and mingle with colleagues. During the COVID-19 pandemic, however, much of the Lab’s workforce is working remotely. On July 30, Computing’s fourth annual Dev Day was held virtually for the first time with more than 90 attendees—almost as many as attended last year in person. Organizers were Kyle Dickerson, Elsa Gonsiorowski, and David Beckingsale.

Dickerson stated, “Dev Day was always envisioned explicitly as an in-person event with the goals of not just seeing the presentations but also fostering the kind of serendipitous conversations that happen when people have a chance to chat between sessions.”

Nevertheless, the team decided to continue the annual event while adhering to restrictions, and the original date was delayed to accommodate adjustments. They organized presentations into two 50-minute deep dives and six 20-minute quick takes broadcast over WebEx. Gonsiorowski added, “We shrunk the formal program so that our audience would not need to attend a video call for the entire day.” This year’s agenda reflects both the event’s adaptation and the pandemic itself.

COVID-19 Research Meets DevOps

To help accelerate discovery of therapeutic antibodies and antiviral drugs for COVID-19, LLNL launched a public data portal in May. The queryable portal contains molecular designs, antibody interactions, chemical structures, and other related data. Behind the scenes, Livermore Computing’s (LC’s) Workflow Enablement Group (WEG) closely collaborated with bioinformatics experts to design and deploy the portal in just two weeks.

WEG web architect Thomas Mendoza took Dev Day attendees on a deep dive into the process of standing up the CoViewer web application, which provides the user interface for the COVID-related dataset. When the Lab’s bioinformatics team began working on a large dataset for public use, they knew they would need help leveraging LC’s Green Data Oasis (GDO)—the unclassified network used for sharing data with external collaborators. Mendoza pointed out, “The WEG’s expertise includes systems, containers, security, and web. We fit the bill for the deployment.”

As the project was under a significant time crunch, the WEG team prioritized features of the app and environment. Provisioning space on the GDO meant understanding data requirements, preparing firewall changes, and creating a development environment. Then came close scrutiny of security details, including all instances where user data is handled. “The app captures some amount of user input when a user queries, filters, or otherwise interacts with the data,” explained Mendoza. The WEG team also accounted for rate limiting, logging, monitoring, and cyber security scans.

Performance considerations were another major component of CoViewer development, particularly avoiding denial of service. MySQL query caching quickly became overwhelmed, so the team switched to ProxySQL running in an intermediate container and added Nginx as well. Mendoza said, “We went cache crazy. Wherever we could turn it on, we did.” The WEG also used LC’s GitLab instance for continuous integration and continuous deployment (CI/CD), which allowed the bioinformaticists to test the CoViewer app when updates were deployed.

The project continues, though Mendoza noted, “It’s a very low-touch system for us now. We can step in when we need to. Because we used containers, we can easily revert a major change if necessary.” The WEG tracks issues through GitLab and, with most staff working remotely, communicates with the scientific team in Mattermost.

Mendoza’s deep dive inspired Dickerson, who stated, “I enjoyed hearing about the CI/CD strategy used by the CoViewer project to rapidly get that application out to the public. I’d like to explore that process to speed up project development and reduce the overhead of deploying new versions.”

Figure 1. The Django-based CoViewer app lets users explore a large, read-only dataset of COVID-19 research. The WEG team assembled an architecture that includes Docker containers, the Gunicorn web server gateway interface, and multiple caching solutions.

Better Planning for Better Projects

In Dev Day’s second deep dive, software developer Timothy Nilsen explained the concept of cross cutting concerns (CCCs)—system-wide functions that affect multiple areas of development and do not fit well into object-oriented programming, such as configuration, monitoring, logging, security, and synchronization. “These types of concerns impact a project’s performance, maintainability, extensibility, testing, debugging, code re-use, and more,” he said. For instance, instantiating a particular class multiple unnecessary times can lead to poor performance.

Nilsen’s work in LLNL’s Global Security Computing Applications Division inspired his deep dive, which was his first experience with Dev Day as well as his first online presentation. “Almost all of the applications I design or maintain have no less than three CCCs that need to be accounted for,” he explained. He has seen “patterns of problems” such as limitations to a security implementation because of how a program was designed.

When Nilsen had the opportunity to design a new application, he figured out best practices for CCCs. “I knew what they were but never had a name for them until I researched more,” he noted. “Now, fortunately, all of my projects start with a design phase where we identify CCCs. The process has saved time and makes working with teams and peers much easier.”

Nilsen offered tips for managing CCCs, emphasizing their identification early in development. “A few extra minutes in the design phase will save much more time later,” he stated. Additional advice includes using abstractions where possible, determining the most important aspects of the application under development, planning for changes as technology is abandoned or updated, and selecting an appropriate design pattern. For example, a service registry design pattern can provide multiple services of the same type but adds complexity to the program.

Ultimately, approaching software development with CCCs in mind will pay off in the long run. “As a developer, I know that programming is the fun part. Design, documentation, and planning can feel like drudgery. But seeing the benefits of planning goes a long way toward convincing people it’s the right path,” Nilsen said. Gonsiorowski added, “This deep dive brought a formalism to problems I’ve faced before. It’s a really cool concept that I want to learn more about.”

Figure 2. CCCs affect multiple layers of a software project, which makes them susceptible to errors, unintended consequences, or re-work.

Lessons from Literature and Life

Dev Day’s quick takes are shorter talks in which a developer reviews and opines on a news article or book chapter relevant to software development, workplace culture, personnel management, or related technology. Some topics are technical while others are less so. “We maintain a list of reading material that we think is valuable for software developers,” said Dickerson. “We try to select themes of particular importance within the Lab, but we expect they’d be valuable to developers regardless of their background.”

During the morning session, Tyler Jackson of Computing’s Applications, Simulations, and Quality Division presented the concepts of “old guard” and “new guard” in corporate culture, including tips for finding shared values and opportunities for collaboration. He noted the timeliness of the topic as LLNL manages a retiring generation alongside longer tenured employees, recent graduates, and student hires.

The afternoon’s quick takes included Trevor Smith’s discussion of choosing mature versus new technology. Smith, who works in LC, noted that every new solution added to a software stack increases the overall cost of maintenance. Mature technologies can offer advantages such as larger developer communities, shared costs, and known failure models.

In another quick take, Gonsiorowski shared development practices, tools, and processes for a remote workforce. She conducted a survey of Dev Day attendees prior to the event, and found themes in the responses around home office space, scheduling issues, and social impacts. For instance, most survey respondents are managing a work-from-home schedule with both rigid and flexible aspects. Gonsiorowski noted, “Keep trying until you find out what’s successful for you. There’s no right answer.”

The survey also gauged developers’ feelings on maintaining focus, workspace setups, and colleague interactions in this new era of maximum telecommuting. Summarizing respondents’ advice as well as her own experience, Gonsiorowski offered tips for productive work routines, team building, group conversations, and more. She explained, “I’ve gotten in the habit of scheduling phone calls with folks whose offices I would normally swing by during the day. I also want to try pair programming over WebEx.” She reminded the audience to take advantage of the Lab’s ergonomic evaluation program and other employee resources, such the new “Lunch and Learn” series focused on working safely during the pandemic.

Figure 3. A survey of Dev Day attendees revealed a range of challenges employees face while working from home. One of the presentations offered tips for managing productivity and stress while telecommuting.

A Virtual Success

Although this year’s Dev Day could not provide invaluable in-person networking or enticing snacks, the organizers consider it a success. Beckingsale noted that the compressed program helped attendees stay engaged, and the event’s timing was convenient for summer interns. “People had a long lunch break and time in the afternoon to catch up on other work,” he said. Gonsiorowski agreed, “I think the limited schedule worked out really well. Perhaps next year we can keep the formal program limited with plenty of time for informal networking.” Dickerson added, “Thankfully, we had no technical difficulties in the virtual setting.”

Read more about the Lab’s efforts to research and respond to COVID-19.