Researchers from LLNL’s Center for Applied Scientific Computing (CASC) are among the Lab’s employees making waves in the data science community, with many prominent accolades, publications, and acceptances in 2018. Data science encompasses some of the hottest technology topics—machine learning (ML), “big data” analysis, artificial intelligence, computer vision, and more—and the Center’s computer scientists eagerly embrace the challenges of this rapidly growing domain.
Moreover, data science advancements and applications are necessary for accomplishing LLNL’s national security mission, and the Lab provides many unique, data-rich opportunities for research. The following examples provide a non-exhaustive summary of CASC’s 2018 research in this space. The Center participated in many more related publications and events—and even secured a patent—throughout the year. CASC deputy director Ana Kupresanin notes, “Our expertise in applied mathematics, discretization, and optimization is a key component of advancing ML algorithms.”
U.S. and International Recognition
In February, research on using attention models for processing electronic health records was accepted at the AAAI Conference on Artificial Intelligence held in New Orleans, LA. The “Attend and Diagnose” paper by Jayaraman Thiagarajan and colleagues at IBM Research was among the 6% of 3,800 submissions selected for an oral presentation.
In May, CASC’s work with generative adversarial networks (GAN) caught the attention of NVIDIA’s developer blog. The GAN paper was authored by Rushil Anirudh, Bhavya Kailkhura, Peer-Timo Bremer, and Thiagarajan. The team’s technique showed promising results in inverse imaging problems, robust classification, and defending adversarial attacks.
Several CASC researchers participated in the springtime International Parallel and Distributed Processing Symposium (IPDPS) in Vancouver, British Columbia. A team of Thiagarajan, Anirudh, Kailkhura, Abhinav Bhatele, Jae-Seung Yeom, Todd Gamblin, and colleagues developed an ML framework called PADDLE (Performance Analysis using a Data-Driven Learning Environment). In the PADDLE paper, they used ML techniques to model execution time and power consumption as well as characterize performance data. “ML gives us insights so we can configure code to run efficiently with minimal resources,” explains Thiagarajan.
CASC work was also accepted at the International Conference on Supercomputing in Beijing, China. Co-authored by Thiagarajan, Anirudh, Bhatele, and Gamblin, the team’s novel bootstrap scheme provides a semi-supervised ML technique that quickly identifies performance-optimizing configurations for high performance computing (HPC) systems.
Image reconstruction research was accepted at the summertime Computer Vision and Pattern Recognition Expo (CVPR) in Salt Lake City, Utah. The “Lose the Views” paper by Anirudh, Hyojin Kim, Thiagarajan, Bremer, and Engineering colleagues demonstrated a new technique that estimates the quality of reconstruction and provides a confidence score for reconstruction reliability. The paper was one of only 7% of submissions selected for a CVPR spotlight presentation.
Kupresanin, chair-elect of the American Statistical Association’s Uncertainty Quantification Interest Group, traveled to Vancouver for the American Statistical Association’s Joint Statistical Meetings (JSM) in August. The JSM is the premier annual conference for statisticians, with attendance topping 6,300. Kupresanin led a roundtable discussion on interpretable ML, where she heard from academic and industry attendees interested in the Lab’s work in statistics and related fields. “JSM participants wanted to know how the Lab thinks about ML interpretability with regard to uncertainty quantification [UQ],” she says. “The Lab is uniquely positioned to contribute to questions of ML interpretability.”
In the nation’s capital, Sookyung Kim presented at the Earth System Grid Federation’s annual conference, which addressed data analysis challenges in climate research. She discussed ML techniques for tracking and predicting extreme climate events such as hurricanes. Meanwhile, Thiagarajan represented CASC at Bay Area Scientific Computing Day, held at Sandia National Laboratories’ Livermore campus in December. His talk explored building physically consistent ML models to act as surrogates for simulations.
Additional recognition in 2018 came from the Laboratory’s newly launched Data Science Institute (DSI), which invited Bremer and Kupresanin to sit on the Data Science Council. The DSI featured Anirudh and Kailkhura in Data Scientist Spotlights, and its inaugural workshop in August included presentations and posters by more than a dozen CASC employees.
SC18 Highlights
No yearly retrospective of CASC’s contributions to HPC in general—and data science in particular—would be complete without Supercomputing (SC), held in November in Dallas, Texas. More than one-third of the Laboratory’s 100-person delegation to this high-profile HPC conference came from CASC.
“ML has become more popular in the last few years at SC,” notes Brian Van Essen, who will serve on SC19’s inaugural ML-focused technical track. “There’s more energy around it with a dramatic increase in publications.”
Most finalists for the prestigious Gordon Bell Prize, awarded every year at SC, used ML techniques in their submitted research. The conference’s “ML in HPC Environments” workshop, a collaboration between LLNL and Oak Ridge National Laboratory, has seen attendance balloon from a few dozen in 2015 to several hundred in 2018. Initially a half-day event, the workshop now requires a day and a half to accommodate this uptick in demand.
Naoya Maruyama and Van Essen co-led a “birds of a feather” session on exascale ML and, along with Andy Yoo, participated in another session on large-scale training of deep neural networks (NNs). The latter presentation showcased the LLNL-developed open-source Aluminum library, which computes while moving data across hybrid architectures. The result is a scalable network that accelerates GPU computation. Van Essen says, “Aluminum’s inter-GPU communication patterns overlap communication and computation to speed up both.”
Another deep learning (DL) workshop at SC18 included Van Essen’s keynote address on learning-based predictive models. He provided an overview of the DL techniques used in an inertial confinement fusion project, which couples ML into the traditional scientific workflow and generates massive data sets for training DL models.
Sam Jacobs and Van Essen also co-organized a workshop on ensemble learning for cancer drug discovery, demonstrating the new Livermore Tournament Fast Batch (LTFB) algorithm. LTFB is part of the open-source Livermore Big Artificial Neural Network toolkit, which includes software for training massive NNs.
“We developed a novel method that applies more parallel resources and reduces the time of training a model to convergence,” explains Van Essen. “The ensemble of NNs evolves during a ‘tournament,’ in which the strongest network is allowed to replicate and re-train the model.” The next generation of LTFB promises to extend from its original purpose in drug discovery to other scientific models and large-scale imagine processing problems.
Stiff Competition
In August, a CASC team co-authored an article with collaborators from the Indian Institute of Technology and Syracuse University. This publication in the Journal of Machine Learning Research (JMLR) addresses the space-filling problem by evaluating the frequencies, or spectra, of change in sample designs.
In November, Thiagarajan co-authored a paper on ML with Arizona State University collaborators. This work on combining two crucial areas of ML—kernel methods and DL—for multimodal problems was published in the journal IEEE Transactions on Neural Networks and Learning Systems (TNNLS). A high-impact journal, TNNLS focuses on theory, design, and applications of NNs.
CASC research also found its way into the 32nd Conference on Neural Information Processing Systems (NeurIPS, formerly NIPS) in Montréal, Quebec. Kailkhura co-authored a paper with colleagues from IBM Research and the University of Michigan about zeroth-order stochastic variance reduction. The team developed a fast, flexible gradient-free algorithm that mitigates errors in image classifiers. Kailkhura says, “Most ML algorithms rely on accessing all parameters of a model. Ours allows you to access just the queried values.” The paper showed this algorithm in practice with a black-box chemical material classification.
Thanks to these efforts, LLNL was the only national laboratory with research accepted to the highly competitive JMLR and NeurIPS. According to Kailkhura, both venues value theoretical understanding of ML algorithms beyond heuristic approaches. “These accomplishments show that we are conducting high-quality research and the Lab is recognized by the outside research community,” he notes.
More Contributions to the Literature
CASC’s publication credits expanded in 2018 in topics as diverse as facial expression recognition from labeled images; optimization of DL methods for limited-size training datasets; a Bayesian approach to multivariate adaptive regression splines; and inferring semantic relationships in natural-language processing. Additional published research advanced applications for electrocardiogram and seismic data via recurrent NN architectures and randomized ML algorithms, respectively.
“CASC’s culture encourages publishing and promoting our research at conferences,” states Kupresanin, who mentored a summer student for a Statistica Sinica publication. “Exchanging ideas is naturally part of our work, as it makes you a better researcher. The Lab’s programs benefit from this.”
Complementing CASC’s prominence in scientific journals, Kailkhura co-authored a book in 2018. Secure Networked Inference with Unreliable Data Sources (published by Springer) proposes distributed learning algorithms and inference solutions for analysis of unreliable data.
Looking Ahead
The Center’s data science work persists in the new year, with continued presence expected at many important scientific research venues in 2019. For example, CASC research on convoluted NNs has been accepted to the next IPDPS in Rio de Janeiro, Brazil. Van Essen is an invited speaker at the upcoming ICS High Performance Conference (formerly the International Supercomputing Conference) in Frankfurt, Germany.
Thiagarajan is pursuing UQ for NNs, stating, “Enabling DL models to quantify confidences about predictions will lead to the next generation of NNs.” Kupresanin also continues research in UQ, building on work that uses Bayesian splines to obtain predictive models. “We expect to insert more rigorous UQ and data science methodologies into the Lab’s stockpile stewardship mission,” she notes.
Kailkhura is working on ensuring reliability of small-data DL in materials science applications. He says, “Because of the Lab’s unique mission in scientific domains, it’s important that we deploy and sustain ML models in practice.”
Van Essen shares principal investigator duties on two multi-laboratory projects, including a new initiative under the Exascale Computing Project aimed at broadening ML’s reach across the Department of Energy complex. “CASC and the Lab play a unique role in the data science revolution because we have a mission-driven focus and apply these solutions to scientific problems,” he states. “Our highly multidisciplinary teams have a unique perspective.”
