2018 Blue Waters Symposium: Abstracts and Presentations

 

A   B   C   D   E   F   G   H   I   J   K   L   M   N   O   P   Q   R   S   T   U   V   W   X   Y   Z

 

 

 

Molecular Dynamics of DNA Origami Nanostructures

Project PI: Aleksei Aksimentiev, University of Illinois at Urbana-Champaign

Presented by: Christopher Maffeo, University of Illinois at Urbana-Champaign

Abstract: DNA origami is an experimental technique that allows folding of a long DNA molecule into an arbitrary three-dimensional shape with sub-nanometer precision. In comparison to conventional nanomanufacturing, the DNA origami method has relatively low cost, is easy to use and has an infinite number of possible applications. Over the past several years, the Aksimentiev group at the University of Illinois has introduced the benefits of high-performance computing to the DNA origami field. Through a combination of continuum, coarse-grained and fully atomistic computational approaches, our group has characterized a range of landmark DNA origami systems, many of which have been subsequently realized in experiment. Using Blue Waters, our group has achieved an unprecedented accuracy in prediction of the in situ shapes of complex DNA origami objects and characterization of their physical properties. Here we highlight recent accomplishments generated by the Blue Waters project.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Coarse-Grained Force Field Development of Room Temperature Ionic Liquids

Project PI: N. R. Aluru, University of Illinois at Urbana-Champaign

Presented by: A. Moradzadeh, University of Illinois at Urbana-Champaign

Abstract: Room temperature ionic liquids are an emerging class of solvents, which are mostly composed of bulky organic cations and inorganic anions. They have many properties distinguishing them from conventional solvents and molten salts, especially their heterogeneous and long-range structure. Many applications are emerging as a result of these properties including energy-storage, electrotunable nano-lubrication, and gas capture. However, there are many unanswered questions regarding their macro- and micro-scale properties. Molecular dynamics simulations have been used to understand their behavior in nano-scale systems. However, they are computationally expensive for large time- and size-scale. In this study, we developed coarse-grained force fields for imidazolium-based ionic liquids, which reproduce the thermodynamic (density and pressure) and structural properties of reference all-atom models for different alkyl chain length at different thermodynamic states. The coarse-grained force field is developed using the relative entropy method to obtain short-range van der Walls and long-range Coulombic interactions with an additional constraint in order to reproduce pressure. The method developed is generalizable to various ionic liquid classes, and it can pave the way for accurate study of large size- and time-scale systems, which, in turn, advances our fundamental understanding of ionic liquids behavior as solvents of future.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

 

Two-Fluid Turbulent Dynamo Simulations on the Blue Waters System

Project PI: Dinshaw Balsara, University of Notre Dame

Abstract: In light of NASA's investments in SOFIA as well as NSF's investments in ALMA, and other observational programs, it has become very interesting to study magnetic field evolution in turbulent two-fluid plasmas. Such plasmas consist of a partially ionized fluid that is strongly coupled to the magnetic field, along with a much denser neutral fluid that couples to the ionized fluid via ion-neutral drag. Understanding magnetized turbulence as well as the generation of magnetic fields via a turbulent dynamo is very interesting in such environments. The interest stems from two major considerations. Firstly, it is interesting because the magnetic field in such plasmas strongly regulates the formation of stars in molecular clouds. (If the magnetic field were frozen-in, star formation in galaxies would become impossible.) Secondly, it is interesting because the various above-mentioned investments in instrumentation ensure that we are poised to measure such magnetic fields.

We present several results, including first-ever results on the growth of magnetic fields in a partially ionized plasma via turbulent dynamos. We demonstrate that the simulations match with theory, suggesting a very nice verification between theory and simulations. These results would not have been possible without the compute power of Blue Waters.

As a bonus, we will also show very recent work that overcomes an age-old stumbling block in computational astrophysics. Many astrophysical systems are spherical, however, the spherical meshing that is used for astrophysical simulations is prone to bottlenecks owing to meshing singularities. Using our recent advances in highly-parallel, geodesically meshed, divergence-free MHD, we show that this age-old stumbling block has now been removed!

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Petaflops Simulation and Design of Nanoscale Materials and Devices

Project PI: Jerzy Bernholc, North Carolina State University

Abstract: We describe large-scale ab initio simulations of nanoscale materials and devices, which can predict the best-performing systems. We focus on graphene nanoribbon (GNR) structures and devices. GNRs have very rich electronic properties that depend on their widths and edge types. We first determine the mechanism of growth of atomically precise GNRs and also consider incomplete conversion, which results in an intermediate state consisting of graphene structure on one edge and polymer structure on the other edge. Using a DFT-based non-equilibrium Green's function method implemented within the highly parallel real-space multigrid (RMG) code suite, we design a novel experimentally realizable GNR-based negative differential resistance (NDR) device based on GNR/intermediate-structure heterojunctions and uncover the general principles governing the design of NDR nanoscale devices. RMG is installed on Blue Waters as a community code. New versions are regularly released at www.rmgdft.org, including binaries for Linux, Windows and MacIntosh systems.

Field of Science: Materials Science

Watch the presentation | View the presentation slides (PDF)

The Role of Cosmic Ray Transport in Shaping the Simulated Circumgalactic Medium

Project PI: Iryna Butsky, University of Washington

Abstract: The majority of galactic baryons reside outside of the galactic disk in the diffuse gas known as the circumgalactic medium (CGM). While state-of-the art simulations excel at reproducing galactic disk properties, many struggle to drive strong galactic winds or to match the observed multiphase structure of the CGM with thermal supernova feedback. To remedy this, recent studies have included non-thermal cosmic ray (CR) stellar feedback prescriptions to drive strong outflows and to better match observed low-ion column densities in the CGM. However promising, these results depend strongly on the choice of CR transport mechanism and its constant parameter values, thus weakening the predictive power of such simulations. Using a suite of simulated isolated disk galaxies, I will demonstrate that the invoked approximation of CR transport affects the predicted temperature and ionization structure of the CGM.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Policy Responses to Climate Change in a Dynamic Stochastic Economy

Project PI: Yongyang Cai, The Ohio State University

Abstract: We build a novel stochastic dynamic regional integrated assessment model (IAM) of the climate and economic system including a number of important climate science elements that are missing in most IAMs. These elements are spatial heat transport from the Equator to the Poles, sea level rise, permafrost thaw and tipping points. We study optimal policies under cooperation and various degrees of competition between regions. We solve the problems over a 500-year horizon, and can do this because we employ efficient parallel dynamic programming methods with the support of Blue Waters. Our results suggest that when the elements of climate science which are accounted for in this paper are ignored, important policy variables such as the social cost of carbon and adaptation could be seriously biased.

Field of Science: Social Sciences

Watch the presentation | View the presentation slides (PDF)

Atomistic Modeling of Biomedical Targets: TRP Channels as Cellular Sensors of Pain

Project PI: Vincenzo Carnevale, Temple University

Presented by: Eleonora Gianti, Temple University

Abstract: Recent progresses in atomistic modeling, combined with groundbreaking advances in high-performance computing, have pushed our understanding of the structure and function of complex biomedical targets to the next level, thereby offering crucial approaches to succeed in drug discovery.

Nociceptive Transient Receptor Potential (TRP) channels are responsible for conducting cations through the cell membrane in response to a variety of stimuli, i.e. the binding endogenous lipids, environmental toxins and agonist or antagonist chemical compounds. Predominantly expressed in sensory neurons, TRPs are involved in various cellular functions, such as sensitivity to chemical irritants, temperature detection and, importantly, nociception and pain sensation.

We combine atomistic simulation performed on Blue Waters with computer-aided drug discovery approaches to exploit TRP channels as pharmacological targets in treating pain. Our results provide new perspectives for understanding the principles governing TRP channels activation by multiple stimuli, and pave the way to the discovery of novel pain medications.

Field of Science: Biosciences

Using Ensembles of Molecular Dynamics Simulations to Give Insight into Biomolecular Structure, Dynamics and Function

Project PI: Thomas Cheatham, University of Utah

Abstract: The coupling together of ensembles of independent AMBER molecular dynamics simulations on GPUs has provided a direct means to give detailed atomistic insight into varied biomolecular structure, dynamics and function. Over the past few years utilizing the Blue Waters Petascale resource, the Cheatham lab has been able to fully converge the conformational ensemble of varied biomolecular systems ranging from di- and tetra-nucleotides to RNA tetraloops. This has been useful for force field assessment and improvement. Recent results elucidate properties of stapled peptides, Ebola virus fusion inhibitor peptides, and varied nucleic acid systems.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Utilizing Massively-Parallel Computations to Predict Nanopatterning of Graphene by Hydrogen-Plasma Etching

Project PI: Huck Beng Chew, University of Illinois at Urbana-Champaign

Abstract: Scalable and precise nanopatterning of graphene is an essential step for graphene-based device fabrication. Hydrogen-plasma reactions have been shown to narrow graphene from the edges, or to selectively produce circular and hexagonal holes in the basal plane of graphene, but the underlying plasma-graphene chemistry is unknown. The peta-scale Blue Waters supercomputing resources has enabled us to quantify the mechanisms of hydrogen-plasma etching of graphene supported on SiO2 substrate across the range of plasma ion energies. Specifically, our molecular dynamics simulation results, based on a reactive force-field potential, have uncover distinct etching mechanisms, operative within narrow ion energy windows, which explain the differing plasma-graphene reactions observed experimentally. These simulation results have provided rich insights into the complex plasma-graphene chemistry, opening up a means for controlled patterning of graphene.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

The Wellspring of all Phases on the Quantum Kagome Antiferromagnet

Project PI: Bryan Clark, University of Illinois at Urbana-Champaign

Abstract: Frustrated quantum magnets are a central theme in condensed matter physics due to the richness of their phase diagrams. They support a panoply of phases including various ordered states and topological phases. In this talk we explain the source of this richness. We report on the discovery of a Hamiltonian which has an exponentially degenerate ground state. We discuss how we use Blue Waters to map out the phase diagram surrounding this exponentially degenerate point and show how it appears to source many of the phases on the kagome lattice including the enigmatic spin-liquid.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

Saving the Inner Solar System with an Early Instability

Project PI: Matt Clement, University of Oklahoma

Abstract: The solar system is in an interesting state where the planet's orbits behave chaotically over long periods of time. Studying the dynamical evolution of the solar system can help us understand whether other planets like Earth exist in the galaxy. An orbital instability between the solar system's giant planets has been shown to reproduce many peculiar qualities of the solar system. We present new simulations of this scenario that demonstrate its ability to accurately reproduce the eccentricity, inclination and resonant structures of the Asteroid Belt. Furthermore, we perform simulations using an integration scheme which accounts for the fragmentation of colliding bodies. The final systems of terrestrial planets formed in these simulations provide a better match to the actual planets' compact mass distribution and dynamically unexcited orbits. An early instability scenario is thus very successful at simultaneously replicating the dynamical state of both the inner and outer solar system.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Estimating Bed Shear Stress Distribution from Numerically Modeled Tides and Wind Waves on Estuarine Mudflats

Project PI: Salme Cook, University of New Hampshire

Abstract: Estuaries are dynamic coastal environments where upland freshwater flows mix with tidally forced coastal marine water to create a highly productive and diverse ecosystem important to local communities and fisheries. Increases in population density and associated anthropogenic impacts have altered the productivity of estuarine environments, resulting in increased nutrient loading and amplified suspended sediment that reduces water quality. Mudflats and salt marshes comprise the intertidal portions of these environments and are the most productive and vulnerable to these stressors. To accurately predict sediment transport, a good understanding of the bed shear stress that drives the sediment erosion, suspension and deposition is essential. In this work, a high resolution three-dimensional coupled hydrodynamic-wave-sediment transport numerical model (COAWST) was implemented in the Great Bay estuary, and is used in conjunction with available observational datasets to predict the shear stress distribution from waves, currents, and combined flows from the tidal channels across the mudflat.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Massive Galaxies and Black Holes at the Cosmic Dawn

Project PI: Tiziana DiMatteo, Carnegie Melon University

Abstract: The first billion years is a pivotal time for cosmic structure formation. The galaxies and black holes that form then shape and influence all future generations of stars and black holes. Understanding and detecting the the first galaxies and black holes is therefore one of the main observational and theoretical challenges in galaxy formation.

I will discuss predictions for the first supermassive black holes and their host galaxies in the BlueTides simulation. BlueTides is a uniquely large volume and high resolution simulation which resolves the scales relevant for the high-z universe which will be investigated with the next generation telescopes (Euclid, JWST and WFIRST).

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Bridging Molecular Timescales with MELD and Blue Waters

Project PI: Ken Dill, Stony Brook University

Presented by: Alberto Perez, Stony Brook University

Abstract: The sheer amount of possible conformations molecules can adopt is beyond enumeration even with the most advanced computation techniques. But, they are governed by the laws of physics. We have developed an advanced sampling technique called MELD based on Hamiltonian and Temperature replica exchange molecular dynamics to access biological processes beyond what traditional techniques can do. We have successfully folded several proteins in a blind competition called CASP (Critical Assessment of Structure Prediction) with strict time limits and the help of Blue Waters supercomputer.

We have furthermore used MELD states and unbiased simulations to construct Markov State Models that tell us about kinetics and folding pathways. This highly parallelizable approach combined with Blue Waters resources allows us to reconstruct at an atomic level of resolution the pathways in a couple of weeks of human time — a task that would require about a year of long molecular dynamics simulations.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Advancing First-principle Symmetry-guided Nuclear Modeling for Studies of Nucleosynthesis and Fundamental Symmetries in Nature

Project PI: Jerry Draayer, Louisiana State University

Presented by: Tomas Dytrych, Louisiana State University

Abstract: We use the Blue Waters system to carry out large-scale first-principle modeling of light and medium-mass nuclei, including short-lived isotopes not yet accessible to experiment but key to understanding astrophysical processes that shape our universe and which are the focus of current and next-generation rare isotope experimental facilities. In our studies, we utilize an innovative theoretical framework for first-principle modeling of nuclear structure, dubbed symmetry-adapted no-core shell model, which is implemented as a hybrid MPI/OpenMP parallel computer code that scales well for hundreds of thousands of processors. The extremely large memory and computational power of the Blue Waters system allows us to model the intricate dynamics of atomic nuclei with a precision hitherto inaccessible to theory, thereby addressing some of the long-lasting challenges of importance to nuclear theory and experiment, as well as astrophysics.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

Direct Numerical Simulation of Pressure Fluctuations Induced by Supersonic Turbulent Boundary Layers

Project PI: Lian Duan, Missouri University of Science and Technology

Abstract: Understanding the physics of the pressure fluctuations induced by turbulent boundary layers is of major theoretical and practical importance. From a practical point of view, an in-depth knowledge of the nature of boundary-layer-induced pressure fluctuations is essential to the structural design of launch vehicles and to the generation of background noise in wind-tunnel facilities. From a theoretical point of view, pressure fluctuations are an important ingredient in turbulence as they appear in statistical correlations such as the pressure-strain correlation terms that redistribute turbulence among different components of fluctuating velocity.

In the presentation, I will talk about our use of the Blue Waters supercomputer to conduct direct numerical simulations that provide the basis for an in-depth understanding of the global pressure field induced by turbulent boundary layers at supersonic speeds. I will also discuss about our experience of using parallel HDF5 for efficient I/O and our effort to maximize node-level parallelism on the Blue Waters.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

 

AI for Drug Discovery in Two Stories

Project PI: Evan Feinberg, Stanford University

Abstract: It costs $1.2 billion to develop a single FDA-approved drug. That pricetag is rapidly rising. Can Artificial Intelligence (AI) live up to the hype it has rightly garnered in other fields and reverse the seemingly inexorable trend of "Eroom's law" in developing new medicines? In this talk, I discuss two papers that tell two distinct stories about AI drug discovery.

1. "Spatial Graph Convolutions for Drug Discovery" describes new deep neural network architectures for modeling drug-receptor interactions. We argue that the future of predicting the interactions between a drug and its prospective target demands more than simply applying deep learning algorithms from other domains, like vision and natural language, to molecules. We used Blue Waters extensively to search for hyperparameters in training our deep neural networks with PyTorch code as optimally installed with the bwpy module.
https://arxiv.org/abs/1803.04465

2. "Machine Learning Harnesses Molecular Dynamics to Discover New μ Opioid Chemotypes" describes an algorithm that leverages protein motion to enrich the search for active molecules. We then applied the method to find a new chemical scaffold that we experimentally verified is an agonist for the μ Opioid Receptor.
https://arxiv.org/abs/1803.04479

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

When Does Uncertainty Matter While Modeling Climate Change in Mountain Headwaters? Contrasting Model Resolution and Complexity Under a Changing Climate in an Alpine Catchment.

Project PI: Lauren Foster, Colorado School of Mines

Abstract: Alpine, snowmelt-dominated catchments are the source of water for more than 1/6th of the world's population. These catchments are topographically complex, leading to nonlinear relationships between water and energy fluxes. Despite being more sensitive to climate warming, mountain headwaters are vastly simplified in climate and water management models. It is critical to determine whether projected climate impacts are robust across typical model resolutions. We find that higher-resolution models predicted larger reductions to snowpack, surface water, and groundwater stores per degree of temperature increase, suggesting that warming signals may be underestimated in simple models. These experiments help to (1) bracket the range of uncertainty in published literature of climate change impacts on headwater hydrology; (2) characterize the role of precipitation and temperature changes on water supply for snowmelt-dominated downstream basins; and (3) identify changes to climate impacts at different scales of simulation.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Modeling and Simulation of Complex Musculoskeletal Architectures

Project PI: Mattia Gazzola, University of Illinois at Urbana-Champaign

Abstract: Musculoskeletal systems consist of bones, muscles, tendons, ligaments and other connective tissues that altogether provide function and structure to natural creatures. One of the most intriguing aspects of these architectures is the often inseparable nexus between actuation and control, topology and mechanics, due to the continuum, non-linear nature of their constitutive elements. As a consequence, and in stark contrast with rigid bodied robots, soft creatures can harness a wide range of deformations and structural instabilities to effectively cope with complex, unstructured and dynamic environments. Here we present a novel approach to model, simulate and design in-silico creatures and compare their performance with biological and robotic replicas.

Field of Science: Engineering

Faintest Galaxies in the JWST Era

Project PI: Nickolay Gnedin, University of Chicago

Abstract: Cosmic reionization - ionization of the bulk of cosmic gas by ultraviolet radiation from first galaxies and quasars - is the least explored epoch in cosmic history. While significant progress has been made recently, the launch of James Webb Space Telescope (JWST) in two years will start nothing short of a revolution in this field. As observers are preparing for the flood of new data, Blue Waters plays a leading role in ensuring that theoretical models stay on par with future observations.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Electronic Structure Characterization of Oxygen-Evolving Electrocatalyst NiFe Oxyhydroxide Active Sites

Project PI: Zachary K. Goldsmith, Yale University

Abstract: NiFe oxyhydroxide is a highly active electrocatalyst for the oxygen evolution reaction, a crucial process for carbon-neutral energy storage. Previously, quantum chemical calculations and spectroelectrochemistry have well characterized the steady-state electronic structure of the material under working potentials.1 However, recent experiments have proven the particularly high activity of Fe dopants at the edges of the material2 and the highly valent nature of such Fe centers.3 This study applies a rigorous electronic structure characterization previously performed for the periodic interior of the thin film material to a model system that includes the catalytically active film terminations. Density functional theory (DFT) calculations with a hybrid functional were performed for Ni-only and NiFe oxyhydroxide nanowire supercells in which the periodicity is broken in one in-plane direction such that the systems contain different interior and exterior metal sites. Different degrees of protonation and hydroxylation of edge Ni and Fe sites were probed as a proxy for the oxygen evolution reaction progress. Metal oxidation states were determined using maximally-localized Wannier functions and the catalytic activity of given sites was determined qualitatively by the projected density of states around the Fermi energy. This work is enabled by Blue Waters given the considerable computational expense yet high level of parallelizability of hybrid functional DFT calculations for large, asymmetric systems with broken periodicity

Field of Science: Chemistry

Watch the presentation | View the presentation slides (PDF)

Forecasting Crop Productivity with High-Resolution Satellite Data: Scaling Up to the Whole US Corn Belt

Project PI: Kaiyu Guan, University of Illinois at Urbana-Champaign

Presented by: Sibo Wang, University of Illinois at Urbana-Champaign

Abstract: High-performance computing, along with satellite datasets that provide a wide spatiotemporal coverage of agricultural lands, enables a novel and data-oriented approach to understand crop growth. Using Blue Waters, we developed a hybrid crop model (CLM-APSIM) that produces reliable long-term yield predictions in the US Corn Belt (Peng et al., 2018). We also developed a generic multi-sensor fusion algorithm, STAIR, that integrates satellite images at a variety of spatial and temporal resolutions (Luo et al., 2018). Finally, a pixel-level crop yield model, ASPIRE, uses remotely sensed images, soil condition, and climate data to predict crop yield at field level (Xu et al. 2018). The ultimate goal of our project is to improve our predictability skill for global crop yield modeling by integrating site measurements, satellite observations, and process-based modeling.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

Magnetic Field Amplification in Binary Neutron Star Merger Remnants

Project PI: Roland Haas, University of Illinois at Urbana-Champaign

Presented by: Phillip Moesta, University of California at Berkeley

Abstract: Binary neutron-star mergers belong to the most energetic transients in the universe. They are key targets for time-domain astronomy surveys and advanced LIGO in the era of multimessenger observations. Magnetic fields play a crucial role in the merger and postmerger and are the key ingredient to determine the lifetime of the hypermassive neutron star remnant, the composition of ejected material, and the formation of a sGRB engine. I will discuss the unique challenges in both input physics and computational modeling for these systems involving all four fundamental forces and present results from recent breakthrough 3D simulations performed on BlueWaters.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Refining the Conformational Ensembles of Flexible Proteins using Multimodal Spectroscopic Data

Project PI: Jennifer Hays, University of Virginia

Abstract: Flexible recognition is a common paradigm in infection; many pathogens have proteins that are structurally and mutationally flexible but still bind human cells. Determining the structural basis of this recognition is computationally difficult as the receptor-bound state can comprise many different structures. Current methods that incorporate experimental data into MD simulations can refine structures of rigid proteins but cannot capture large backbone changes of dynamic proteins, especially for complex experimental data. We have developed a methodology that leverages petascale computing by simulating many ensemble members restrained by complex spectroscopic data in a coupled fashion to refine conformational ensembles of flexible proteins

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Next-generation Galaxy Formation Simulations with FIRE

Project PI: Philip Hopkins, California Institute of Technology

Presented by: Shea Garrison-Kimmel, California Institute of Technology

Abstract: I present recent results from the FIRE (Feedback in Realistic Environments) project, which focuses on simulating galactic evolution in cosmological contexts with physically-motivated prescriptions for the interactions between gas and stars on parsec scales. The simulations, which are presently the highest resolution of their kind in the field, therefore trace the evolution of structure for thirteen billion years while resolving a dynamic range of over a million in length. I show that the FIRE prescriptions self-consistently reproduce galaxy properties over many orders of magnitude in stellar mass. Thus, the FIRE simulations suggest that the usual "small-scale" problems identified in gravity-only simulations within the standard paradigm are explained by known, standard model physics. I additionally present preliminary results from the in-progress "Triple Latte" run, which will provide predictions for the number and properties of the least luminous and most dark matter-dominated galaxies in the Universe, ultra-faint dwarfs, by resolving the Milky Way with over a billion particles.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Multimessenger Astronomy at the University of Illinois

Project PI: Eliu Huerta, University of Illinois at Urbana-Champaign

Presented by: Roland Haas, University of Illinois at Urbana-Champaign

Abstract: We present highlights of NCSA Gravity Group's trans-disciplinary program to fuse the computational power of the Blue Waters supercomputer with innovative applications of machine and deep learning to create scenarios for multi-messenger astronomy. We describe the construction of a novel computational framework that combined for the first time the Blue Waters supercomputer, containers and the Open Science Grid to validate the first detection of colliding neutron stars in gravitational waves and light; the use of numerical relativity simulations to model the formation and merger of black hole binaries in dense stellar environments; and pioneering work at the interface of HPC and artificial intelligence to disrupt the paradigm of gravitational wave data analysis. Finally, we present a new open source package to monitor and post-process large-scale numerical relativity campaigns to validate the astrophysical origin of gravitational wave discoveries with Blue Waters.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Computational Tools for Advanced Molten Salt Reactors simulation

Project PI: Kathryn D. Huff, University of Illinois at Urbana-Champaign

Presented by: Andrei Rykhlevskii, University of Illinois at Urbana-Champaign

Abstract: The Advanced Reactors and Fuel Cycles (ARFC) group models and simulates the design, safety, and performance of advanced nuclear reactors. Such reactors involve tight coupling between physics such as thermal-hydraulic phenomena, neutron transport, and fuel performance. Current work introduces an extension of the MOOSE framework, Moltres, to appropriately model coupled thermal-hydraulics and neutronics of promising molten-salt-fueled reactor designs. Additionally, we have developed an online reprocessing simulation tool, SaltProc which includes fission product removal, fissile material separations, and refuelling for time dependent analysis of fuel-salt evolution. Initial simulations of the Molten Salt Reactor Experiment and the conceptual Molten Salt Breeder Reactor have been conducted on Blue Waters with deterministic multiphysics and Monte Carlo methods respectively. Steady state, transient, and fuel cycle analysis simulations have been run in 2D as well as 3D and compared against the Molten Salt Reactor Experiment. Fuel cycle dynamics and quasi-equilibrium compositions were obtained from depletion and reprocessing simulations for a 20-year time frame. The MSBR full-core safety analysis was performed at the startup and equilibrium fuel salt compositions, for important reactor safety parameters. In this work, conducted on Blue Waters, ARFC has demonstrated the capability to model complex physics in an advanced molten-salt-fueled nuclear reactor.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

Bigger GPUs and Bigger Nodes

Project PI: Wen-Mei Hwu, University of Illinois at Urbana-Champaign

Presented by: Carl Pearson, University of Illinois at Urbana-Champaign

Abstract: While Blue Waters has been online, GPUs have matured as a tool for enabling high-performance computing. Codes that were originally GPU-accelerated are now facing challenges of moving to next-generation systems, with new GPU architectures and new performance profiles. Furthermore, a typical supercomputing node has become much larger, with multiple CPUs, multiple GPUs, and corresponding high-bandwidth intra-node interconnects to connect these components. This talk discusses one case study of moving existing HPC GPU code from Blue Waters onto a new architecture, and describes some of the performance capabilities of next-generation GPUs and accelerator interconnects.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

 

Molecular and Electronic Dynamics Using the OpenAtom Software

Project PI: Sohrab Ismail-Beigi, Yale University

Abstract: We begin with a description of the OpenAtom software, our collaborative effort to develop and enhance an open source ab initio density functional software package based on plane waves and pseudopotentials (http://charm.cs.uiuc.edu/OpenAtom/) for describing molecular dynamics and electronic excitations of large systems. OpenAtom takes advantage of the Charm++ parallel framework to enable massively parallel scaling. Our scientific collaboration is enabled and supported by an NSF SI2-SSI grant (ACI-1339804). Next, we describe our use of OpenAtom on Blue Waters to understand the dynamics of hydrogen inside of complex metal organic framework (MOF) materials which are of interest for hydrogen storage. While computationally expensive, first principles molecular dynamics approaches permit one to simulate dynamic and time-dependent phenomena in physics, chemistry, and materials science without the use of empirical potentials or ad hoc assumptions about the interatomic interactions since they describe electrons, nuclei and their interactions explicitly.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

 

More Power to the Many: Scalable Ensemble-based Simulations and Data Analysis

Project PI: Shantenu Jha, Rutgers University

Abstract: This talk has two tracks: (i) the development of building block based middleware to support the efficient execution of task parallel applications, and (ii) the application of this middleware to scientific applications.

In the first track we will describe RADICAL-Pilot and characterize its performance to execute 4096 concurrent MPI tasks over 128K cores on a Cray supercomputer. We will then discuss how it forms the runtime system for MIDAS (MIddleware for Data-intensive AnalysiS) and ExTASY (Extensible Tools for Advanced Sampling and analYsis).

In the second track we will discuss the application of developed middleware to the analysis of bimolecular trajectory, and the application of ExTASY to explore the same conformational space of Bovine pancreatic trypsin inhibitor (BPTI) as was done by D.E Shaw et al 2010, but so as to enable the study of more complex proteins at longer time scales. Ongoing studies on BlueWaters have helped to pinpoint the parameters to improve sampling and are being benchmarked for computational efficiency agains the work of D. E. Shaw et al 2010 for BPTI.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

 

Topology-aware and Load Balancing Techniques using Charm++, AMPI and Related technologies

Project PI: Laxmikant V. Kale, University of Illinois at Urbana-Champaign

Presented by: Juan Galvez, University of Illinois at Urbana-Champaign

Abstract: As part of the Blue Waters PAID (Petascale Application Improvement Discovery) project, we have developed software to assist in optimizing the resource utilization of applications, namely their communication performance through topology-aware placement of tasks, and through computational load balancing. This includes many improvements in Charm++/AMPI, as well as development of general-purpose tools. In this talk we will review two such tools developed during this project: a utility used during launching of MPI jobs that optimizes rank placement in the machine topology; and a library to assist applications in load balancing decisions. We will also discuss important developments in the Charm++ load balancing framework, including: (a) new efficient load balancing strategies aimed at preserving the mapping of objects to their machine/network topology locations; (b) load balancing performance improvements; (c) recent advances in heterogeneous and vector load balancers, that allow load to be dynamically balanced across accelerator devices.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

Ensemble Simulations with Experimental Restraints: Running at Scale

Project PI: Peter Kasson, University of Virginia

Abstract: Many problems in biomolecular science involve generating molecular models for processes where experimental data are also available. Many spectroscopic techniques can provide information regarding biomolecular conformational ensembles, but techniques to incorporate these into molecular simulations typically require custom modification of the molecular simulation package. We have developed a plugin architecture and python API to permit arbitrary experimental restraints to be incorporated in ensemble simulations without recompiling the underlying molecular dynamics package. We demonstrate this for DEER spectroscopy of flexible proteins, but it is broadly applicable to many techniques including FRET, SAXS, and cryo-EM and permits running such ensembles at scale on Blue Waters.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Advanced Nanoelectronic Device Design with Atomistic Simulations

Project PI: Gerhard Klimeck, Purdue University

Abstract: Today's transistors have scaled down to the size of countable number of atoms. Tomorrow's transistors require exploration of different configurations and materials, which require the models to be atomistic rather than continuum descriptions. Quantum effects such as tunneling, confinement, and bandstructure dominate device behaviors and must treated. NEMO5 takes advantage of Blue Waters for large scale simulations on a variety of aspects in transistor design including: 1) scattering in Cu interconnects, 2) GaAs based devices for measuring strong electron interference, 3) anti-ambipolar device simulation in novel 2D materials such as MoS2 and black phosphorus, 4) explicit screening, full band quantum transport models, 5) Nitride TFETs designs, 6) channel thickness designs for TFETs, and 7) benchmark of mode-space-based basis reductions in nonequilibrium Green's function simulations with inelastic scattering. We review the scientific impact of NEMO5 and highlight its impact via nanoHUB on education and research.

Field of Science: Materials Science

Watch the presentation | View the presentation slides (PDF)

Improving Virtually Guided Product Certification with Implicit Finite Element Analysis at Scale

Project PI: Seid Koric, University of Illinois at Urbana-Champaign

Presented by: Erman Guleryus, University of Illinois at Urbana-Champaign

Abstract: We continue advancing the state of the art in implicit finite element analysis (FEA) by studying real world models and identifying and removing scaling barriers. We study large-scale representative gas turbine engine models used in virtually guided certification, to support physical engine testing, and for post-test validation of modeling and simulation techniques. Blue Waters is an enabling platform where a massively parallel solver technology can be tested and advanced. We have solved a model with over 100 million finite element equations, which is the largest problem ever solved in-core with implicit LS-DYNA, and likely with any implicit FEA code using a direct solution method. While some of the challenges were known in the abstract, their details were only discovered when we tried to scale. Given the popularity of implicit finite element methods, the results of this effort will further open the doors for high fidelity multi-physics modeling of complex structures.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

Hundreds of New Planets Candidates from Kepler and K2

Project PI: Ethan Kruse, University of Washington

Abstract: NASA's Kepler mission monitored over 200,000 stars for four years during its prime mission; during its K2 mission it studied 20,000 new stars every three months. I have developed a robust method to search for transiting planet candidates, including those often overlooked by other techniques. My planet search pipeline is sensitive to nearly all transiting planets, including ultra short period planets, planets with transit timing variations, and long period planets that only transit once or twice. Here I present my newest planet candidate catalogue and describe how I use Blue Waters to speed up the search of half a million stars. I discuss how my K2 planet candidates complement the original Kepler sample and what we have learned about planet occurrences around stars of different masses and ages.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Towards Quantifying Defect Tolerance in Semiconductors

Project PI: Rachel Kurchin, Massachusetts Institute of Technology

Abstract: Defect tolerance, or the resilience of optoelectronic properties to the presence of defects, is a critical quality for a photovoltaic (PV) material amenable to low-cost manufacturing techniques. Defect-tolerant materials, exemplified by hybrid lead halide perovskites (LHP's), have several attributes, foremost among them the shallow character of native point defects. There has been significant interest in understanding what leads to these shallow defects since the emergence of LHP's within the PV research community. These remarkable materials are able to achieve >20% efficiency despite low-temperature solution synthesis, but suffer from poor long-term stability as well as concerns around toxicity due to their lead content, motivating a search for other materials sharing their defect-tolerant behavior without these drawbacks.

In this work, we learn from prior theoretical/computational as well as experimental work to improve upon prior metrics for predicting defect tolerance in novel materials as well as formulating new ones. We test these metrics in a variety of chemistries and crystal structures using modern state-of-the-art DFT defect formation energy calculations.

Field of Science: Materials Science

Watch the presentation | View the presentation slides (PDF)

 

"Breathing" Clouds and Storms: Inflow and Entrainment, Precipitation and Outflow

Project PI: Sonia Lasher-Trapp, University of Illinois at Urbana-Champaign

Abstract: Clouds and storms develop due to inflow of warm, moist air from near the ground up into their bases. They also ingest air through their edges (entrainment), which may be substantially cooler and drier. Thus, clouds "inhale" air with a variety of characteristics, influencing their ability to precipitate. Some of this air exits the cloud near its top, but the remainder is "exhaled", descending toward the ground (outflow) and cooled by melting or evaporating precipitation. Depending on these conditions, the cloud/storm outflow may be sufficient to generate new, neighboring clouds or storms. We will present high-resolution (down to 50 m) 3D simulations of lines of clouds, as well as individual deeper thunderstorms and thunderstorm complexes, designed to study these aspects. The ultimate goal is to improve fundamental understanding of entrainment and precipitation that can lead to improved forecasts of convective clouds and storms.

Long-standing problems in meteorology include the prediction of (a) how quickly entrainment mixes dry environmental air into clouds/storms, limiting their lifetime and their ability to precipitate, and (b) if/when the outflow from a cloud/storm is sufficient to generate new clouds/storms. A lack of computing power has limited past studies. High-resolution (down to 50 m) 3D simulations of convective lines of clouds, as well as individual deeper thunderstorms and thunderstorm complexes, require a computer like Blue Waters, to properly resolve the smaller turbulent eddies responsible for the entrainment and hold in memory the large arrays of variables that must be computed to evaluate the production of precipitation. The simulations also must be run over large domains (hundreds of kilometers) to represent the larger-scale storm outflows, and over periods 6 hours or more to evaluate their evolution. Such simulations generate very large, complex sets of data, for which Blue Waters is also ideal for processing and analyzing.

Our work on Blue Waters has produced key new findings. Clouds/storms spaced by different amounts along a line differ minimally in terms of their entrainment and its effects, a surprising result, but can differ greatly in terms of their outflows and their ability to generate new clouds/storms. Simulations of supercells thunderstorms show that entrainment is significant in their early stages, before the storms start to rotate, and current work is examining how entrainment changes afterward, and the effects on precipitation during those different stages. The outflows of large complexes of storms, capable of generating new storms, have been shown to be predominantly controlled by the melting of large ice particles created inside the storms, which can be linked to the propagation speed of the outflows, holding some promise in deriving a predictive relationship for new storm generation.

The scientific impact of our work includes weather forecasting and climate modeling. Entrainment has been implicated as a critical limiting factor for thunderstorm development and concomitant rainfall, hailfall, and/or severe winds. The weather forecasting community fully appreciates that storm outflows can produce severe wind damage, and/or additional storms, and even day-long severe weather outbreaks, but knowledge is lacking to identify when such outbreaks will and won't occur. While other research groups around the world investigate how to parameterize the effects of entrainment and thunderstorm outflows in coarser numerical weather prediction models and global climate simulations, our group is providing new, fundamental knowledge of these topics, providing the scientific underpinning for these parameterization efforts for global weather and climate modeling efforts.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

Western U.S. Volcanism due to Intruding Oceanic Mantle Driven by Ancient Farallon Slabs

Project PI: Lijun Liu, University of Illinois at Urbana-Champaign

Presented by: Quan Zhou, University of Illinois at Urbana-Champaign

Abstract: The western United States (U.S.) volcanisms include several components during the last 20 million years. The Columbia River Flood Basalt is a wide area of past volcanism around 17 million years ago. It is ~700 km away from the plate boundary along the western coast of U.S., compared to the typical ~100 km distance of more common volcanoes along convergent plate boundaries The Yellowstone volcano is ~ 1200 km away from the plate boundary. The Yellowstone volcano is also associated with a ~700 km time-transgressive track of past volcanism along the southwest direction. Another volcanism track, the Newberry hot spot track, trends northwest. The Basin & Range province has host volcanisms as well. The origin of these intra-plate volcanisms has been a long debate. The uncertain evolution of the mantle below the U.S. is one of the key reasons for this long debate. To solve this problem, we reconstruct the mantle thermal states beneath the western U.S. during the past 20 million years using an inverse geodynamic model with data assimilation. We implement this algorithm into an open source well-parallelized code CitcomS and apply this code on the Blue Waters. We find that volcanisms correspond to similar eastward intruding oceanic mantle driven mostly by the sinking Farallon slab below central-eastern U.S. The hot mantle forming the Columbia River flood basalt and subsequent Yellowstone-Newberry hotspot tracks first enters the western U.S. through tears within the Juan de Fuca slab. Subsequent coexistence of westward asthenospheric flow above the retreating Juan de Fuca slab and eastward propagating mantle beyond the back-arc region reproduces the bifurcating hotspot chains. A similar but weaker heat source intrudes below the Basin & Range around the southern edge of the slab, and can explain the diffuse basaltic volcanism in this region. According to our models, the putative Yellowstone plume contributes little to the Yellowstone hot spot track formation.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

Orientation and Stability of Magnetic Reconnection x-line at Earth's Magnetopause

Project PI: Yi-Hsin Liu, Dartmouth College

Abstract: Magnetic reconnection plays the critical role in the plasma transport and magnetic energy release at Earth's magnetopause, the sharp boundary separating Earth's magnetosphere and solar wind plasmas. The orientation and stability of the reconnection x-line are studied using petascale 3D particle-in-cell simulations on Blue Waters. We initiate reconnection at the center of a large simulation domain to minimize the boundary effect. The resulting x-line has sufficient freedom to develop along an optimal orientation. Companion 2D simulations indicate that this x-line orientation maximizes the reconnection rate. The further numerical experiments suggest that reconnection tends to radiate secondary oblique tearing modes if it is externally (globally) forced to proceed along an orientation not favored by the local physics.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Multi-scale Simulations of Whole Yeast Cells

Project PI: Zaida Luthey-Schulten, University of Illinois at Urbana-Champaign

Presented by: Tyler M. Earnest, University of Illinois at Urbana-Champaign

Abstract: Cells are spatially heterogeneous systems whose dynamics span multiple length-, time-, and concentration-scales. Due to the highly coupled nature of the chemical reaction networks driving the cell's life cycle, it is not sufficient to focus on a single aspect of these dynamics. However, no single simulation modality is capable of probing behavior spanning these scales. We present our results on hybrid simulation techniques, produced using Blue Waters, which couple the deterministic dynamics of abundant metabolite molecules with the stochastic dynamics of gene expression, on the galactose genetic switch of S. cerevisiae. Through the coupling of reaction-diffusion master equation simulations to well-mixed deterministic chemical kinetics using our GPU-accelerated Lattice Microbes (LM) software, we are capable of simulating hours of simulation time of a system spanning nanomolar to millimolar concentrations. Blue Waters provided the necessary computing power to complete this research as well as directed the development of LM to maximize performance.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

High Energy Physics on Blue Waters

Project PI: Paul Mackenzie, Fermilab

Presented by: Steven Gottlieb, Indiana University

Abstract: Quantum Chromodynamics (QCD) is the theory of Nature's strong force, responsible for the binding of quarks into protons and neutrons which comprise the bulk of stable matter. Blue Waters has enabled lattice QCD calculations of precision only dreamed of before its installation. We can use finer grids and lighter quark masses then previously possible. With these improvements, the reduction in systematic errors allows stringent tests of the Standard Model of Elementary Particle Physics. Using the Highly Improved Staggered Quark (HISQ) action, we use a lattice spacing of 0.042 fm (10-15 m) on a 1443 X 288 grid with physical masses for dynamical up, down, strange, and charm quarks. Using the domain wall (DW) action, we use a lattice spacing of 0.082 fm on a 643 X 128 X 12 grid with physical masses for up, down, and strange dynamical quarks. The data we create on Blue Waters is shared internationally.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

Integrating Physics-based Earthquake Cycle Simulator Models and High-Resolution Ground Motion Simulations into a Physics-based Probabilistic Seismic Hazard Model

Project PI: Philip Maechling, University of Southern California

Presented by: Scott Callaghan, University of Southern California

Abstract: The Southern California Earthquake Center (SCEC) is using Blue Waters to develop an empirically-calibrated physics-based earthquake forecast, and to characterize the influence of source models and non-linear material response to strong shaking on the accuracy of ground motion simulations. We summarize recent progress achieved through these simulations. During the last year, we used the physics-based earthquake cycle simulator code, called RSQSim, to produce several million-year earthquake catalogs to investigate how fault complexities affect the probabilities of large, multi-fault ruptures and multi-event sequences. We also performed systematic verification of the methods and procedures currently used in three-dimensional high-frequency (f ≤ 5 Hz) earthquake ground motion simulations, using alternative numerical methods and codes. This research is improving the physical representations of earthquake processes and the deterministic codes for simulating earthquakes, which will improve probabilistic seismic hazard analysis in the United States and benefit earthquake system science worldwide.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

Search for Missing Variants in Large Exome Sequencing Projects by Optimization of Analytic Pipelines, in application to Alzheimer's disease

Project PI: Liudmila Sergeevna Mainzer, University of Illinois at Urbana-Champaign

Presented by: Yingxue Ren, Mayo Clinic

Abstract: The identification of genetic risk factors in complex diseases has shifted to rare variants s. The read-to-variant analytic pipelines are known to provide an incomplete set of variants, and need to be applied together to detect rare variants in large sample cohorts. Our study utilizes the data from the Alzheimer's Disease Sequencing Project, consisting of over 10,000 whole exome sequencing samples. Alzheimer's disease is the most common form of dementia and the 6th leading cause of death in the United States. We tested many combinations of software packages and parameters in a "workflow integration" approach. Until now this has been impossible to accomplish due to the prohibitive amounts of compute time. In this work we recovered 50% additional variants for the ADSP that were missed by the standard protocol. Correct and complete identification of such variants is the prerequisite to successful resolution of this important societal problem.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Quantum-Classical Path Integral Simulation of Electron Transfer Reactions in Solution

Project PI: Nancy Makri, University of Illinois at Urbana-Champaign

Abstract: Quantum mechanics is a nonlocal theory, which requires computational effort that scales exponentially with the number of coupled degrees of freedom. A classical treatment of the nuclear degrees of freedom, governed by Newtonian forces on each electronic state, is efficient and often sufficiently accurate. However, accounting for the change of electronic states in a classical trajectory treatment presents a major challenge because classical trajectories are local in space, while quantum mechanical wavefunctions are delocalized. As a result, attempts to combine quantum and classical descriptions have in the past relied on ad hoc assumptions and uncontrolled approximations. The path integral formulation of time-dependent quantum mechanics provides the ideal framework for rigorous quantum-classical or quantum-semiclassical treatments, as the spatially localized, trajectory-like nature of the quantum paths circumvents the need for mean-field-type assumptions. However, the number of system paths grows exponentially with the number of propagation steps. In addition, each path of the quantum system generally gives rise to a distinct classical solvent trajectory. This exponential proliferation of trajectories with propagation time is the quantum-classical manifestation of nonlocality. The quantum-classical path integral (QCPI) offers a rigorous, yet efficient approach for simulating charge transfer processes in condensed phase environments. The methodology is free of ad hoc assumptions or adjustable parameters. It is well suited to a decomposition based on multi-level parallelism, and Blue Waters provides the ideal platform for its implementation. QCPI simulations of electron transfer reactions in solution, where over 3 10 atoms are treated in full atomistic detail, yield results of unprecedented accuracy. It is seen that the interaction with the electron transfer pair introduces quantum delocalization to the otherwise classical solvent atoms. The QCPI methodology has also been adapted to the calculation of reaction rates, providing a unified framework applicable to fast as well as very slow reactive processes.

Field of Science: Chemistry

Watch the presentation | View the presentation slides (PDF)

Understanding and Design of Hierarchical Self-assembly of Biocompatible Optoelectronic Nanostructures through a Patchy Minimal Model

Project PI: Rachael A. Mansbach, University of Illinois at Urbana-Champaign

Abstract: Self-assembling peptides containing aromatic groups are an attractive target for bioelectronic materials design due to their ease of manufacture, biocompatibility, aqueous solubility, and controllability of side chain chemistry. Microscopic understanding of the properties that control assembly is a prerequisite for rational design. We employ a patchy particle model in conjunction with the high performance computing abilities of Blue Waters to traverse the parameter space of interactions in systems of thousands of molecules over timescales of hundreds of microseconds. We study the effects of side chain and aromatic group interaction strength and steric constraints on peptide assembly. We identify parameters that lead to rapid growth of linear aggregates with strongly interacting aromatic cores and identify candidates for high-resolution computational and experimental testing. Our work leads to greater understanding of the parameters that control aggregation and demonstrates a method for identification of candidate peptides to assemble hierarchical nanostructures with desirable optoelectronic properties.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

Thermodynamic Characterization of Conformational Landscape in Proton-Coupled Oligopeptide Transporters

Project PI: Mahmoud Moradi, University of Arkansas, Fayetteville

Abstract: Proton-coupled oligopeptide transporters (POTs) are membrane transport proteins that uptake small peptides and peptide-like molecules. Human POT transporters PepT1 and PepT2 play a key role in absorbing and retaining dietary proteins and uptake several important families of peptide-like drugs such as β-lactam antibiotics. In this project, Blue Waters resources are used to study the conformational landscape of GkPOT, a bacterial homolog of PepT1 and PepT2. Employing enhanced sampling techniques, free energy calculations, and path-finding algorithms, within a novel Riemannian framework, our simulations have characterized the conformational free energy landscape of GkPOT. The algorithms are primarily based on a loosely-coupled multiple-copy scheme, requiring the parallel execution of hundreds of all-atom molecular dynamics simulations that utilize Blue Waters resources efficiently and sample the complex protein conformational landscapes effectively. A detailed description of GkPOT conformational landscape sheds light on the structure-function relationship in POTs and other membrane transporters.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Petascale Particle-in-cell Simulations of High Energy Density (HED) Plasmas

Project PI: Warren Mori, University of California, Los Angeles

Presented by: Frank Tsung, University of California, Los Angeles

Abstract: In high energy plasmas, the so-called plasma parameter λ, is finite and must be taken into account. In that spirit, the UCLA Simulation Group develops and uses a hierarchy of kinetic software tools to study a variety of problems in high energy density plasma physics, high intensity laser and beam plasma interactions, plasma based acceleration, the nonlinear optics of plasmas, and inertial confinement plasmas, and astrophysics. These tools run efficiently on both single cores and on over a million cores. The computational resources at Blue Waters have been essential in allowing our group to perform large-scale simulations which have led to new discoveries and publications in high impact journals. In this talk, we will highlight our research results from the past year on plasma based acceleration and the nonlinear optics of plasmas of relevance to inertial confinement fusion. We will also mention software development efforts including enabling the software to take advantage of current and future many-core architectures. The software development is partially supported through the NSF funded Particle-in-Cell Kinetic Simulation Software Center (PICKSC). Much of the software is available through GitHub on the PICKSC website, http://picksc.idre.ucla.edu

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

ArcticDEM: 2m Topography and Surface Change Detection over the Arctic

Project PI: Paul Morin, University of Minnesota

Presented by: Claire Porter, Polar Geospatial Center, University of Minnesota

Abstract: Accurate topography is a foundational dataset that enables research in geology, glaciology, hydrology, and ecology. While the rest of the world has well-funded campaigns to gather such information, the Polar Regions have been long neglected. This project takes advantage of a unique opportunity to leverage unprecedented commercial satellite imagery coverage in Polar Regions to derive elevation data from stereoscopic satellite imagery using open source software. Over 3PB of DigitalGlobe imagery licensed for US federal scientific was run through the Surface Extraction by TIN-based Search-space Minimization (SETSM) package to extract 2m resolution Digital Surface Models of the entire Arctic north of 60 degrees latitude. The resulting products, both a seamless mosaic and individual time-stamped DEM strips, are being released to the public and will serve both as a reference dataset and a way of detecting and quantifying change in this dynamic region of the Earth.

Field of Science: Earth Sciences

Watch the presentation | View the presentation slides (PDF)

 

Parallel Algorithms for Solving Large Assignment Problems

Project PI: Rakesh Nagi, University of Illinois at Urbana-Champaign

Presented by: Ketan Date, University of Illinois at Urbana-Champaign

Abstract: In this work, we discuss efficient parallel algorithms for optimally solving large instances of the Linear Assignment Problem (LAP) and the Quadratic Assignment Problem (QAP). Our parallel architecture is comprised of both multi-core processors and Compute Unified Device Architecture (CUDA) enabled NVIDIA Graphics Processing Units (GPUs) on the Blue Waters supercomputer at the University of Illinois at Urbana-Champaign. We propose novel parallelization of the Hungarian algorithm on the GPUs, which shows excellent parallel speedup for large LAPs, with up to 400 million variables. We also propose novel parallelization of the Dual Ascent algorithm on the GPUs, for solving the RLT2 linearization of the QAP, with the LAP sub-problems being solved using our parallel Hungarian algorithm. We show that this GPU-accelerated approach can be used to obtain quick and tight lower bounds on large instances of the QAP (with up to 42 facilities and locations), which can be extremely valuable in a branch-and-bound scheme.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

Predicting the Transient Signals from Galactic Centers: Circumbinary Disks and Tidal Disruptions Around Black Holes

Project PI: Scott C. Noble, The University of Tulsa, NASA-GSFC

Abstract: Binary systems of supermassive black holes hold the key to resolving open questions regarding galactic evolution and massive black hole growth. We aim to advance our theoretical understanding of these systems to prepare for their detection using electromagnetic and gravitational waves. We have been developed novel codes in the past few years on Blue Waters enabling us to realistically simulate these systems in 3-d using general relativity for the first time. The simulation has now been ray-traced, resulting in the first self-consistent time-dependent sequence of an accreting binary's electromagnetic spectrum. The dynamics of the accretion flow and the dependence on the spectrum to accretion rate, angle of observation, and time will be shared. We will further explore how circumbinary disk tilt modifies the picture from a new series of simulations. We will characterize the the degree of disk alignment, and explain how the tilt imprints affects the accretion dynamics.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Realistic Simulations of the Intergalactic Medium: The Search for Missing Physics

Project PI: Michael L Norman, University of California at San Diego

Abstract: We have performed state-of-the-art simulations of the intergalactic medium (IGM) during the epoch of helium reionization. UV radiation from quasars ionize the intergalactic helium over a period of several billion years beginning about 1 billion years after the Big Bang. We have carried out a suite of the first fully-coupled radiation hydrodynamic cosmological simulations which treat the quasars as a time varying population of point sources. We have performed multigroup radiative transfer self-consistently coupled to the cosmological hydrodynamics of the IGM at sufficient resolution and domain size to examine the photoionization and photoheating processes in detail. We have discovered that helium reionization completes significantly later compared to models which treat the quasar radiation as a homogeneous background. This modifies the heating history of the IGM substantially, with a maximum mean temperature of 14,000 K being achieved at a redshift of 3, consistent with observations.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Exploring Galactic Environments with Adaptive Mesh Simulations on Blue Waters

Project PI: Brian O'Shea, Michigan State University

Abstract: When people talk about galaxies, they are usually referring to the disk or ball of stars that can be seen with the naked eye. In reality, a "galaxy" is a much larger system than just the stars - it includes a huge dark matter halo, as well as a cloud of hot, diffuse plasma that surrounds the stars. This "circumgalactic medium," or CGM, comprises half of the baryonic mass of the system, and is tightly coupled to the stars in the galaxy in a variety of important ways. In this talk, I will describe our collaboration's recent attempts to understand how stars and galaxies in the circumgalactic medium relate to each other, and how simulating these environments correctly presents challenges that require machines like Blue Waters in order to understand them.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Reducing Communication in Sparse Matrix Operations

Project PI: Luke Olson, University of Illinois at Urbana-Champaign

Abstract: Sparse matrix computations abound in large scale simulations. Iterative sparse solvers such as conjugate gradient or algebraic multigrid, for example, rely on efficient and scalable sparse matrix operators. However, high parallel communication costs can lead to severe limitations in performance. In this talk, we explore new methods for handling communication and the resulting impact on these operations on Blue Waters. A key feature of current and upcoming architectures is node-level parallelism. Standard approaches to inter-process communication will send data regardless of the locations of send and receive processes. Yet, there are notable differences in the cost of intra- and inter-node communication. In response, communication can be restructured to take advantage of the less costly intra-node communication, reducing both the number and size of inter-node messages. We show results on Blue Waters that leads to improved efficiency and scalability of the underlying sparse operations.

Field of Science: Computer Science

Watch the presentation | View the presentation slides (PDF)

Understanding the Development and Evolution of Violent Tornadoes in Supercell Thunderstorms

Project PI: Leigh Orf, University of Wisconsin-Madison

Abstract: Since 2013 our team has been conducting simulations of violent tornado-producing supercell thunderstorms at unprecedented spatial resolution (up to 15 m grid spacing). In order to facilitate the highest possible temporal resolution in saved model data, a file system (LOFS) was recently developed that utilizes the HDF5 core driver with lossy floating point compression (ZFP). This approach has allowed us to save many model fields every model time step (1/6 s) for the entire life cycle of an EF5-strength tornado, a total of 32,400 saved times, comprising 67 TB of output. I will describe LOFS and the tornado science it is facilitating, which includes the ability to do Lagrangian parcel tracking analysis from saved model data on modest hardware that until now required the (expensive) approach of integrating parcels during model integration, which requires a large Blue Waters job for each experiment involving following sets of trajectories.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

Computational Methods for the Design of Self-assembled Macromolecular Therapeutic Agents

Project PI: William M. Payne, University of Nebraska Medical Center

Abstract: As the cost and development time required for new drugs to reach clinical use continues to increase, the field of nanomedicine has emerged as a promising area of research in an attempt to substantially improve therapeutic outcome, particularly in cancer chemotherapy. However, while the number nanomedicine publications have continued to rise, very few of these inventions make it to market. This problem is exacerbated by the lack of meaningful methods to determine the efficacy and efficiency of a given formulation; comparing one nanomedicine to another or even objectively describing how well the nanomedicine treats a given disease is nearly impossible. We present the design and synthesis of a library of macromolecular contrast agents for use in cancer imaging or treatment. The physiochemical properties of these conjugates are evaluated experimentally through the use of light scattering, which is then correlated to theoretical investigations using molecular dynamics simulations.

Field of Science: Chemistry

Watch the presentation | View the presentation slides (PDF)

Unveiling the Functions of the HIV-1 and Hepatitis B Virus Capsids through the Computational Microscope

Project PI: Juan R. Perilla, University of Delaware

Abstract: Viruses represent a serious public health threat, and millions of people die every year from viral diseases. In an effort to prevent and combat viral infection, researchers worldwide are endeavoring to develop vaccines and drug-based treatments. Virus capsids, specialized protein shells that play essential structural and functional roles in housing, protecting, and ultimately delivering the genome of viral pathogens, are currently of great pharmacological interest as drug targets. Employing all-atom molecular dynamics (MD) simulations we investigated the capsids of hepatitis B virus (HBV) and human immunodeficiency virus 1 (HIV-1), with the goal of revealing critical insights that will enhance the development of therapeutic interventions.

Field of Science: Biosciences

Modeling Heliospheric Phenomena with Multi-Scale Fluid-Kinetic Simulation Suite

Project PI: Nikolai Pogorelov, University of Alabama in Huntsville

Abstract: The key challenge of our project is to perform multi-scale simulations of the solar wind (SW) from the solar surface to the boundary of the heliosphere, and its interaction with the local interstellar medium (LISM). Our simulations have both fundamental and practical applications: (1) the physics of SW-LISM interaction is common to many astrophysical objects and fusion plasma, (2) description of the SW flow and magnetic field at Earth and planets allows us to predict hazardous conditions affecting humans and electronics in space. Our accomplishments: (1) SW simulations with pickup ions (PUIs) and turbulence included, (2) development of a data-driven model for coronal mass ejections; and (3) modeling the SW-LISM interaction to interpret observations from Voyagers and IBEX spacecraft. MS-FLUKSS solves self-consistently the system of MHD equations for the ion-electron mixture, kinetic Boltzmann equation for neutral atoms, and two separate systems for PUIs and turbulence generated by them.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Unified Modeling of Galaxy Populations in Clusters

Project PI: Thomas Quinn, University of Washington

Abstract: Clusters of galaxies are both a useful probe of cosmology and a laboratory for understanding galactic feedback processes. However, modeling galactic scale feedback processes in the context of a cluster presents a computational challenge because of the large dynamic range involved. Through the use of a highly scalable N-body/Smooth Particle Hydrodynamics code running on Blue Waters, this project is tackling this challenging problem. Preliminary results show that models that have successfully reproduced the morphology and number densities of field galaxies can also produce realistic models of cluster galaxies, while at the same time reproducing the observed properties of the Intra Cluster Medium. Large computational resources with high performance networks are necessary for these calculations.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Effects of Forcing Scheme on the Flow and the Relative Motion of Inertial Particles in DNS of Isotropic Turbulence

Project PI: Sarma L. Rani, University of Alabama in Huntsville

In direct numerical simulations (DNS) of homogeneous isotropic turbulence, statistical stationarity is achieved through a forcing scheme that adds energy to the large scales. The forcing schemes may be broadly classified into deterministic and stochastic forcing schemes. In deterministic schemes, the forcing added is such that the turbulent kinetic energy dissipated during a time step is resupplied, whereas in stochastic schemes, the forcing is determined based on the evolution of Ornstein-Uhlenbeck processes. Both approaches add the forcing within a band of wavenumbers at the low-wavenumber end of the energy spectrum. The goal of the current study is to investigate the effects of the forcing schemes on the flow, as well as on the relative motion of inertial particles in DNS of isotropic turbulence. An important parameter in stochastic forcing is the characteristic time scale of the applied forcing, TF. In this study, DNS was performed using the deterministic forcing, as well as stochastic forcing for five values of TF = Teddy/4, Teddy/2, Teddy, 2Teddy, 4Teddy. Here Teddy is the eddy turnover time obtained from the DNS with deterministic forcing. Three Taylor micro-scale Reynolds numbers Reλ = 76, 131, and 210, and twelve particle Stokes numbers based on the Kolmogorov time-scale, Stη ranging from 0.05 to 40 were considered. Detailed analysis of the effects of forcing time scales on both fluid and particle statistics was undertaken.

Field of Science: Fluid Systems

Watch the presentation | View the presentation slides (PDF)

Mapping Proton Quark Structure

Project PI: Caroline Riedl, University of Illinois at Urbana-Champaign

Presented by: Matthias Grosse Perdekamp, University of Illinois at Urbana-Champaign

Abstract: COMPASS probes proton substructure by scattering high-energy pion- and muon-beams on nuclear targets at CERN. The experiment explores the momentum and coordinate phase space of quarks inside the proton. Observing correlations between proton spin and the intrinsic transverse momentum of quarks will shed light on the quark dynamics inside the proton and will provide a critical test of fundamental predictions derived from Quantum Chromo Dynamics, the quantum field theory describing the strong nuclear force. The measurements will produce 10 petabytes of experimental and simulated data. Blue Waters' balance of processing capabilities and data storage and handling is well suited for the analysis of the large COMPASS data samples as these require significant algorithmic processing per pion/muon-proton scattering event. In addition to raw data processing and physics-level analysis, COMPASS carries out extensive studies of systematic effects as Blue Waters allows for a detailed simulation of COMPASS detector and environmental properties.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

Kinetic Simulations of Astrophysical Plasma Turbulence

Project PI: Vadim Roytershteyn, Space Science Institute

Abstract: Plasma turbulence is ubiquitous in space and astrophysical systems. In many of these systems, turbulence is thought to provide an important, if not dominant, source of heating, viscosity, and may determine other macroscopic properties of plasma. Yet, in hot and/or rarefied plasmas often encountered in space, the deposition of turbulent energy into plasma is a highly nontrivial process. Understanding of the relevant microscopic physical mechanisms requires challenging simulations that are capable of bridging a wide range of scales, while simultaneously including a faithful description of physical processes associated with those scales. In this presentation, we will discuss the progress made by our Blue Waters project that is aiming to use large-scale simulations to shed light onto the physics of small scales in solar wind turbulence. Using complimentary simulation techniques, we were able to extend investigation to challenging parameters relevant to near-Sun regions, where the solar wind turbulence originates.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

 

High Throughput Search for New Plasmonic Materials

Project PI: Andre Schleife, University of Illinois at Urbana-Champaign

Presented by: Ethan Shapera, University of Illinois at Urbana-Champaign

Abstract: Plasmonics seeks to manipulate light at the nanoscale. Precise control over plasmon response enables many applications including: sub-wavelength waveguides, nanoantennas, superlenses, subwavelength imaging, nanocircuitry, and biosensors. Finding new metals and doped semiconductors which may act as viable choices for plasmonic applications remains an outstanding problem. This work uses high throughput density functional theory to calculate quality factors for 970 metals randomly chosen from the Materials Project database. These quality factors are used to train machine learning models which enable rapid estimation of quality factors without requiring DFT calculations. For materials predicted to have high quality factors, crystal, electronic, and optical properties will be computed with DFT. This work will use the application of plasmonics to motivate the study of physical effects which influence optical absorption of crystals, including bandstructure, optical transition matrix elements, spin-orbit coupling, exciton formation, and surface geometry.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

Full-scale Biophysical Modeling of Hippocampal Networks during Spatial Navigation

Project PI: Ivan Soltesz, Stanford University

Presented by: Ivan Raikov, Stanford University

Abstract: The hippocampus provides the basis for episodic memory in the brain, remembering events experienced in the past and linking them with their spatio-temporal context. The hippocampal circuits are comprised of diverse cell types, each exhibiting distinct dynamics and complex patterns of synaptic connectivity. Our project combines experimental data from in-vivo large-scale monitoring of identified neurons with computational modeling to elucidate how the hippocampus generates oscillatory events called sharp-wave ripples, which represent replay of episodic memory sequences and are required for memory consolidation. To aid in the interpretation of experimental data, we are building 1:1 scale computational models of hippocampal circuits with detailed neuronal morphology and biophysics. Simulations of the dentate gyrus subfield during spatial navigation show place fields and biophysical oscillatory population dynamics. To solve the general problem of tuning large-scale network models to fit high-level neural function, we have developed a powerful software package to perform parallel multi-objective optimization.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Massively Parallel Simulations to Analyze Viral Infection Propagation Risk in Air Travel

Project PI: Ashok Srinivasan, Florida State University

Presented by: Sirish Namilae, Embry Riddle Aeronautical University

Abstract: Air travel has been identified as a leading factor in the spread of infections. In earlier work, we showed that changes to boarding and deplaning procedures could substantially reduce the number of contacts between passengers, with potential reduction in the risk of transmitting Ebola on flights. We extended this to airport grates. Our approach leads to a large parameter sweep, leading to a high computational cost. We showed that a Low Discrepancy parameter sweep, in contrast to the traditional lattice based sweep, can lead to a one to three orders of magnitude reduction in computational effort. However, this has greater load imbalance issues than the traditional parameter sweep. We develop solutions to this load imbalance, and relate it to number theoretic properties of the low discrepancy sequence.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

Analyzing Tropical Cyclone-Climate Connections using the Community Earth System Model

Project PI: Ryan Sriver, University of Illinois at Urbana-Champaign

Abstract: This project explores the relationship between tropical cyclones and climate using a high-resolution state-of-the-art Earth system model (Community Earth System Model -- CESM), capable of simulating realistic tropical cyclone circulations and global cyclone activity metrics (number, intensity distribution, seasonality, etc.). We conducted a series of model experiments using Blue Waters that analyze tropical cyclones in CESM on a global scale. The experiment consists of three multi-decadal simulations, featuring the 25 km resolution global atmosphere and varying levels of ocean coupling. Outputs from these simulations are used as inputs to atmosphere-only and ocean-only experiments, which enable us to analyze how cutting ocean-atmosphere interactions and feedbacks can influence the mean state and variability within the ocean and atmosphere separately. This comprehensive modeling framework provides a useful testbed for analyzing tropical cyclone activity and variability for different levels of ocean-atmosphere coupling.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

High Resolution Earth System Modeling for International Climate Assessment

Project PI: Ryan Sriver, University of Illinois at Urbana-Champaign

Presented by: Susan Bates, National Center for Atmospheric Research

Abstract: This project conducts high-resolution climate simulations using the Community Earth System Model (CESM) on Blue Waters to investigate high impact climate events such as tropical cyclones (including analysis of landfalling hurricanes), mid-latitude storms and storm tracks, heat extremes, atmospheric rivers, and mesoscale convective systems. This talk will address the representation of these processes in high-resolution versus coarser resolution simulations, driving mechanisms of these processes, and how they may change in a warmer, future time period. Using Blue Waters resources, we have significantly contributed to a large set of comparable simulations of varying ocean and atmosphere resolution ideal for assessing the impact of model resolution on climate processes and future climate change. Without resources such as Blue Waters, efforts such as this that include a large number of climate simulations at high resolution would not be possible.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

Automatic Knowledge Base Construction and Hypothesis Generation: Antibiotic Resistance Mechanisms for Escherichia Coli

Project PI: Ilias Tagkopoulos, University of California at Davis

Presented by: Nicholas Joodi, University of California at Davis

Abstract: Antibiotic resistance is one of the leading threats to global health, food security and development today. The construction of a cohesive knowledge base for antibiotic resistance that can be a source for machine learning methods will make broad impacts in the field and eventually enable an artificial intelligent (AI) system to automate knowledge discovery in unprecedentedly efficient and unbiased ways. We are building a knowledge base for the E. coli antibiotic resistance mechanisms that are specifically structured for efficient machine learning. The constructed knowledge base is a set of tuples in graph format where nodes represent entities and edges represent relations provided with a confidence score of each tuple. It integrates information from existing antibiotic resistance databases (CARD and PATRIC), gene-regulatory relation databases (EcoCyc and RegulonDB), high-throughput profiles and manual curation of missing information from literature, if any. Due to their nature of interdependency, we are quantifying the confidence of each tuple as well as the confidence of its supporting evidence by measuring one another iteratively until a convergence is reached. In parallel, we are building a hypothesis generator over the ever-growing knowledge graph by leveraging the HPC parallel computing platform to train and further optimize a fused prior model that incorporates the Entity-Relation Multilayered Perceptron (ER-MLP) and the Path Ranking Algorithm (PRA).

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Probing Protein Mechanics with Molecular Dynamics Simulations and Single-Molecule Experiments

Project PI: Emad Tajkhorshid, University of Illinois at Urbana-Champaign

Presented by: Rafael C. Bernardi, University of Illinois at Urbana-Champaign

Abstract: Steered Molecular Dynamics (SMD) simulations have been used to depict the underlying molecular details of protein mechanics. Cells can sense and respond to mechanical cues in their environment by, for example, exhibiting differential biochemical activities. At the molecular level, these behaviors are governed by mechanically active proteins. Such proteins can sense and respond to force by undergoing conformational changes or modulating their function in a variety of ways. Extracellular protein complexes, e.g., cellulosomes from cellulolytic bacteria, or adhesion proteins from pathogenic bacteria, were found to be some the most mechanically stable proteins known to date. How do cellulosomal complexes become ultrastable when exposed to force? and what are the mechanics that make pathogenic bacteria adhere to their human hosts so viciously? The mechanism of these mechanical stabilities has been investigated combining in silico, employing SMD, and in vitro, through molecular biology assays and AFM-based single molecule force spectroscopy.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

Simulation of Geometrically Accurate, Multibillion Atom Cellular Membrane Structures

Project PI: Emad Tajkhorshid, University of Illinois at Urbana-Champaign

Presented by: Noah Trebesch, University of Illinois at Urbana-Champaign

Abstract: Membranes are fundamental to the function and structure of cells and organelles, and molecular dynamics (MD) simulations have the potential to reveal deep insights about the basic physics and biochemistry that produce their behavior. However, MD simulations of cellular membranes have been extremely limited up to this point due to the geometric complexity and sheer size of such membranes. To overcome these limitations, we have developed xMAS (Experimentally-Derived Membranes of Arbitrary Shape) Builder, software designed to turn low-resolution 3D meshes derived from experimental techniques into atomistic membrane models that can be simulated using MD. In the first application of this software, we have used an experimentally-derived 3D mesh and lipid composition to develop a realistic atomistic model of a Terasaki ramp, a helicoidal membrane structure found at the junction between sheets of the endoplasmic reticulum. The model measures approximately 1.97μm by 1.59μm by 0.61μm and is composed of ~36.6 million lipids (~4.5 billion atoms), making it one of the largest atomistic biological models to ever be built. Building this model with xMAS Builder involves several methodologically innovative steps, including using MD to simulate ~36.6 million Lennard-Jones particles while attracted to grid-based potentials to optimize the packing of the membrane lipids, running billion atom MD simulations to fix ring piercings and other complex lipid clashes using a newly developed energy minimization technique, and utilizing grid-based potentials to equilibrate the model while maintaining its experimentally-derived shape. Simulations of the cellular membrane models built using xMAS Builder will allow us to computationally probe the complex interactions that give rise to structural stability and function at an unprecedented atomistic level of resolution, and they will allow us to build better coarse models of the systems for simulation of longer timescale phenomena.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

GPU-accelerated Simulations of Misaligned Accretion onto Spinning Black Holes

Project PI: Alexander Tchekhovskoy, Northwestern University

Abstract: Accreting black holes emit copious radiation, eject outflows, and enrich the Universe with heavy elements. Particularly interesting and challenging are luminous accretion flows, which are easy to observe because they are bright but challenging to model because the radiation cools them into thin, difficult-to-resolve disks. The numerical challenge becomes extreme in the typical but relatively unexplored case of tilted accretion, where the black hole and accretion midplanes are misaligned. For the past 40 years, the standard expectation has been that the inner regions of tilted thin disks align with the black hole midplane; tantalizingly, general relativistic simulations have shown no such alignment. Facilitated by Blue Waters, we developed a GPU-accelerated adaptive mesh refinement (AMR) general relativistic MHD code, H-AMR, and carried out the simulations of thinnest tilted disks to date. We find that the inner regions of the disk not only align with the black hole but also break off from the outer, misaligned parts of the disk, resulting in wide range of theoretical and observational consequences.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Computation of Gravitational Waveforms from Compact Binaries

Project PI: Saul Teukolsky, Cornell University

Presented by: Mark A. Scheel, California Institute of Technology

Abstract: The Laser Interferometer Gravitational-wave Observatory (LIGO) has revolutionized astronomy by discovering gravitational waves from black-hole and neutron-star binaries. Analyzing gravitational-wave data requires comparing detected signals to predictions of Einstein's equations; these predictions can be computed numerically, or by fits to numerical computations. We discuss simulations done on Blue Waters by a code SpEC that solves Einstein's equations and produces a waveform model for LIGO data analysis. Binaries that contain neutron stars are computationally more difficult than the black hole case because of additional physics (e.g. hydrodynamics, neutrinos); current neutron star codes are not yet accurate enough for LIGO's needs. We describe a new code SpECTRE that is aimed at solving this problem. SpECTRE's novel ingredients are discontinuous Galerkin methods and task-based parallelism. We discuss scaling of a SpECTRE prototype on Blue Waters.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

Multiphysics Modeling of Steel Continuous Casting

Project PI: Brian G. Thomas, University of Illinois at Urbana-Champaign

Presented by: Seong-Mook Cho, Colorado School of Mines

Abstract: Continuous casting is used to manufacture over 95% of steel in the world and many defects in final steel products are related to complex multiphysics phenomena in the mold region of this process, including turbulent multiphase flow, particle transport and capture, MagnetoHydroDynamics (MHD), heat transfer, and solidification. This project aims to develop computationally-intensive multiphysics models to predict defect formation mechanisms, and to apply them to improve this important manufacturing process. Recently, bubble behavior and size distributions in the turbulent flow have been investigated using a hybrid multiphase-flow model. In addition, LES coupled with MHD and Lagrangian particle-transport models have been applied to quantify the effects of static or moving magnetic fields on transient fluid flow, particle capture into the steel shell, and heat transfer in the mold, using both the multi-GPU in-house code, CUFLOW on Blue Waters XK-node and the commercial CFD code ANSYS-Fluent with numerous User-Defined Functions on XE-node.

Field of Science: Engineering

Watch the presentation | View the presentation slides (PDF)

Advanced Space Weather Modeling

Project PI: Gabor Toth, University of Michigan

Presented by: Ward Manchester, University of Michigan

Abstract: We achieve breakthrough advances in simulating space weather from magnetic fields emerging from the solar interior to erupt and impact the magnetosphere and produce geomagnetic storms. Flux emergence simulations are performed with our high-resolution magnetohydrodynamic (MHD) Spherical Wedge Active Region Model (SWARM) whose domain extends the full size of a solar active region. The coupled model extends from the convection zone to the corona allowing rigorous simulation of the formation and evolution of active regions. We also perform three-dimensional (3D) simulations of Earth's magnetosphere with kinetic reconnection physics to capture the interaction between solar wind and Earth's magnetosphere. Our global BATS-R-US MHD simulations are embedded with the iPIC3D particle-in-cell model located at the dayside magnetopause. The two-way coupled (MHD-EPIC) system describes reconnection related phenomena: lower hybrid drift instability (LHDI), X-line motion, and flux transfer events (FTEs) along the magnetopause, which agree with space-based Magnetospheric Multiscale Mission (MMS) and ground-based observations.

Field of Science: Astronomy and Astrophysics

Watch the presentation | View the presentation slides (PDF)

 

Viral Morphogenesis Through the Lens of Large-Scale Coarse-Grained Simulations

Project PI: Gregory Voth, University of Chicago

Presented by: Alexander Pak, University of Chicago

Abstract: Viruses take advantage of host cell machinery to create new viral particles that package, prepare, and release genetic material for proliferation. While an understanding of these highly dynamical processes has clear biomedical implications, molecular details have been difficult to resolve through conventional experiments. Instead, we naturally turn toward molecular dynamics simulations. However, the large number of involved constituents, which interact over large length- and time-scales, preclude the use of atomistic simulations. Our strategy is to systematically develop coarse-grained (CG) molecular models that retain essential biomolecular behavior (e.g., for proteins and lipids) with reduced atomic detail. These models provide a computational lens into biophysical dynamics that would otherwise be difficult or impossible to study. With the aid of the Blue Waters computing platform, and in collaboration with experimental scientists, we present our most recent uses of high-fidelity CG models to investigate HIV-1 and influenza morphogenesis.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

 

A Data-centered Approach to Understanding Emergent Quantum Behaviors in Materials

Project PI: Lucas K. Wagner, University of Illinois at Urbana-Champaign

Abstract: Effective models are critical to understanding and simulating the behavior of systems. Much of our thinking about materials uses some simplified effective model to make the computations easier, and many of the big successes in materials physics have been the development of high quality effective models. However, these two approaches of effective models and first principles simulations are typically only connected in an ad-hoc, system-by-system way, particularly when the effective models are themselves quantum mechanical in nature. We have developed a formal and rigorous method based on data analysis for linking first principles simulations to effective quantum models, based on analyzing data taken from first principles calculations, and so it can use data science tools to validate effective models and to help find features. In this talk, I will discuss some recent results using Blue Waters to derive effective quantum models for materials.

Field of Science: Physics

Watch the presentation | View the presentation slides (PDF)

TERADACTAL: A Scalable Divide-And-Conquer Approach for Constructing Large Phylogenetic Trees (almost) without Alignments

Project PI: Tandy Warnow, University of Illinois at Urbana-Champaign

Presented by: Erin Molloy, University of Illinois at Urbana-Champaign

Abstract: In this talk, we present TERADACTAL, a novel divide-and-conquer approach for estimating large evolutionary trees. Phylogeny estimation from sequence data typically involves two computationally intensive tasks: constructing a multiple sequence alignment and then building a maximum likelihood tree. Using this two-phase approach to analyze ultra-large datasets requires enormous computational resources (weeks to years of CPU time and hundreds of GBs of storage). TERADACTAL differs from prior approaches by avoiding (1) alignment estimation on the full dataset, (2) maximum likelihood tree estimation on the full dataset, and (3) supertree estimation (which under a variety of optimization criteria is NP-hard). This is accomplished through our new highly accurate and parallelizable technique for merging trees built from subsets of the data. Finally, we present the results of a simulation study (performed on Blue Waters) demonstrating that TERADACTAL achieves similar accuracy to the leading two-phase methods. Thus, TERADACTAL provides a breakthrough in parallel algorithm design for ultra-large phylogeny estimation.

Field of Science: Biosciences

Watch the presentation | View the presentation slides (PDF)

A New PPMstar Code for Blue Waters and its Follow-On System

Project PI: Paul Woodward, University of Minnesota

Abstract: We have been developing a new PPMstar code to support our research in convective boundary mixing in stellar interiors and its impacts upon stellar evolution and nucleosynthesis. The new code uses a relatively clean F90 expression that can be translated into CUDA by a python tool, enabling GPUs to be exploited if desired. The new code also is designed around an ultimate 3-level AMR capability, which is still in development. The code's new MPI messaging structure not only permits AMR to be introduced, but it also enables a resilient form of execution. Immediate visualization and production of partially analyzed data products as the code runs greatly reduce the amount of human labor. The code's redesign to enable the dynamic load balancing that AMR requires has the side effect of making very flexible code execution possible. These aspects of the code design and experience with its use will be presented.

Field of Science: Astronomy and Astrophysics

High Resolution Dynamical Model Analyses and Projections of Climate Change and Air Quality Patterns over the United States

Project PI: Donald J. Wuebbles, University of Illinois at Urbana-Champaign

Abstract: This study is aimed at a) projections of extreme temperatures and shifting precipitation patterns over the continental United States for the 21st century using an ensemble of high spatial resolution dynamically downscaled model simulations with different boundary conditions and b) studies of effects of global changes in climate and emissions on U.S. air quality, focusing on fine particulate matter and ozone, projecting their future trends and quantifying key source attribution. For the first part, the downscaling uses the WRF model at a spatial resolution of 12 km with boundary conditions based on three different global climate models. For the second part, we use a dynamic prediction system that couples a global climate-chemical model with regional climate-air quality models, to determine individual and combined impacts of global climate and emissions changes on U.S. air quality from the present to 2050 under multiple scenarios. Blue Waters is essential to our project goals.

Field of Science: Atmospheric and Climate Science

Watch the presentation | View the presentation slides (PDF)

 

Resolution Effects and Local Averaging in Turbulence Simulations up to 4 Trillion Grid Points

Project PI: P.K Yeung, Georgia Institute of Technology

Abstract: Petascale architectures such as Blue Waters have enabled turbulence simulations at extreme core counts and very high grid resolution, with a primary science driver being the study of intermittency at high Reynolds number. We take stock of these advances but also investigate the effects of resolution in both time and space, as well as aliasing errors which may affect quality of the numerical solution where localized events of high amplitude are concerned. In particular, it appears essential to minimize aliasing errors by using a time step smaller than previously thought, or (at greater cost) remove them entirely by truncating at a wavenumber cutoff lower than usually practiced. At the same time, statistics of dissipation rate averaged over domains of intermediate size in three-dimensional space are more robust, and have been led to new understanding. Challenges in maintenance of a Petabyte-sized computational database are briefly addressed.

Field of Science: Fluid Systems

Watch the presentation | View the presentation slides (PDF)

 

Energetics of a Model Ocean Basin with Surface Buoyancy and Wind Forcing

Project PI: Barbara Zemskova, University of North Carolina at Chapel Hill

Abstract: We present numerical results for rotating, wind-forced horizontal convection as a simple model for the Southern Ocean branch of the Meridional Overturning Circulation (MOC). The flow is driven by differential buoyancy forcing applied along the horizontal surface, with surface cooling at one end (to represent the pole) and surface warming at the other (to represent the equatorial region) and a zonally re-entrant channel to represent the Antarctic Circumpolar Current (ACC). Zonally-uniform surface wind forcing is applied with a similar pattern to the westerlies and with varying magnitude relative to the buoyancy forcing. The problem is solved numerically using a 3D DNS model based on a finite-volume solver for the Boussinesq Navier-Stokes equations with rotation. The overall dynamics, including large-scale overturning, baroclinic eddying, turbulent mixing, and resulting energy cascades are investigated using the local Available Potential Energy framework introduced in [Scotti and White, J. Fluid Mech., 2014]. We find that both the magnitude and shape of the zonal wind stress profile are important to the spatial pattern of the overturning circulation. However, perhaps surprisingly, the essential circulation and the energetics in cases with wind are similar to the base case with buoyancy forcing alone, suggesting that surface APE generation by the buoyancy forcing is dominant in setting the circulation.

Field of Science: Fluid Systems

Watch the presentation | View the presentation slides (PDF)

Blue Waters is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) the State of Illinois, and the National Geospatial-Intelligence Agency.

Contact Blue Waters Team with questions regarding this page.
Copyright 2024 Board of Trustees of the University of Illinois. All rights reserved. Web privacy notice.