2017 Blue Waters Symposium: Abstracts and Presentations
Quantifying the Contributions of Root System Function to Forest Drought Resilience in the Amazon Rainforest
Project PI: Elizabeth Agee, University of Michigan
Abstract: Plant water transport is a dominant mechanism for the cycling of water from the land surface to the atmosphere, but has limited implementation in large scale models of land surface hydrology. As drought events become more frequent and severe in forested regions, it is important to understand the response of vegetation to soil water limitations. Using a modified version of PFLOTRAN, a highly parallelized model which simulates flow and transport in variably saturated soils, we are able to quantify three-dimensional water uptake for heterogeneous, overlapping root systems. This work highlights the inter- and intra-species responses of forest trees to seasonal droughts in an area of the largely biodiverse Amazon rainforest where a multitude of hydraulic strategies exist. This work will help quantify the contributions of root system hydraulics to forest resilience and resistance to drought and improve our understanding of the mechanisms underlying forest fate under changing precipitation regimes.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
The Molecular Mechanism of Transport Selectivity across the Nuclear Pore Complex
Project PI: Aleksei Aksimentiev, University of Illinois at Urbana-Champaign
Presented by: David Winogradoff, University of Illinois at Urbana-Champaign
Abstract: The nuclear pore complex (NPC) regulates the transport of all RNA and proteins across the nuclear envelope. The key barrier to the nuclear transit of such biomolecules is the disordered central mesh of the NPC, composed of nucleoporins, or Nups. Complementing the experimental work conducted by our collaborators in the lab of C. Dekker (TU Delft), we are performing molecular dynamics (MD) simulations on Blue Waters to characterize the structural and electrical properties of the NPC diffusion barrier with atomic precision. In our on-going project, we have already shown the ability of all-atom MD to reproduce key experimental results of biomimetic NPCs. The simulations performed on Blue Waters will offer new insight into the physical mechanism of nuclear transport, with important implications for several human diseases and the development of novel gene therapy techniques.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Molybdenum Disulfide (MoS2) as a Novel 2D Nanoporous Membrane for Water Desalination
Project PI: Narayana R. Aluru, University of Illinois at Urbana-Champaign
Presented by: Mohammad Heiranian, University of Illinois at Urbana-Champaign
Abstract: Power generation from the chemical potential gradient between seawater and fresh water is a clean method of harvesting energy. Under a chemical potential gradient resulting from a salt gradient, power is generated when electrolyte is driven through selective (charged) nanoporous membrane. 2D materials are preferable as flux scales inversely with the thickness of the membrane. Recent advances in nanotechnologies have paved the road for fabrication of different nanoporous membranes. Here, we demonstrate single-layer molybdenum disulfide (MoS2) as a novel 2D membrane for power generation. By performing extensive molecular dynamics simulations, we observe induced currents from a salt gradient with exceptionally high power density of ~106 W/m2. We further show that the resulting high power density is due to the atomic thinness of the membrane and it exponentially decreases for multilayer membranes. Compared to thicker membranes, the ionic mobility inside the nanopore is significantly stronger leading to higher conductance and power.
Field of Science: Materials Science
Watch the presentation | View presentation slides (PDF)
Petascale Quantum Simulations of Nanosystems and Biomolecules
Project PI: Jerry Bernholc, NC State University
Abstract: Petascale computation enables realistic simulations of complex, nanostructured materials using only the fundamental laws of quantum mechanics. We develop, use and distribute the petaflops-capable RMG (Real-space Multigrid) suite of codes for quantum simulations of materials and devices. It solves the density-functional-theory equations on real-space grids, which allow for natural parallelization via domain decomposition. RMG is a GPU-capable, cross platform open source package that has been used in studies of a wide range of systems, including semiconductors, biomolecules, and nanoscale electronic devices. It is installed on Blue Waters as community code. New versions are regularly released at www.rmgdft.org, including binaries for Linux, Windows and Macintosh systems. We will describe recent applications focusing on (i) nanodevice structures used for molecular detection, such as glucose sensors, (ii) larger structures for monitoring DNA replication, and (iii) devices for beyond the Moore's law era.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
A Lagrangian Perspective on the Floating Sargassum Ecosystem in the Atlantic
Project PI: Maureen T. Brooks, University of Maryland Center for Environmental Science
Abstract: The floating macroalgae Sargassum is the foundation of a mobile ecosystem in the Gulf of Mexico and Atlantic Ocean. To understand its seasonal cycle and variability, a coupled model system combining ocean circulation, biogeochemical, Lagrangian particle, and Sargassum physiology models was run on Blue Waters at 1/12° resolution. Model experiments investigating vegetative propagation of Sargassum show that explicitly including this reproductive mode reduces model bias. Connectivity analysis and particle seeding experiments have highlighted the western Gulf of Mexico and Western Tropical Atlantic as the regions most important to Sargassum seasonal variability. Closer examination of these two regions using Lagrangian Coherent Structure analysis highlights mesoscale eddies and fronts which both influence the rate and timing of particle export and can alter the nutrient conditions available for Sargassum growth. Understanding these interacting physical and biogeochemical drivers will be necessary to predict costly Sargassum beaching events in the coming years.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
The Role of Cosmic Rays in Isolated Disk Galaxies
Project PI: Iryna Butsky, University of Washington
Abstract: Cosmic ray (CR) feedback in cosmological simulations is integral to creating robust models of stellar feedback and reproducing galactic structure that is consistent with observations. Unlike thermal gas, relativistic CRs retain pressure under adiabatic expansion better and cool on significantly longer time scales. Because CRs stream along magnetic field lines, it is necessary to employ fully anisotropic, magnetohydrodynamic (MHD) treatment to capture their behavior. In this talk I will describe the implementation of anisotropic CR physics in ENZO, a publicly available cosmological MHD simulation code. I will then discuss the effects of CRs in simulations of isolated disk galaxies in which supernovae supply both CRs and magnetic fields, focusing on the galactic magnetic dynamo and the pressure distribution in the circumgalactic medium.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
How Function Shapes Dynamics in Evolution of Proteins
Project PI: Gustavo Caetano-Anollés, University of Illinois at Urbana-Champaign
Presented by: Fizza Mughal, University of Illinois at Urbana-Champaign
Abstract: Protein loops offer flexibility and function to the three-dimensional structure of proteins. Flexibility is an evolutionarily conserved feature that impacts protein dynamics. Here we aim to investigate the presence (or absence) of specific motions corresponding to specific molecular functions, placing motions within an evolutionary perspective. In our current Blue Waters allocation, we have completed molecular dynamics simulations of 116 protein loops present in protein structural domains of metaconsensus enzymes. These enzymes are derived from a consensus of three well-established methods in comparative bioinformatics that involve molecular sequences, structures and biochemical reactions. Using the experience of a Blue Waters analysis of the loops of aminoacyl tRNA synthetase enzymes, we are here extending the network analyses of simulation of residue motions using community structures. In addition, we are constructing a dynamics space, a "dynasome," which we want to map to a chronology of protein domain evolution. We have modified our analysis approach by searching for correlations between community structure metrics and contact order, which capture the evolutionary essence of protein flexibility.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
TRPA1 as a Drug Target for the Treatment of Chronic Pain
Project PI: Vincenzo Carnevale, Temple University
Presented by: Eleonora Gianti, Temple University
Abstract: The superfamily of Transient Receptor Potential (TRP) channels is responsible for conducing cations through the cell membrane in response to a variety of stimuli, amongst which the binding of chemical compounds with agonistic or antagonistic properties. TRPA1, a member of the TRP superfamily, is predominantly expressed in sensory neurons; it is involved in a variety of cellular functions, such as nociception, sensitivity to chemical irritants, itch and pain sensation. Therefore, TRPA1 has been established as a promising drug target for the treatment of chronic pain. We use long time-scale molecular dynamics (MD) simulations to explore binding of ligands to TRPA1, with the ultimate goal of characterizing this pharmacological target for structure based drug design. The available electron microscopy structure of human TRPA1, determined in complex with the antagonists A-967079 and HC-030031, shows insufficient resolution density to determine the atomic coordinates of several structural loops and of the bound ligands. We generated a refined structure of the human TRPA1 channel in a complex with A-967079, and verified that the binding mode of the ligand is in agreement with experimental observations. Molecular docking and MD simulations of TRPA1 in complex with A-967079 and other ligands reveal the molecular underpinnings of TRPA1 modulation, thereby laying the basis for the design of novel anti-pain drugs.
Field of Science: Biosciences
AMBER Biomolecular Dynamics Simulations Provide Novel Insight into Nucleic Acid Structure, Dynamics and Function
Project PI: Thomas Cheatham III, University of Utah
Abstract: Utilizing the Blue Waters petascale resource with AMBER molecular dynamics simulation we can converge the conformational ensembles of various nucleic acid motifs. This provides the means to assess and validate the molecular potentials or "force fields" and also to provide detailed insight into nucleic acid structure and dynamics. Recent work focuses on the interaction of various copper containing compounds that interact with DNA that can interact via minor groove binding, intercalation, or base pair eversion, depending on the functional modifications. The extended sets of MD simulation help explain and are consistent with experimental results obtained by our lab. Simulations on various RNA motifs, including atomistic models derived from coarse-grained structure predictions, show improved structural agreement when the simulations are biased to maintain well-known motif structures with selective restraints. Altogether, the simulation approaches provide better insight into nucleic acid structure, dynamics and function.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Enabling Redistricting Reform: A Computational Study of Zoning Optimization
Project PI: Wendy K. Tam Cho, University of Illinois at Urbana-Champaign
Abstract: Political redistricting, a well-known problem in political science and geographic information science, can be formulated as a combinatorial optimization problem, with objectives and constraints defined to meet legal requirements. The formulated optimization problem is NP-hard. We have developed a scalable evolutionary computational approach utilizing massively parallel high performance computing for political redistricting optimization and analysis at fine levels of granularity. Our computational approach is based in strong substantive knowledge and deep adherence to Supreme Court mandates. Experimental results demonstrate desirable effectiveness and scalability of our approach (up to 131,000 processors) for solving large redistricting problems, which enables substantive research into the relationship between democratic ideals and phenomena such as gerrymandering. Our early research has been well-received on both the technical and substantive fronts. Our current focus is on further enhancing the efficiency of the optimization algorithm as well as to utilize our computational tool to embark on a focused analysis of how best to institute redistricting reform in the United States.
Field of Science: Social Sciences
MELD: A Physics-based Approach to Predict Protein Shapes and Interactions
Project PI: Ken Dill, SUNY Stony Brook
Presented by: Emiliano Brini, SUNY Stony Brook
Abstract: Proteins are the nano-machines that perform most biological functions. Pharmaceuticals typically operate by enhancing or altering proteins activity. The 3D shape of a protein dictates its ability to properly function and to interact with its environment. Several methods have been proposed to computationally predict the shape of a protein and its interaction with other molecules, but none has been completely successful yet. Physics based search methods like Molecular Dynamics are in principle well suited for such endeavors. Usually, they are too slow to effectively find single global optima on a complex, frustrated landscape—like the protein conformational space. MELD (Modeling Employing Limited Data) tackles this weakness by combining physics with knowledge and experimental data. We tested this approach in the worldwide competitions of structure prediction (CASP) and of protein-protein interactions prediction (CAPRI). We also broke new ground to predict the affinity of a protein with several drug candidate molecules.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
The Next Frontier of Massive Galaxies and Quasars at the Cosmic Dawn
Project PI: Tiziana Di Matteo, Carnegie Mellon University
Abstract: Upcoming observations from a range of current and planned instruments, from the successor to Hubble, The James Webb Telescope, to the Large Synoptic Survey Telescope, to the WFIRST Satellite will have the opportunity to reach the first galaxies and black holes that form in the early universe. This will open up the study of galaxies and quasars to the type of statistical studies that have made modern cosmology a precision science. Our team has led the development of the cosmological code MP-Gadget adapted to Petascale supercomputers to understand how supermassive blackholes and galaxies formed, from the smallest to the rarest and most luminous. We will report on the extension of the BlueTides cosmological simulation, with an unprecedented volume and resolution, to cover the evolution of the first billion years of cosmic history. The goal is to significantly increase the scientific impact of this calculation to the community as well as using it as a path-finder for developing methods/calculations for future cosmological hydrodynamical simulations.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Innovative Ab Initio Symmetry
Project PI: Jerry Draayer, Louisiana State University
Presented by: Tomas Dytrych, Louisiana State University
Abstract: We use the Blue Waters system to carry out large-scale first-principle modeling of light and medium-mass nuclei, including short-lived isotopes not yet accessible to experiment but key to understanding astrophysical processes that shape our universe and which are the focus of current and next-generation rare isotope experimental facilities. In our studies, we utilize an innovative theoretical framework for first-principle modeling of nuclear structure, dubbed symmetry-adapted no-core shell model, which is implemented as a hybrid MPI/OpenMP parallel computer code that scales well for hundreds of thousands of processors. The extremely large memory and computational power of the Blue Waters system allows us to model the intricate dynamics of atomic nuclei with a precision hitherto inaccessible to theory, thereby addressing some of the long-lasting challenges of importance to nuclear theory and experiment, as well as astrophysics.
Field of Science: Physics
Watch the presentation | View presentation slides (PDF)
Direct Numerical Simulation of Pressure Fluctuations Induced by Supersonic Turbulent Boundary Layers
Project PI: Lian Duan, Missouri University of Science and Technology
Abstract: Understanding the physics of the pressure fluctuations induced by turbulent boundary layers is of major theoretical and practical importance. From a practical point of view, an in-depth knowledge of the nature of boundary-layer-induced pressure fluctuations is essential to the structural design of launch vehicles and to the generation of background noise in wind-tunnel facilities. From a theoretical point of view, pressure fluctuations are an important ingredient in turbulence as they appear in statistical correlations such as the pressure-strain correlation terms that redistribute turbulence among different components of fluctuating velocity.
In the presentation, I will talk about our use of the Blue Waters supercomputer to conduct direct numerical simulations that provide the basis for an in-depth understanding of the global pressure field induced by turbulent boundary layers at supersonic speeds. I will also discuss about our experience of using parallel HDF5 for efficient I/O and our effort to hybridize OpenMP and MPI to maximize node-level parallelism on the Blue Waters.
Field of Science: Engineering
Watch the presentation | View presentation slides (PDF)
Quantum Monte Carlo Database: Automated Calculations of the Properties of Solids and Point Defects
Project PI: Elif Ertekin, University of Illinois at Urbana-Champaign
Abstract: We will present our recent efforts highlighting the use of the quantum Monte Carlo method to enable high-accuracy calculations of the properties of solid materials and point defects in semiconductors. Quantum Monte Carlo (QMC) methods comprise a suite of stochastic approaches for the direct solution of the many-body Schrodinger equation. Despite their longstanding record of highly-accurate, benchmark quality results, their computational cost has historically limited their widescale adoption for the simulation and modeling of real materials. We have been able to take advantage of Blue Waters to help extend the method now to these challenging systems. With performance showing a near-linear scaling on Blue Waters, we have been able to calculate formation enthalpies of a large class of approximately twenty solid materials, as well as several detailed studies of point defects in several of these solids. Our results have been compiled into the quantum Monte Carlo database, an online shared community resource that we are developing for the benefit of the QMC user community.
Field of Science: Materials Science
Large Eddy Simulation of Sediment Transport and Hydrodynamics at River Bifurcations using a Highly Scalable Spectral Element Eased CFD Solver
Project PI: Marcelo H. Garcia, University of Illinois at Urbana-Champaign
Presented by: Som Dutta, University of Illinois at Urbana-Champaign
Abstract: ?Bifurcations are fundamental features of all river systems. The current study focuses on a specific class of bifurcations called diversions, where one source channel splits into two channels: a main channel and a lateral branch outlet. Experiments have shown that the distribution of near-bed sediment between the downstream channels is not proportional to the flow distribution, with a disproportionately higher amount of sediment going into the lateral channel. The current study investigates the mechanisms behind this phenomenon through high-resolution flow and sediment transport simulations using a highly scalable spectral element based incompressible Navier-Stokes solver, Nek5000.
Large Eddy Simulations (LES) and Direct Numerical Simulations (DNS) of the flow in idealized diversions of different configurations have been conducted with sediment being modeled as Lagrangian particles. The simulation conditions are comparable to the laboratory experiments, which make these simulations one of the largest and most complex to date in the area of river mechanics. Blue Waters provides the computing resources to undertake these highly-resolved simulations. Currently, simulations have been conducted for up to 240 million computational points, with strong scaling being shown up to 32,768 MPI ranks. A semi-implicit Lagrangian particle algorithm has been developed for simulating the sediment, and hydrodynamic and particle simulations have also been completed for a range of Reynolds number, flow splits and bifurcation angles. The simulations have provided a clear picture of the mechanism in play, with the simulation results agreeing with the experiments. We have currently used almost all our allocated time on BW, and are in the process of finishing the last few simulations. The results have not only helped us to understand the underlying mechanism, it will be used to identify a simple physics-based theoretical model for the phenomena of non-linear distribution of near-bed sediment.
A better understanding of the aforementioned phenomena will help in efficient design of river diversions, which among uses like navigation and flood-mitigation have also been put forth as a solution for reclaiming deltas going under the sea due to increasing sea level. A prime example is the Mississippi delta, for which different diversion designs are being explored currently for diverting water and sediment from Mississippi River. This study will also help in understanding of the short and long-term geomorphological evolution of river bifurcations, thus furthering the state of art in the field of river mechanics.
Field of Science: Engineering
Watch the presentation | View presentation slides (PDF)
Extracting Annual Crop Irrigation Fields at 30 meters using Landsat Time Series and Deep Learning
Project PI: Kaiyu Guan, University of Illinois at Urbana-Champaign
Presented by: Yunan Luo, University of Illinois at Urbana-Champaign
Abstract: Accurate characterization on irrigation of croplands have become one of the important information of agriculture, earth science, and global change researches. However, the dominated approaches of obtaining irrigation information so far are mainly based on the labor-intensive and time-consuming statistics of county-level surveys, which is difficult to extend to larger scale and finer resolution. Existing satellite-based approaches rely on coarse resolution data and have low accuracy. In this talk, we present our novel method that classifies irrigated and non-irrigated croplands using the Landsat satellite time-series images. With the deep learning based models and the Blue Waters, we trained our deep neural network model with multi-year time-series data of Landsat satellite images, which captures the temporal and spatial patterns of cropland changes. Our method provides large-scale predictions of irrigation on the whole Nebraska state. The results demonstrate that our method can provide irrigation prediction with high coverage, high resolution and high accuracy.?
Field of Science: Biosciences
View presentation slides (PDF)
Preserving Intra-patient Variance Improves Phylogenetic Inference of HIV Transmission Networks
Project PI: August Guang, Brown University
Abstract: Phylogenetic analyses of HIV sequences across patients are used to infer features of the transmission network, such as to identify transmission clusters. Current methods rely on consensus sequences that discard intra-patient variation, ignoring potentially informative data that can be derived from Next-Generation Sequencing (NGS) platforms. We developed a new phylogenetic approach that uses the probabilistic aligner HMMER to generate per-site nucleotide frequency distributions from NGS data, generates a population of "synthetic" sequences that sample from this distribution, and then conducts a phylogenetic analysis that includes multiple synthetic sequences per patient to capture intra-patient sequence variation. On a dataset of adults diagnosed with HIV in 2013 as well as on simulated reads generated from transmission networks, our new phylogenetic method produced a more highly resolved virus phylogeny compared to commonly used methods that rely on consensus sequences. Enhanced transmission cluster detection has the potential to improve HIV transmission prevention.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Density Functional Theory Predictions of the Electronic Density of Diatomic Molecules in a Chemically Relevant Region
Project PI: Sharon Hammes-Schiffer, University of Illinois at Urbana-Champaign
Presented by: Kurt Brorsen, University of Illinois at Urbana-Champaign
Abstract: Density functional theory (DFT) in the Kohn-Sham formalism requires the approximation of the exchange-correlation functional. By examining the predicted DFT density of a test set of atoms, recent calculations have claimed that DFT functional development has strayed from approximating the exact functional and while energetics for functionals developed since the early 2000s have improved, the errors in density have trended upwards. In the present study, we examine the density and atomization energy of 90 functionals on a test set of diatomics that should give a better indication of the functionals ability to predict density for chemical systems. For functionals of hybrid generalized gradient approximation (hGGA) type, a decoupling of the errors in density and atomization energy occurred beginning in the early 2000s. For non-hGGA functionals, no such decoupling occurs. For the Minnesota family of functionals, many of the functionals predict a more accurate density for diatomics than for atomic systems.
Field of Science: Chemistry
Watch the presentation | View presentation slides (PDF)
Policy Responses to Climate Change in a Dynamic Stochastic Economy
Project PI: Lars Hansen, University of Chicago
Presented by: Ken Judd, Hoover Institution
Abstract: Integrated assessment models used to predict the human impact on the climate generally ignore uncertainty in future economic activity. Statistical analysis of economic data over the last century shows that economic growth rates are very persistent, and that the two-sigma range of GDP in 2100 is greater than ten. Our model, DSICE, merges statistical estimates of economic growth with DICE, the standard IAM benchmark, to create essentially a seven-dimensional parabolic partial differential equation. We solve it over a 600-year horizon, and can do this because we combine Blue Waters with efficient numerical methods for quadrature, approximation, and optimization. The results significantly affect predictions. First, the chances of high levels of warming is substantially greater. Second, meeting a target, such as two degrees, requires substantially greater mitigation now than implied by the standard models.
Field of Science: Social Science
Watch the presentation | View presentation slides (PDF)
Codon-based Models of Molecular Evolution and Bayes Factors Reveal Genome-wide Conflict Deep in the Amphibian Tree of Life
Project PI: Paul M. Hime, University of Kentucky
Abstract: Advances in genome sequencing and new statistical models for phylogenetic inference are providing unprecedented opportunities to unravel the mysteries of the Tree of Life. Yet, as phylogeneticists push these boundaries, it is becoming clear that different regions of the genome may provide vastly different resolutions of evolutionary relationships and that the magnitude of this signal may differ dramatically across genes. I use Bayes factor tests of topology to demonstrate that deep amphibian relationships are obscured by gene tree-species tree discordance and that the magnitude of phylogenetic information supporting these different inter-ordinal hypotheses varies greatly across amphibian genomes. These tests of topology involve Metropolis-coupled Markov chain Monte Carlo simulation across rugged likelihood surfaces, necessitating complex and highly-parameterized analyses which the use of Blue Waters has enabled. These results provide new perspectives for parsing phylogenetic information content from stochastic noise at deep phylogenetic divergences.
Field of Science: Biological Sciences
View presentation slides (PDF)
Multiscale Simulations of Complex Fluid Rheology
Project PI: Michael P. Howard, Princeton University
Abstract: Complex fluids, readily encountered in biology, consumer products, and industrial processing, are multicomponent mixtures that exhibit a rich variety of flow behaviors. A classic example is the cornstarch-water "oobleck" mixture, which acts like a liquid when pressed slowly but can thicken enough to support the weight of a person when struck quickly. Such peculiar macroscopically observable flow properties of complex fluids are fundamentally controlled by microscopic molecular structures. Computer simulations are ideal tools for studying this nontrivial and difficult-to-predict relationship; however, performing simulations at physically relevant scales presents a considerable challenge. I will present our development of the first massively parallel, open-source implementation of the multiparticle collision dynamics method for graphics processing units. Our software bridges the important gap between atomistic simulations and continuum fluid mechanics and can fully exploit computational resources from desktop workstations up to the scale of Blue Waters.
Field of Science: Fluid Systems
Watch the presentation | View presentation slides (PDF)
Understanding Hydrogen Storage in Metal Organic Frameworks from First Principles
Project PI: Sohrab Ismail-Beigi, University of Illinois at Urbana-Champaign
Abstract: First principles molecular dynamics approaches permit one to simulate dynamic and time-dependent phenomena in physics, chemistry, and materials science without the use of empirical potentials or ad hoc assumptions about the interatomic interactions since they describe electrons, nuclei and their interactions explicitly. We present an update on our use of the OpenAtom ab initio molecular dynamics software on Blue Waters to describe the dynamics of hydrogen inside of complex metal organic framework (MOF) materials which are of interest for hydrogen storage. In addition to highlighting computed results and understanding gained, we describe our collaborative efforts in developing and enhancing the OpenAtom open source ab initio density functional software package based on plane waves and pseudopotentials. OpenAtom takes advantage of the Charm++ parallel framework to enable massively parallel scaling.
Field of Science: Physics
Watch the presentation | View presentation slides (PDF)
The Power Of Many: Runtime Environment for Executing Dynamic Heterogeneous Workloads
Project PI: Shantenu Jha, Rutgers University
Abstract: High-performance computing systems such as Blue Waters have been optimized to support mostly monolithic and single-job workloads. Many important scientific applications however, have workloads comprised of multiple heterogeneous tasks that are not known in advance and temporally vary in the resources required. The default runtime environment is often unable to meet the requirements of such workloads. We will discuss how RADICAL-Pilot has been designed to serve as a runtime system that meets the requirements of heterogeneous and dynamic workloads on Blue Waters. We will discuss how it is engineered to spawn up to 100 tasks/second and maintain a steady state of up to 8,000 concurrent tasks. We discuss the architecture of the RADICAL-Pilot agent and how RADICAL-Pilot is interfaced with the ORTE layer to achieve necessary performance. We will conclude with a discussion of several scientific questions under way using RADICAL-Pilot on Blue Waters.
Field of Science: Computer Science
Watch the presentation | View presentation slides (PDF)
Advances in Physics-based Modeling of High-frequency Ground Motions and Probabilistic Seismic Hazard Analysis using Blue Waters
Project PI: Tom Jordan, University of Southern California
Presented by: Christine Goulet, SCEC, University of Southern California
Abstract: Over the past two decades, multidisciplinary groups of earth scientists, engineers, and computer scientists affiliated with the Southern California Earthquake Center (SCEC) have used high-performance computing to advance knowledge of earthquake system science. SCEC scientists develop and apply the best available geoscientific understanding of faulting, dynamic rupture, and wave propagation processes, together with state-of-the-art computation techniques, to produce the next generation of physics-based seismic hazard models. Information from such models can help significantly reduce uncertainty in seismic hazard assessments, an essential pre-requisite toward seismic risk reduction. The models are validated against recordings from past earthquakes, and physics principles are used for the extrapolation of predictions for events that are expected to occur in the future. In our presentation, we summarize recent progress in the scientific understanding of earthquake processes achieved through these high-end simulations on Blue Waters. We focus the discussion on (i) the validation of ground motions with increasingly higher frequencies using realistic, complex, physical models, and on (ii) the latest computations from CyberShake, our physics-based platform for conducting probabilistic seismic hazard analyses.
Field of Science: Earth Sciences
Watch the presentation | View presentation slides (PDF)
Understanding the Role of Lipid Membranes in Influenza Viral Entry
Project PI: Peter Kasson, University of Virginia
Presented by: Jennifer Hays, University of Virginia
Abstract: Influenza virus enters cells via a process of membrane fusion with a host cell membrane. Viral membrane fusion is mediated by the hemagglutinin protein, but experimental data have also shown that the viral and host membranes play an important role in determining the outcome of fusion. We have previously shown via electron cryo-microscopy that viral membrane cholesterol composition alters the spatial organization of the hemagglutinin protein and also changes fusion kinetics. Here, we discuss large-scale simulations of fusion and membrane organization to better understand the results of these experiments and also single-virus fusion experiments by our laboratory and others. These simulations yield new insight into the role of the membrane in viral entry.
Field of Science: Biosciences
Molecular Dynamics Simulations of Paracellular Transport Mechanism
Project PI: Fatemeh Khalili-Araghi, University of Illinois at Chicago
Abstract: Claudins are one of the major components of tight junctions that control the permeability and selectivity of paracellular channels in epithelia. Paracellular pathways are oriented parallel to the membranes and are formed by the assembly of multiple claudins in each cellular membrane, which interact with each other to form these macromolecular structures. Here we present an atomic model of claudin pores that span two parallel membranes and form continuous strands of tight junction. Simulations of ion transport through claudin pores in this model showed formation of size-and charge-selective ion channels that resemble classicalion channels. The model was validated against experiments by prediction of site-specific mutations that changed the charge-selectivity of the pore and created an anion-selective channel.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
High Fidelity Blade-resolved Wind Farm Simulations
Project PI: Andrew Kirby, University of Wyoming
Abstract: Until recently, blade-resolved simulations of wind energy applications have been considered too expensive and complex to be practical. Reduced fidelity modeling of wind farms often use actuator disks to represent turbines. To properly engineer wind energy applications, blade-resolved aerodynamics are needed to capture the complex physics on turbine blades and to capture the unsteady flow interactions between wind turbines. Wind turbine aeromechanics traverse several magnitudes of spatial and temporal scales. The approach taken here utilizes computational fluid dynamics flow solvers combined in a dynamic overset grid assembler. Efficient flow solver technologies including adaptive methods and high-order accurate discretizations are used to effectively resolve the wide range of spatial and temporal scales. The use of meso-scale/micro-scale coupling introduces physical simulation environments by introduction of atmospheric boundary layer turbulence. Studies of single/double turbine configurations are performed as well as a full wind farm.
Field of Science: Engineering
Watch the presentation | View presentation slides (PDF)
Nanoelectronic Modeling on Blue Waters with NEMO5
Project PI: Gerhard Klimeck, Purdue University
Presented by: Xinchen Guo, Purdue University
Abstract: Transistor size has reached a countable number of atoms. To account for these effects transistors needs to be modeled as being composed of atoms and not in the continuum. Additionally, quantum effects such as tunneling and confinement need to be included for an accurate model. For reproducible devices, a device engineer needs to know the uncertainty associated with device variations originating from disorder such as impurities and roughness. NEMO5 takes advantage of Blue Waters for large scale simulations including: 1) accessing the uncertainty due to roughness and electron-phonon scattering 2) developing and conducting LED simulations including quantum effects and 3) development of an accurate compact model for grain boundaries in copper. We are planning to expand several nanoHUB applications based on the work reported here and enable their usage in education and research.
Field of Science: Materials Science
Watch the presentation | View presentation slides (PDF)
Cumulus Entrainment in Convective Clouds and Storms
Project PI: Sonia Lasher-Trapp, University of Illinois at Urbana-Champaign
Presented by: Daniel Moser, University of Illinois at Urbana-Champaign
Abstract: A long-standing problem in meteorology has been to understand how quickly entrainment, the introduction of dry air from outside the cloud to its interior, dilutes cumulus clouds. Entrainment is a critical limiting factor for thunderstorm development and concomitant rainfall, hailfall, and/or severe winds. Its quantification with field observations is impractical, and was heretofore also limited computationally, until the advent of high-performance computers like Blue Waters. To correctly assess entrainment in storms, cloud motions on scales of tens of meters need to be resolved, over domains hundreds of kilometers in width. We will present results from a variety of novel projects quantifying entrainment and its effects, ranging from individual clouds and the initial stages of thunderstorms, to lines of clouds and storms, addressing (i) how entrainment can change during their development (and alter that development), (ii) the impact of entrainment upon rainfall, and (iii) implications for weather forecasting and climate modeling.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
Core-collapse Supernovae through Cosmic Time
Project PI: Eric Lentz, University of Tennessee, Knoxville
Abstract: From the earliest stars in the Universe to the present, explosions of massive stars (8 or more times the solar mass) known as core-collapse supernovae (CCSNe), have contributed to the physical and chemical evolution of galaxies. Significant progress has been made recently by several groups in computing axisymmetric (2D) simulations of CCSNe that include all of the relevant known physics for many pre-explosion progenitor stars. I will present several recent and ongoing models in 3D computed on Blue Waters with our detailed Chimera code. Emphasis will be placed on our efforts to understand the first CCSN in the universe and to resolve the relevant physical scales in 3D.
Field of Science: Astronomy & Astrophysics
Investigating the Origin of Seismic Anisotropy using Data-oriented Geodynamic Models
Project PI: Lijun Liu, University of Illinois at Urbana-Champaign
Presented by: Jiashun Hu, University of Illinois at Urbana-Champaign
Abstract: Seismic anisotropy records both the past and present deformation of the Earth. In the mantle, seismic anisotropy is mainly attributed to the lattice preferred orientation of mineral fabrics, caused by the shear deformation of mantle flow. However, questions remain in quantitatively explaining the observed seismic anisotropy and its relation to the evolving mantle flow field.
Here, we construct large-scale mantle convection models with data assimilation in South America to investigate the relation between seismic anisotropy and mantle deformation. Each model takes more than 100,000 core-hours to finish, and we performed ~100 such models. The mantle flow field is then taken to forward predict seismic anisotropy with a Fortran code that is implemented with hybrid MPI and OpenMP, which takes more than 5,000 core-hours.
Results show that the seismic anisotropy beneath the subducting and overriding plates originates from different mechanisms: the former arises from plate-motion-induced Couette flow and the latter from subduction-controlled Poiseuille flow. These results also shed new light on the structure and temporal evolution of continental lithospheres.
Watch the presentation | View presentation slides (PDF)
Three-dimensional Nature of Magnetic Reconnection X-line in Asymmetric Current Sheets
Project PI: Yi-Hsin Liu, NASA-Goddard Space Flight Center
Abstract: Magnetic reconnection is the process whereby a change in topology of magnetic field lines allows for a rapid conversion of magnetic energy into thermal and kinetic energy of the surrounding plasma. This physical process plays a key role in many astrophysical and laboratory contexts, ranging from magnetospheric substorms, solar eruptions, sawtooth crashes in fusion devices and potentially the super-flares in Crab Nebula. At Earth's magnetopause, the plasma and magnetic field conditions are different on the two sides of current sheet. In such an asymmetric configuration, it is unclear whether there is a simple principle to determine the orientation of 3D reconnection x-line (i.e., where the field lines break and energy dissipation occurs). Knowing the local physics mechanism that controls the orientation is especially important in predicting the location of reconnection in a global context. Petascale simulations at Blue Waters provide the opportunity to carefully study this 3D nature of magnetic reconnection.
Field of Science: Astronomy & Astrophysics
View presentation slides (PDF)
Studying Quarks and Gluons on Blue Waters
Project PI: Paul Mackenzie, Fermilab
Presented by: Steven Gottlieb, Indiana University
Abstract: Quantum Chromodynamics is the theory of Nature's strong force. It is responsible for the binding of quarks into protons and neutrons that comprise the bulk of stable matter. The MILC Collaboration has been studying QCD on Blue Waters since the computer was first installed using the technique of lattice field theory invented in the mid-1980s by Nobel Laureate Kenneth Wilson. The great computational power of Blue Waters has enabled calculations of precision only dreamed of before its installation. On Blue Waters we are able to reduce the masses of the up and down quarks to the value they have in Nature. We can also use finer grids than previously feasible. With these improvements the reduction in systematic errors allows stringent tests of the Standard Model of Elementary Particle and Nuclear Physics. Some of these tests show tensions between theory and experiment. With somewhat higher precision in both we may find long sought evidence for new forces in Nature. This talk will focus on both the physics and the efforts to achieve high efficiency on Blue Waters.
Field of Science: Physics
Watch the presentation | View presentation slides (PDF)
Genotyping for Very Large Cohorts
Project PI: Liudmila Mainzer, University of Illinois at Urbana-Champaign
Presented by: Yan Asmann, Mayo Clinic
Abstract: Rescue the missing variants-lessons learned from large sequencing projects: identifying novel disease variants through next generation sequencing has been successful in the discoveries of new disease mechanisms as well as therapeutic strategies. The GATK best practices have since been established to provide general guidelines on core processing steps from raw reads to final variant call sets. However, with the sample size drastically increasing in today's sequencing projects, many default variant calling strategies and the choice of tools call for a closer examination. Using the whole exome sequencing data from Alzheimer's Disease Sequencing Project of more than 10,000 individuals, we identified a large number of good-quality variants missed by the current GATK recommendations by testing alternative tool and parameters choices. We plan to test a more comprehensive set of parameters for variant calling on Blue Waters.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Assembling a Map the Universe with Blue Waters: The Dark Energy Survey Production Pipelines
Project PI: Felipe Menanteau, University of Illinois at Urbana-Champaign
Abstract: The Dark Energy Survey (DES) is performing a 5,000 square-degree wide field survey in 5-band optical bands of the Southern sky and a 30 square-degree deep supernova survey with the aim to understanding the nature of Dark Energy and the accelerating Universe. DES uses the new 3 square-degree CCD camera (DECam), installed at the prime focus of on the Blanco 4-m to record the positions and shapes of 300 million galaxies up to redshift 1.4. During a normal night of observations, DES produces about 1 TB of raw data, including science and calibration images, which are transported automatically from Chile to NCSA in Urbana, Illinois to be archived and reduced. The DES data management system at NCSA is in charge of the processing, calibration and archiving of these data. We use Blue Waters to run our most intense I/O pipelines that process raw DECam survey images to into science-ready data products for analysis by the DES Collaboration and the public.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Transformative Petascale Particle-in-cell Simulations
Project PI: Warren Mori, University of California, Los Angeles
Presented by: Frank Tsung, University of California, Los Angeles
Abstract: The UCLA Simulation Group develops and uses a hierarchy of particle-in-cell software to study a variety of problems in high energy density plasma physics, high intensity laser and beam plasma interactions, plasma based acceleration, the nonlinear optics of plasmas, and inertial confinement plasmas, and astrophysics. This software runs efficiently on both single cores and on over a million cores. The computational resources at Blue Waters have been essential in allowing our group to perform large-scale simulations which have led to new discoveries and publications in high impact journals. In this talk, we will highlight our research results from the past year on plasma based acceleration and the nonlinear optics of plasmas of relevance to inertial confinement fusion. We will also mention software development efforts including enabling the software to take advantage of future many-core architectures. The software development is partially supported through the NSF funded Particle-in-Cell Kinetic Simulation Software Center (PICKSC). Much of the software is available through GitHub on the PICKSC website, http://picksc.idre.ucla.edu/
Field of Science: Physics
Watch the presentation | View presentation slides (PDF)
Path Integral Monte Carlo Simulations on the Blue Waters System
Project PI: Burkhard Militzer, University of California, Berkeley
Abstract: The properties of materials at extreme pressure and temperature conditions are important in astrophysics and fusion science. When models for Jupiter's interior are constructed to match gravity data from the NASA mission Juno, an accurate knowledge of the equation of state of hydrogen-helium mixtures is essential. Modern dynamic high pressure experiments typically probe megabar and gigabar pressures. In order to provide a comprehensive theoretical description of materials at such extreme conditions, we present results from path integral Monte Carlo simulations. We present equation of state results for first-row elements including carbon, CH plastic, oxygen, water, nitrogen, and neon that were derived with restricted path calculations that relied on free-particle nodes. We compute shock Hugoniot curves and compare with experimental results. We describe how bound states can be incorporated efficiently into the nodal structure (Phys. Rev. Lett. 115:176403, 2015), which enabled us to simulate heavier elements including sodium and silicon.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Seeing the Poles and Watching them Change; High Resolution Topography from Commercial Satellite Imagery, Petascale Computing and Open Source Software
Project PI: Paul Morin, University of Minnesota
Abstract: Surface topography is among the most fundamental data sets for geosciences, essential for disciplines ranging from glaciology to geodynamics. Two new projects are using sub-meter, commercial imagery licensed by the National Geospatial-Intelligence Agency and open source photogrammetry software to produce a time-tagged 2m posting elevation model of the Arctic and an 8m posting reference elevation model for the Antarctic. When complete, this publically available data will be at higher resolution than any elevation models that cover the entirety of the Western United States.
These two polar projects are made possible due to three equally important factors: 1) open-source photogrammetry software, 2) petascale computing, and 3) sub-meter imagery licensed to the United States Government. Our talk will detail the technical challenges of using automated photogrammetry software; the rapid workflow evolution to allow DEM production; the science results to date; how the data is being shared; and the derived products made by the wider world.
Field of Science: Earth Sciences
Watch the presentation | View presentation slides (PDF)
Enabling Discoveries at the Large Hadron Collider through Advanced Computation and Deep Learning
Project PI: Mark Neubauer, University of Illinois at Urbana-Champaign
Abstract: The goal of particle physics is to explain the Universe at its most fundamental level. The Large Hadron Collider (LHC) is the world's most powerful accelerator, designed to elucidate the fundamental buildings blocks of matter and their interactions by colliding protons at the highest energies. The University of Illinois is a key contributor to the ATLAS experiment, one of two general purpose LHC experiments. The complexity of the detector, software and LHC environment places enormous demands on the required computing resources. We make use of Blue Waters to enable discovery of new phenomena at the LHC by processing, simulating, and analyzing data produced by the ATLAS experiment and utilizing modern deep learning techniques to identify highly-boosted Higgs bosons—a signal of new physics. These activities make use of the unique capabilities of Blue Waters and advances in machine learning technologies to substantially improve the sensitivity of searches for new phenomena.
Field of Science: Physics
Watch the presentation | View presentation slides (PDF)
Predicting the Transient Signals from Galactic Centers: Circumbinary Disks and Tidal Disruptions around Black Holes
Project PI: Scott C. Noble, The University of Tulsa
Abstract: The ability to record the sky at high frame rate, like that of the Large Sky Synoptic Telescope currently under construction, has the potential to discover new transient phenomena and enhance our understanding the most powerful engines in the cosmos: supermassive black holes. We are performing the first general relativistic magnetohydrodynamics simulations of accreting supermassive binary black holes that incorporate mini-disks about each black hole, enabling us to provide a more realistic and steady-state model of these multi-messenger sources. We will present how magnetic stresses alter the level of mass transfer between the circumbinary disk and mini-disks, and between the two mini-disks themselves. These two aspects play a key role in understanding the luminosity, light curve and spectrum of these sources, and hence are critical to electromagnetic identification and search campaigns. We will also provide the latest results on our multi-patch simulations of tidal disruption events.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Cello and Enzo-P: Pursuing Petascale Astrophysics
Project PI: Michael L. Norman, University of California, San Diego
Presented by: James Bordner, University of California, San Diego
Abstract: Cello is a highly scalable adaptive mesh refinement (AMR) framework, and Enzo-P is an astrophysics and cosmology application built using Cello. Two fundamental advances of Enzo-P compared to its predecessor, Enzo, are its AMR design and parallelization: Cello implements fully-distributed array of octree AMR, and is parallelized using the Charm++ parallel programming system. To date, Enzo-P has demonstrated excellent parallel weak-scaling on Blue Waters, in both time and memory use, through 256,000 floating-point cores. More recent capabilities include support for particle methods, and a scalable and robust multigrid-based linear solver. We will describe Cello's distributed AMR data structure design, and Enzo-P's scalable linear solver algorithm and implementation. We will also present updated parallel scaling results, and discuss how Blue Waters has helped us in our pursuit of petascale astrophysics and cosmology.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Unlocking the Mysteries of the Most Violent Tornadoes
Project PI: Leigh Orf, University of Wisconsin - Madison
Abstract: Each year, supercell thunderstorms are responsible for significant loss of life and property across the United States. The vast majority of damage is caused by the least common, but most destructive, tornadoes spawned from the storms. We present simulation results run at extremely high resolution (up to 15 meter grid spacing) where long-lived tornadoes producing winds in excess of 300 mph (the strongest winds ever observed in the atmosphere) are found for long periods of time. In addition to recent simulation results, we will also discuss our adaptation of the lossy floating point ZFP compression algorithm as an HDF5 filter. ZFP is fast, highly flexible, and intuitive to use. We find compression ratios on the order of 100:1 to be useful for visualization and many forms of post-processing, allowing the capture of data at extremely high temporal resolution (up to every time step) for segments of the tornado's life cycle.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
Simulations of Milky Way-type Galaxies, their Environments, and Progenitors
Project PI: Brian O'Shea, Michigan State University
Presented by: John Wise, Georgia Institute of Technology
Abstract: An accurate numerical representation of galaxy formation involves various astrophysical processes being modelled on spatial and temporal scales ranging from the cosmological to the stellar. We therefore rely on multi-scale methods in which sub-resolution models emulate the formation of stars and their subsequent feedback into its surroundings. It is paramount to calibrate such models, given a set of solvers, to reproduce galaxies consistent with observations. We have investigated Milky Way-type galaxies and their environments with isolated models that allow for high resolution and model calibration and cosmological models that use these calibrated models and follow their complete assembly histories. The suite of simulations run on the Blue Waters are crucial in the interpretation of recent observations of galaxies throughout cosmic time, their environments and magnetic fields. I will discuss preliminary results from this suite, focusing on two connections: star formation, the circumgalactic medium, and the baryon cycle; and Milky Way-type galaxies, their satellite galaxies, and the first generations of galaxies.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Modeling Hyperenergetic and Superluminous Core-collapse Supernovae
Project PI: Christian Ott, California Institute of Technology
Presented by: Philipp Moesta, California Institute of Technology
Abstract: Hyperenergetic and superluminous core-collapse supernovae belong to the most extreme transient events in the universe and are a key factor in the supernova gamma-ray burst connection. I will discuss the unique challenges in both input physics and computational modeling that come with a problem involving all four fundamental forces and highlight recent breakthroughs overcoming these challenges in full 3D simulations performed on BlueWaters. I will pay particular attention to how these simulations can be used to reveal the engines driving the explosion and conclude by discussing what remains to be done in order to maximize what we can learn from current and future time-domain transient surveys.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Machine Learning Reveals Ligand-directed Conformational Change of μ Opioid Receptor
Project PI: Vijay Pande, Stanford University
Presented by: Amir Barati Farimani, Stanford University
Abstract: The μ Opioid Receptor (μOR) is a G-Protein Coupled Receptor (GPCR) that mediates pain and is a key target for clinically administered analgesics. The current generation of prescribed opiates—drugs that bind to μOR—engender dangerous side effects such as respiratory depression and addiction due to the ligand-induced off-target conformations of the receptor. To determine both the key conformations of μOR to atomic resolution as well as the transitions between them, long timescale molecular dynamics (MD) simulations were conducted using BlueWaters supercomputer. These simulations predict new and potentially druggable metastable states that have not been observed by crystallography. We used statistical algorithms (e.g., tICA and Transfer Entropy) to perform our analysis and discover key conformations from simulation, presenting a transferable and systematic analysis scheme. Our approach provides a complete, predictive model of the dynamics, structure of states, and structure-ligand relationships of μOR with broad applicability to GPCR biophysics and medicinal chemistry applications.
Field of Science: Biosciences
View presentation slides (PDF)
Modeling Heliospheric Phenomena with Multi-scale Fluid-kinetic Simulation Suite
Project PI: Nikolai Pogorelov, University of Alabama in Huntsville
Abstract: The key challenge of our project is to perform multi-scale simulations of the solar wind (SW) flow starting from the solar surface to the boundary of the heliosphere, and its interaction with the local interstellar medium (LISM), which makes them especially suitable for Blue Waters. Our simulations have both fundamental and practical applications: (1) SW-LISM interaction is a natural laboratory to investigate flows of partially ionized plasma; (2) description of the SW flow and magnetic field at Earth and remote planets makes it possible to predict and mitigate against hazardous conditions affecting humans and electronics in space. Our accomplishments include: (1) SW simulations and their comparison with data at Earth, Pluto, and Uranus; (2) development of a SW model based entirely on solar observations, and (3) analysis of the heliotail and explanation of observations from Voyagers and IBEX spacecraft. This also resulted in a substantial extension of the MS-FLUKSS capabilities.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Unified Modeling of Galaxy Populations in Clusters
Project PI: Thomas Quinn, University of Washington
Abstract: Modeling galaxies and the Intra-Cluster Medium (ICM) in galaxy clusters is a formidable challenge. The morphology of the galaxies, and the energy and metal content of the gas is ultimately controlled by star formation processes that happen on molecular cloud scales of less than a million solar masses. On the other, total cluster masses exceed 1015 solar masses; hence a dynamic range in mass of over a billion is necessary for consistently modeling galaxies within this context. Because groups and clusters of galaxies contain a significant fraction of the galaxies in the Universe, understanding the physical processes that occur in these environments is key to gaining insights into the evolution of baryons and galaxies across the age of the Universe. We are now in a position to tackle this challenge with better hydrodynamic algorithms, scalable codes, and the power of Blue Waters.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Effects of Forcing Scheme on the Flow and the Relative Motion of Inertial Particles in DNS of Isotropic Turbulence
Project PI: Sarma L. Rani, University of Alabama in Huntsville
Presented by: Vijay Krishna Rani, University of Alabama in Huntsville
Abstract: In direct numerical simulations (DNS) of homogeneous isotropic turbulence, statistical stationarity is achieved through a forcing scheme that adds energy to the large scales. The forcing schemes may be broadly classi?ed into deterministic and stochastic forcing schemes. In deterministic schemes, the forcing added is such that the turbulent kinetic energy dissipated during a time step is resupplied, whereas in stochastic schemes, the forcing is determined based on the evolution of Ornstein-Uhlenbeck processes. Both approaches add the forcing within a band of wavenumbers at the low-wavenumber end of the energy spectrum. The goal of the current study is to investigate the effects of the forcing schemes on the ?ow, as well as on the relative motion of inertial particles in DNS of isotropic turbulence. An important parameter in stochastic forcing is the characteristic time scale of the applied forcing, TF. In this study, DNS was performed using the deterministic forcing, as well as stochastic forcing for five values of TF = Teddy/4, Teddy/2, Teddy, 2Teddy, 4Teddy. Here Teddy is the eddy turnover time obtained from the DNS with deterministic forcing. Three Taylor micro-scale Reynolds numbers Reλ = 76, 131, and 210, and twelve particle Stokes numbers based on the Kolmogorov time-scale, Stη ranging from 0.05 to 40 were considered. Detailed analysis of the effects of forcing time scales on both fluid and particle statistics was undertaken.
Field of Science: Fluid Systems
Watch the presentation | View presentation slides (PDF)
Monte Carlo Radiation Transport in Core-collapse Supernovae
Project PI: Sherwood Richers, California Institute of Technology
Abstract: When massive stars exhaust their nuclear fuel, they die in a cataclysmic core-collapse supernova explosion. However, the details of how a collapsing star leads to explosion are yet poorly understood. High-performance computing is required to model the neutrino radiation hydrodynamics in the collapsing star. The treatment of neutrino radiation transport in particular is extremely computationally intensive, key to the explosion dynamics, and presently included only approximately. We present the results of Blue Waters simulations that have allowed us to verify the accuracy and convergence of a Monte Carlo and of a Discrete Ordinates method to solve the full steady-state Boltzmann radiation transport problem in multiple dimensions. We also present progress coupling Monte Carlo radiation transport to a more approximate dynamical two-moment transport method that will enable highly accurate multidimensional simulations of core-collapse supernovae.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Energy Dissipation in Astrophysical Plasma Turbulence
Project PI: Vadim Roytershteyn, Space Science Institute
Abstract: In astrophysical plasmas turbulence provides a mechanism of efficient energy and particle transport thus playing a key role in the dynamics of many objects. Due to availability of in situ measurements, solar wind represents the best accessible laboratory for studies of large-scale plasma turbulence. The goal of the Blue Waters project is to study kinetic process in solar wind-like plasmas, most notably the mechanisms leading to dissipation of the turbulent energy. The simulations are closely informed by and compared against relevant spacecraft observations. We review the progress made during the first year of the project, focusing specifically on two distinct types of problems: i) kinetic ion dynamics in the inertial range (i.e. at large scales) and ii) electron dynamics and dissipation at small scales in relatively cold tenuous plasmas (as encountered for example in the solar corona).
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Non-adiabatic Electron-Ion Dynamics in Ion-irradiated Carbon Nanomembranes
Project PI: Andre Schleife, University of Illinois at Urbana-Champaign
Presented by: Alina Kononov, University of Illinois at Urbana-Champaign
Abstract: Low-dimensional materials are known to possess a variety of unique properties, but their potential applications depend on the development of efficient high-precision imaging and processing methods. We inform techniques such as ion beam microscopy and nano-structuring by studying the electronic response of two-dimensional materials to irradiation with charged particles. Using a massively parallel implementation of time-dependent density functional theory (Qbox/Qb@ll), we harness the computational prowess of Blue Waters to investigate the dependence of charge transfer, energy transfer, and secondary electron emission on projectile velocity, trajectory, and initial charge when protons, alpha particle s, or highly charged xenon ions traverse a carbon nanomembrane. In this talk, we present preliminary results, describe how Blue Waters enables this important project, and share strategies for overcoming challenges encountered in computational studies of dynamic processes in two-dimensional systems.
Field of Science: Materials Science
Watch the presentation | View presentation slides (PDF)
Atomistic Determination the Conformational Switches in Membrane Transporter Proteins
Project PI: Diwakar Shukla, University of Illinois at Urbana-Champaign
Presented by: Balaji Selvam, University of Illinois at Urbana-Champaign
Abstract: Prokaryotes and Eukaryotes such as bacteria, plants, animals etc., uptake essential nutrients through transporters. Transporters are carrier proteins that transport the molecules in and out of the cell. These proteins undergo large conformational changes such as inward facing (open to the cell), occluded (closed at both ends) and outward facing state (close to the cell) to transport the molecules. Unlikely, it's not possible to obtain crystal structures of different states using experimental procedures as this class of proteins are highly flexible in nature. Using Blue Waters, we performed hundreds of microsecond-long simulations of membrane transporter proteins to predict the key conformational states to understand the substrate transport mechanism. We identified crucial residues that acts as a barrier for conformational switching of transporter form one state to other. For the first time, we characterize the complete cycle of transporter proteins using unbiased simulation using the Blue Waters supercomputer.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Developing a Hybrid Atomistic-Continuum Method for Simulating Large-scale Heterogeneous Biomolecular Systems
Project PI: Sean L Seyler, Arizona State University
Abstract: Biological macromolecules such as proteins, usually immersed in an ionic aqueous environment, are often highly heterogeneous systems whose dynamics can span femtosecond to millisecond timescales and beyond. Explicit solvent molecular dynamics (MD) is necessary to fully capture solute-solvent interactions, though water molecules contribute to most of the computational cost; sufficiently long simulations that can capture relatively slow processes, like large-scale conformational changes, are often infeasible. One approach is to hybridize MD with a comparatively frugal hydrodynamic (HD) model that replaces some (or all) of the solvent. We have been developing a hybrid method—with a view toward biomolecular simulation—that couples the LAMMPS MD engine to a discontinuous-Galerkin-based fluctuating HD solver. Several hydrodynamic test problems carried out on Blue Waters, used to assess the performance of the HD code, are presented. We also report on the current state of development of the prototype hybrid code and discuss remaining challenges to be addressed.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Full-scale Biophysical Modeling of Hippocampal Networks during Spatial Navigation
Project PI: Ivan Soltesz, Stanford University
Presented by: Ivan Raikov, Stanford University
Abstract: The hippocampus provides the basis for spatial navigation and episodic memory in the brain, remembering events experienced in the past and linking them with their spatio-temporal context. The hippocampal circuits that store and recall spatial information are comprised of diverse cell types, each exhibiting distinct dynamics and complex patterns of synaptic connectivity. We have developed full-scale detailed biophysical models of two regions in the rodent hippocampus, CA1 and dentate gyrus, and have analyzed the oscillation dynamics of those networks given either unstructured stochastic inputs or structured spatial input mimicking the patterns produced by entorhinal grid cells, an important class of cortical neuron that contributes to spatial representation. Perturbation experiments have revealed the roles of specific interneurons, as well as interneuron diversity itself as important factors in the generation of theta oscillations, which are associated with behavior and locomotion. These results reveal new insights into the spatiotemporal organization of the hippocampal circuits during theta oscillations.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Viral Infection Propagation through Air Travel
Project PI: Ashok Srinivasan, Florida State University
Abstract: Air travel has been identified as a leading factor in the spread of infections. In earlier work, we showed that changes to boarding and deplaning procedures could substantially reduce the number of contacts between passengers, with potential reduction in the risk of transmitting Ebola on flights. This was accomplished through a fine-scaled model that tracks the movement of passengers in airplanes. We recently linked this model with an infection transmission model to estimate the number of new infections. We then combined this with a phylogeographic model to estimate the spread of Ebola at a global scale. Our approach leads to a large computational cost, which is addressed through massive parallelization. We will discuss optimizations on Blue Waters to enable our simulation infrastructure to produce timely results in the face of threats to public health.
Field of Science: Computer Science
Watch the presentation | View presentation slides (PDF)
Analyzing Tropical Cyclone-Climate Interactions using the Community Earth System Model (CESM)
Project PI: Ryan Sriver, University of Illinois at Urbana-Champaign
Abstract: This project examines the relationship between tropical cyclones (TCs) and Earth's climate system using a high-resolution, state-of-the-art Earth system model (Community Earth System Model—CESM). We performed a suite of CESM simulations in which the high resolution atmosphere component (with 25 km horizontal resolution) is configured with three different levels of ocean coupling: prescribed sea surface temperature (SST), slab mixed layer ocean (no dynamics), and an ocean general circulation model with full dynamics and thermodynamics. We found that the inclusion of ocean coupling largely affects the simulated TC characteristics, including storm frequency, geographic distribution, maximum TC wind and storm intensification. Key differences in TC characteristics are attributed to the variations of the modeled large-scale circulations that arise from the combined effect of intrinsic model biases and air-sea interactions. These results provide new insights into the importance of coupled ocean-atmosphere interactions and feedbacks for understanding the connections between TCs and climate.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
The Impacts of Hydrometeor Centrifuging on Tornado Dynamics
Project PI: Ronald Stenz, University of North Dakota
Abstract: Current numerical simulations of tornadoes lack centrifuging of precipitation, producing an unrealistic maximum of precipitation in simulated tornado cores. This buildup of precipitation in the tornado core creates an unrealistic source of negative buoyancy in the tornado updraft, limiting the stretching of vertical vorticity. A centrifuging algorithm developed for the non-hydrostatic thunderstorm simulation model, CM1, and trajectory analysis will be used to determine the importance of the inclusion of centrifuging on tornado vorticity budgets. This study will improve the realism of tornado simulations and provide further insights into the dynamical processes occurring within tornadoes. For the first time, impacts that centrifuging of precipitation has on the vorticity budget for numerically simulated, supercell-spawned tornadoes will be quantified.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
Towards Deep Mechanistic Neural Networks for Multi-omics Modeling
Project PI: Illias Tagkopolous, University of California at Davis
Presented by: Ameen Eetemadi, University of California at Davis
Abstract: The plethora of omics data available provide a unique opportunity and challenge. The opportunity arises due to the fact that concomitantly harnessing the information of genome-scale measurements from multiple layers can provide more accurate and insightful predictions of cellular behavior and response than just one layer alone. The challenge lies on how to integrate these layers in a cohesive compendium and how to create predictive models that can train on it and capture its information. In this talk, we will present one such effort to build a normalized omics compendium and multi-omics model for Escherichia coli. We then discuss Transcription Regulation Neural Network (TRNN), a novel neural network architecture inspired by molecular thermodynamics of gene transcription. TRNN is accompanied with a novel back-propagation based training algorithm, all tailored for prediction of steady state mRNA concentration levels under genetic perturbation. The HPC parallel computing platform enabled thorough evaluation of competing neural network architectures on a cohort of data gathered from in-vivo experiments and in silico simulations.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Molecular Modeling and Simulation of Chemosensory Arrays on Blue Waters
Project PI: Emad Tajkhorshid, University of Illinois at Urbana-Champaign
Presented by: Keith Cassidy, University of Illinois at Urbana-Champaign
Abstract: The ability of an organism to sense, interpret, and respond to environmental signals is central to its survival. Chemotaxis is a ubiquitous signaling system by which biological cells translate environmental chemical information into a motile response. Bacteria, in particular, have evolved sophisticated protein networks that survey chemicals in the environment and position cells optimally within their habitat. This feat relies especially on the formation of large sensory complexes tens of thousands of proteins in size, known as chemosensory arrays, which mediate the transduction and regulation of signals that ultimately control cellular motility. Here, we present the first all-atom structure of the intact transmembrane chemosensory array of Escherichia coli, based primarily on crystallographic and electron microscopy data. Large-scale molecular dynamics simulations on Blue Waters are being used to investigate the dynamical properties of the array and its individual components to provide insight into their amazing information processing and control capabilities.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
GPU-based Simulations of Tilted Black Hole Accretion and Jets
Project PI: Alexander (Sasha) Tchekhovskoy, University of California, Berkeley
Abstract: We present a new massively parallel GPU-based general relativistic magnetohydrodynamic (GRMHD) code H-AMR. NVIDIA K20x and P100 GPUs speed up the code by factors of 15 and 60, respectively, relative to the CPU of a Blue Waters XK7 node. Blue Waters enabled us to develop advanced code features, adaptive mesh refinement and local adaptive time-stepping, that give an additional order of magnitude speed-up relative to conventional GRMHD codes and enable attacks on the most difficult problems. In tidal disruption events, the black hole spin axis is generally misaligned with the angular momentum axis of the disrupted star and the resulting accretion disk. Enabled by Blue Waters, our long-duration global simulations of black hole accretion demonstrate for the first time that (i) tilted disks produce jets, which (ii) precess together with the disk. These highest resolution simulations to date show the importance of numerically resolving the complex disk-jet interaction.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Glassy Dynamics and Identity Crises in Hard Particle Systems
Project PI: Erin Teich, University of Michigan
Abstract: The physics of glassy behavior is relevant in systems ranging from superconductors to sand. Despite this ubiquity, the thermodynamics undergirding glass formation remain frustratingly murky, due in large part to the significant slowing down of any system as it approaches the glass transition. This requires investigations of the glass transition to dauntingly resolve system dynamics on time scales that vary by orders of magnitude. We used Blue Waters to examine the role that entropy plays during vitrification by simulating and analyzing both the dynamics and the local structure of systems of faceted hard particles on these various time scales. We present our results, in which entropic forces between faceted hard particles emerge and induce the formation of a variety of incommensurate local motifs, causing the system to experience an identity crisis and to exhibit glassy behavior.
Field of Science: Physics
Watch the presentation | View presentation slides (PDF)
Fluid Flow and Stress Analysis of Steel Continuous Casting
Project PI: Brian Thomas, University of Illinois at Urbana-Champaign
Presented by: Seong-Mook Cho, University of Illinois at Urbana-Champaign
Abstract: Many defects in final steel products produced in continuous casting are related to turbulent multiphase flow, heat transfer, and stress in the mold. This project aims to develop accurate multiphysics models of the process, to simulate turbulent fluid flow, magnetohydrodynamics, particle transport, two-phase interfacial flow, heat transfer, solidification, and thermal-mechanical behavior. These computationally-intensive models are applied to gain better understanding and practical insights into defect formation, and to improve this important manufacturing process. For high resolution simulations with so many process phenomena, a large number of cells is required with extremely small time steps and long times, which is enabled by Blue Waters, with speedup of more than 3,400 times for some cases. Recently, effects of caster dimensions and process conditions including nozzle port angle, nozzle depth, mold aspect ratio, electromagnetic system on the transient flow in the mold, have been investigated using both commercial FLUENT and in-house CUFLOW codes.
Field of Science: Engineering
Watch the presentation | View presentation slides (PDF)
Advanced Space Weather Modeling: Year 1
Project PI: Gabor Toth, University of Michigan
Abstract: Our project addresses two major challenges in space weather modeling: how the buildup of magnetic energy results in solar eruptions and how magnetic reconnection results in geomagnetic storms. Both of these problems are crucial in space weather and they are both extremely demanding due to large separation of spatial and temporal scales. We will show how a combination of physical insight, numerical algorithms and high performance computing can be used to successfully tackle these difficult problems. We are now running the largest flux emergence simulation on Blue Waters with unprecedented spatial extent and grid resolution. The simulation shows the formation of a full-scale active region with highly energized magnetic fields that closely matches observations. We have also performed the first 1-hour-long 3D magnetosphere simulation with a fully kinetic treatment of the day-side reconnection. The simulation produces flux transfer events as well as reconnection signatures observed by the MMS spacecraft.
Field of Science: Astronomy & Astrophysics
Watch the presentation | View presentation slides (PDF)
Hurricane-spawned Tornadoes under Anthropogenic Climate Change
Project PI: Robert J. Trapp, University of Illinois at Urbana-Champaign
Presented by: Dereka Carroll-Smith, University of Illinois at Urbana-Champaign
Abstract: The purpose of this study is to investigate the impacts of anthropogenic climate change on the frequency of tornadoes spawned by tropical cyclones making landfall in the U.S. Atlantic Basin. Hurricane Ivan (2004), a prolific tropical cyclone tornado (TCT)-producing storm, is the particular case under consideration. Using the Weather Research and Forecasting model, Hurricane Ivan is simulated under its current-climate forcings. Such a control simulation is compared to simulations conducted using a "pseudo-global" warming (PGW) approach, which allows for an assessment of both near-future and long-term impacts of changes in greenhouse gas concentrations and associated radiative forcing. The PGW simulations involve future climate conditions over the mid (2040-2050) and late (2080-2090) century time periods. Changes in tropical cyclone intensity and TCT production for the PGW-perturbed Ivan are documented and analyzed. Ultimately, these simulations will be paired with demographic and socioeconomic data to assess the resultant disaster risk under future climates.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
Impacts of Orientation, Shape, and Size of Small Atmospheric Ice Crystals on Calculated Scattering Properties: Implications for In Situ Aircraft Measurements, Satellite Retrievals, and Numerical Models
Project PI: Junshik Um, University of Illinois at Urbana-Champaign
Abstract: The single-scattering properties of ice crystals smaller than 50 micrometers are calculated at a wavelength of 0.55 micrometer using a numerically exact method (discrete dipole approximation), an approximate method (geometric optics method), and Mie theory. For these calculations, hexagonal column, spheroid, and sphere are used to represent the shapes of the small ice crystals. Further, because morphological feature of non-spherical ice crystal is closely related to the single-scattering properties, alternate aspect ratios are considered for the hexagonal column and spheroid shapes in the scattering calculations. The results show the impacts of orientation, shape, and aspect ratio of ice crystals on the directional intensity and polarization of scattered light, which are subsequently used to solve an inverse problem. Based on these calculations, potential errors in the current forward scattering probes caused by the uncertainties in the shape of small ice crystals are quantified, including uncertainties in the total ice crystal concentration.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
A Different Type of "Computer Virus"
Project PI: Gregory Voth, University of Chicago
Presented by: John Grime, University of Chicago
Abstract: Coarse-grained (CG) molecular models provide computational efficiency by removing specific details while retaining the important molecular behaviors. CG models allow computer simulations to access much longer time- and length-scales than are feasible in e.g. simulations with atomic resolution. The use of CG models is therefore appealing for the study of biological phenomena that require large numbers of interacting molecules, and the results of these simulations can help to elucidate aspects of biology which are difficult (or impossible) to study at a molecular level with conventional experiments.
This presentation will describe a selection of CG simulations performed on Blue Waters by members of the Voth research group (Dept. of Chemistry, The University of Chicago). These simulations, performed in collaboration with experimental scientists from around the globe, focus on two biomedically relevant viral systems: human immunodeficiency virus type 1 (HIV-1), and influenza.
Field of Science: Biosciences
Watch the presentation | View presentation slides (PDF)
Quantum Many-body Explorations of Materials from First Principles
Project PI: Lucas K. Wagner, University of Illinois at Urbana-Champaign
Abstract: We will discuss our work on directly simulating many electrons in materials to determine their properties. These simulations use Monte Carlo to compute the properties of many interacting particles with quantum mechanical behavior and have minimal approximations. We use Blue Waters and these QMC techniques to simulate electrons in materials using very accurate computer models. This advance allows us to obtain quantitatively accurate simulations knowing only the positions of the atomic nuclei. This quantitative accuracy is an important component of in silico design of the next generation of materials. We will use these simulations to explore the process by which collections of electrons and nuclei result in the wide variation of materials properties seen in the world, often called emergence. By analyzing the detailed QMC calculations, we extract effective models. In addition to the physical insight from these models, they can then be used to study larger distance and longer time scales than possible from first principles
Field of Science: Materials Sciences
Watch the presentation | View presentation slides (PDF)
Accelerating and Scaling the PPM Multifluid Gas Dynamics Computation for Simulations of Hydrogen Ingestion Flashes in Evolved Stars
Project PI: Paul Woodward, University of Minnesota
Abstract: With support from the PAID program and visits to the University of Zurich and CSCS in Switzerland, we have been addressing two challenges: (1) finding the best way we can to structure our PPM code so that performance is very high on both the CPU and the GPU, and (2) finding some way to express that code so that it can be easily maintained for both target devices. We have been more successful with (1) than with (2). Our results from both efforts will be reported. Our overarching goal is to make our PPMstar code run as fast as possible on as large a subset of Blue Waters as possible. This has involved us in 2 additional challenges: (a) making the code run accurately in 32-bit precision, and (b) enabling a 3-level AMR implementation to scale to 20,000 nodes or more while maintaining high efficiency. We have solved challenge (a) and are confident that we can solve (b). These modifications to our PPMstar code will be described in the context of our simulations of semi-explosive fuel ingestion flashes in the interiors of evolved stars.
Field of Science: Astronomy & Astrophysics
Particulate Matter Prediction and Source Attribution for U.S. Air Quality Management in a Changing World
Project PI: Donald J. Wuebbles, University of Illinois at Urbana-Champaign
Abstract: This project aims at enhanced understanding of how global changes in emissions of important gases and particles, and corresponding changes in climate, could affect air quality in the United States. Currently, a state-of-the-art climate-chemistry model (the CAM5-chem components of NCAR's Community Earth System Model) is being used on Blue Waters for extended multi-year runs for extensive comparisons with observations. These benchmark multi-year runs will then be coupled with a much higher-resolution regional climate air-quality model over North America to determine individual and combined impacts of global climate and emission changes on U.S. air quality. Uncertainty evaluations will be done under multiple emissions and climate change scenarios. We will also examine the effects from using two different sets of reanalysis meteorological data on resulting levels of pollutants. Comparisons with observations will also prepare us for analyzing new halocarbons being proposed by industry for a variety of societal applications.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
High Resolution Earth System Modeling using Blue Waters Capabilities
Project PI: Don Wuebbles/Robert Rauber, University of Illinois at Urbana-Champaign
Presented by: Susan Bates, NCAR
Abstract: This project uses the Community Earth System Model (CESM) and the Weather and Regional Forecasting (WRF) model to advance the study of climate change and its potential impacts on our planet. This presentation will update the Blue Waters community on the activities during 2016 and lay the path forward for 2017 and beyond. Using CESM at the highest model resolution feasible for long, climate simulations as well as pushing model resolution to use the highest resolution currently available with the CESM ocean component, and using WRF at an even higher resolution (12km), our group has generated an extensive set of high-resolution simulations that can be assessed for present day and future climate change in terms of climate extremes. These include: tropical cyclones, mid-latitude storms and storm tracks, heat extremes, flash floods, and potential ocean influences on these processes. Without resources such as Blue Waters, climate simulations at these high resolutions would not be possible.
Field of Science: Atmospheric & Climate Science
Watch the presentation | View presentation slides (PDF)
Extreme Events, Resolution Effects and a New Parallel Algorithm for Turbulent Mixing on Blue Waters
Project PI: P.K. Yeung, Georgia Institute of Technology
Presented by: P.K. Yeung and Matthew Clay, Georgia Institute of Technology
Abstract: Substantial advances have been made in science and computing for fluid turbulence using Blue Waters. Results from simulations with up to 4 trillion grid points show clear differences between the energy dissipation rate locally averaged in three versus one dimension (the latter measurement being more common in past literature). Resolution effects on extreme events are examined critically using a multi-resolution approach designed to separate resolution errors from limitations of finite statistical sampling. We have also developed a new algorithm for turbulent mixing at low diffusivity, where fluctuations in the scalar field arise at smaller scales than in the velocity field. We use a dual communicator approach in which disjoint groups of MPI processes compute the velocity and scalar fields with different grid resolutions and numerical schemes. A hybrid MPI-OpenMP approach with dedicated communication threads is found to scale well up to 524,288 cores.
Field of Science: Fluid Systems
