Should Datacenters Converge and Supercomputers Converge?

Indranil Gupta, University of Illinois at Urbana-Champaign

Usage Details

Cloud computing software systems (e.g., Hadoop, key-value stores, stream processing system, etc.) are projected by Forrester to become a $240 billion market by the year 2020. These systems are today run atop datacenters. However, datacenters themselves are increasingly using supercomputer-like features including fast networks and shared memory. This proposal seeks to evaluate a variety of widely-used cloud computing systems in three areas: storage (Cassandra, Riak), computation (Hadoop, graph computation, Storm), networking (Software Defined Networks). Our main goal is to evaluate the performance of these the data-intensive cloud software systems atop a Blue Waters-like supercomputer, traditionally intended for compute-centric workloads. To do so, we will evaluate four ongoing research projects. The project has the potential to impact large swathes of the cloud computing providers, customers, and users. It will also lead to wisdom on how to build supercomputer architectures for today’s cloud applications. This project is complementary to two ongoing NSF projects, and one ongoing Air Force (AFOSR/AFRL) projects involving PI Gupta.