Blue Waters Innovation and Exploration Allocations
Blue Waters is one of the world's most powerful computing systems. Each year that Blue Waters is in operation, up to 8 million node-hours could be allocated to projects for innovation and exploration of new uses for high performance computation and data analysis. As each node has many powerful cores, this is significantly more computing power than most systems available, and this resource provides a unique opportunity to perform groundbreaking work in computational science.
Uniqueness of Blue Waters
Blue Waters is a uniquely powerful resource for computational science. Blue Waters is a Cray system with a combination of XE6 nodes (two AMD Interlagos CPU modules – four processor chips) and XK7 nodes (one AMD Interlagos CPU module and one NVIDIA Kepler K20X GPU), connected with Cray's Gemini interconnect, a high performance network. A high-performance, high-capacity file system is also part of the system as well as large nearline storage system. A summary of the key capabilities is listed below:
- Over 1.3 PetaFLOPS (10^15 floating-point operations per second) sustained performance (13.34 PetaFLOPS peak performance)
- 1.66 PetaBytes(PB) memory, 25PB Disk, 300PB tape
- 4,228 XK7 GPU nodes with NVIDIA K20x GPUs
- 22,640 XE6 nodes with over 360,000 core modules of 720,000 integer cores
- Over 4 PetaFLOPS peak GPU performance (typically¼PetaFLOPS sustained)
- Over 1 TeraBytes/sec (10^12 bytes/second) sustained disk bandwidth
Additional details of the system design may be obtained from: https://bluewaters.ncsa.illinois.edu/hardware-summary
The Blue Waters allocations are intended for problems that need the unique capabilities of Blue Waters at large scale, which need not be just floating‐point speed; it could be total memory, data size or data bandwidth, or even computational scale. Project proposals are expected to demonstrate that no other resource would be suitable for a given problem, as Blue Waters is not merely a large source of compute cycles. For a partial list of alternative resources of compute cycles see Alternate Sources of Compute Cycles.
Requests for Time on Blue Waters
Requests for resources should meet the goals of the Blue Waters Allocation Policy, quoted below:
"Up to 5% of the available cycles, at the discretion of the PI, will be allocated outside of the national allocation process provided that they are allocated to further research or education in the national science and engineering community and to broaden participation in high performance computing. The Awardee is encouraged to use these discretionary cycles to foster discovery and innovation."
There are three types of allocations: MOU, intended to provide resources to support memorandums of understanding for collaborative efforts between NCSA/UIUC and other institutions, general, intended to demonstrate innovative uses for Blue Waters that do not fit other categories of allocations, and strategic, intended for large-scale research investigations that are in the best interest of the Blue Water project.
Innovation allocation requests may be considered at any time if there is a need for an expedited allocation to meet a deadline, such as for proposal submission; inquiries regarding these time-sensitive requests may be made directly to email@example.com.
Number of Projects
At any time there is expected to be no more than 8 MOU projects, 8 general projects and 2 strategic initiatives.
Only Principal Investigators who are faculty or senior staff at US research organizations are eligible to submit an Innovation allocation proposal. The proposals can, however, include co-PIs and collaborators from other institutions.
MOU projects support innovative collaborative activities between NCSA/BlueWaters and other research and development organizations within the context of formally defined memorandums of understanding/agreements. Examples include the Joint Laboratory for Extreme Scale Computing and the NCAR/NCSA MOU. Reciprocal allocations on the other organizations’ resources are expected for NCSA/UIUC researchers as appropriate and needed. MOU projects are renewable and are for periods of 6‐12 months for 20,000-50,000 node hours. Projects requiring smaller computational scales should consider applying for one of the XSEDE startup accounts or other resources. See Alternate Sources of Compute Cycles.
General project allocations are for significant research projects. A typical general proposal will be for 30,000 to 250,000 node hours and will be 3-5 pages in length and will provide sufficient detail to justify the merits of the project. General projects are non-renewable and are for periods of 3-6 months. Projects requiring more time are encouraged to apply for time through NSF's PRAC program or other allocation processes.
Strategic initiatives allocations are for significant innovation initiatives initiated by the Blue Waters Project or other related efforts. Examples include exploring Big Data applications using Blue Waters, Graph Analysis uses and other non-traditional activities that potentially take advantage of Blue Waters unique characteristics. Strategic initiatives may sub allocate time via a defined/approved review process that is part of the initiative. A typical strategic initiative requires 250,000 to 3 million node hours and lasts 12 to18 months. Interim reports of progress are required. Strategic initiatives will have evaluation criteria to assess progress and impact.
A project proposal should contain the following:
- Name(s) of PI and Co-PIs and if applicable, name(s) of collaborators and their institutions
- Project Abstract
- Field of Science
- Project Overview
- Project or Initiative Goals
- Description of Code(s)
- Readiness, Usage Plans and Funding Source(s)
- Resources Required
- Requested Start Date and Duration
Explanations and guidelines are described in detail below.
The Principal Investigator must be a faculty or staff member of a US institutuion engaged in research. Multiple Co-PIs may be listed and they may be students, postdocs, or faculty/staff members from the PI's home institution or from other institutions. Young investigators are encouraged to apply. Only one Principal Investigator should be listed for each proposal. The proposal should include the PI and Co-PIs name, title, department, university and contact information.
A one-paragraph (about 150 words) project summary including why Blue Waters is necessary for this activity. This summary may be shared with the community on the Blue Waters website.
Field of Science
Clearly indicate the project's field of science. A list of possible categories can be found at: https://bluewaters.ncsa.illinois.edu/foslist
Include a one-page overview of the project that describes the science/engineering problem to be solved and the computational approach, including challenges. Also describe the possible scientific impact to the specific field of science or to the greater scientific community. This overview will be shared with the community on the Blue Waters website.
A description of the specific research question(s) that the resources requested will be used to answer and the scientific and societal impact of the proposed work. Include an explanation of why a petascale resource of the leading-edge capability that Blue Waters represents is necessary to address this research.
Describe the structure of the application codes that you intend to use. These may either currently exist, might require enhancement, or they may be in development. Include descriptions of any novel computational or data driven approaches. Please include details about the algorithms involved and the approach that you intend to use to ensure that the code scales effectively on the Blue Waters architecture. Describe how your code(s) will use each of the major system elements: the memory hierarchy, the communications network, the computational elements, GPU nodes, and the I/O subsystem. Identify which system element(s) is/are likely to be the main bottlenecks and how the design of your application minimizes the impact of these bottlenecks. Describe how you intend to analyze the output resulting from your use of Blue Waters. IMPORTANT: Please describe any run-time libraries or special system software or program development environment features that you will require and the types of graphics support that you would find most useful.
Describe the current state of readiness of the application codes that you intend to use and your plans for developing these to the point where they are ready to run in production mode on the Blue Waters system. Evidence of suitability for running on Blue Waters may include data on the efficiency of the node code and analysis and/or demonstration of scalability.
Provide an estimated Blue Waters usage schedule. The estimate should be per quarter and may be represented as a percent of the requested allocation (e.g. Q1: 10%, Q2: 20%, Q3: 50%, Q4: 20%).
Please identify the source(s), amount(s) and duration of existing funding that is supporting the development of your petascale application. If not currently funded, please describe how you intend to support any required development work.
Describe the Blue Waters resources required to complete research on the Target Problem. This description should include the number of system nodes needed for your runs, the anticipated actual memory usage, the expected numbers of each major class of arithmetic and logical operation, the expected numbers of local and remote memory accesses, the total number of node-hours required, the anticipated input and output requirements, the amount of data that you anticipate transferring to or from the Blue Waters enclave, the amount and type of storage required and any other system resource needs that you anticipate. For assistance in computing node hours for Blue Waters see:https://bluewaters.ncsa.illinois.edu/node_core_comparison
For a description of the default storage quotas see: https://bluewaters.ncsa.illinois.edu/storage
If your project will require storage limits that exceed the standard quotas, provide a justification in support of your request.
Usage of allocation is expected to begin immediately upon award and complete within the allocated allocation period. Projects should time their request to align with their other resources so the usage of Blue Waters can begin upon allocation award.
Projects will be judged on their scientific and/or educational merit, their suitability for Blue Waters; the degree to which the proposed work furthers research or education in the national science and engineering community, broadens participation on high performance computing and/or fosters discovery and innovation; and the demonstrated need for the unique capabilities of Blue Waters. Reviews will be conducted by the Blue Waters Project office with possible assistance from outside reviewers.
NCSA Staff Assistance
Blue Waters information is available to guide potential proposals. Much system and programming environment information is available at the Blue Waters website, training material from past workshops, as well as complete documentation about using Blue Waters is available at the Blue Waters website. See Training: https://bluewaters.ncsa.illinois.edu/training or Documentation: https://bluewaters.ncsa.illinois.edu/documentation. Blue Waters support staff are available during the workday to provide limited assistance for proposal writers such as answering specific questions, discussing algorithms and method approaches, scoping problems, providing general performance metrics, etc. Contacting the Blue Waters Support by submitting a service request via email to firstname.lastname@example.org or calling 217- 244-6689 between 9 a.m. and 5 p.m. Monday through Friday.
Reports are required for all awards under this program. The report should include accomplishments achieved with the research conducted on Blue Waters — including publications, grants submitted, grants obtained, and talks presented, as well as details of any issues or problems encountered with the use of Blue Waters. Instructions for the final report are available at: https://bluewaters.ncsa.illinois.edu/reports. PIs will be responsible for submitting a final report and a status update mid-allocation. Projects are encouraged to contact Blue Waters support about issues or problems as they occur rather than using these reports to bring issues to the attention of the Blue Waters project.
The text found on Acknowledging Support from Blue Waters or something similar must be included in publications of all kinds that report on work performed using an allocation on Blue Waters made under this program.
There are a number of sources of compute cycles available to researchers. These include both national and campus resources, and should be considered for projects that do not require the full power of Blue Waters. A partial list follows:
- NSF provides access to a number of high-end systems through the Extreme Science and Engineering Discovery Environment (XSEDE) project, including both startup and large-scale allocations. For more information, see xsede.org
- DOE offers time on high‐end systems operated by the Office of Science through the INCITE program:http://www.doeleadershipcomputing.org/
- A resource that Illinois researchers and/or their units may buy into is the Campus Cluster, for details, see https://campuscluster.illinois.edu/
- NSF's PRAC program provides access to Blue Waters (NSF controls roughly 80% of the available time on Blue Waters). Information on this annual program is available at: http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503224