Blue Waters allocations for the University of Illinois at Urbana-Champaign

 

Proposal Types

General  - 

Illinois faculty and staff may request time on Blue Waters for the 2021 calendar year. This will be the final call for Illinois Blue Waters proposals as this is the final planned year of operations for the system.

Proposals are due by November 2, 2020

(Conitgent on continued funding from the National Geospatial-Intelligence Agency)

 

Instructions to Submit General Proposals

Slides for system information:

Submit Proposals

EasyChair links for proposal submissions:

Please allow 6-8 weeks from the date of closing for award notiication of submitted proposals. 

Overview

Blue Waters is one of the world's most powerful computing systems. In past years that Blue Waters has been in operation, about 3 to 4 million node-hours were allocated to projects from the University of Illinois at Urbana-Champaign. In 2021 the total Illinois allocation will exceed 20 million node-hours. As each node has many powerful cores, this is significantly more computing power than most universities have available for their use, and this resource provides University faculty and staff with a unique opportunity to perform groundbreaking work in computational science. 

Uniqueness of Blue Waters

Blue Waters is a uniquely powerful resource for computational science. Blue Waters is a Cray system with a combination of XE6 nodes (two AMD Interlagos CPU modules—four processor chips) and XK7 nodes (one AMD Interlagos CPU module and one NVIDIA Kepler K20X GPU), connected with Cray's Gemini interconnect, a high performance network. A high-performance, high-capacity file system is also part of the system. A summary of the key capabilities is listed below:

  • Over 1.3 petaflops (1015 floating-point operations per second) sustained performance (13.34 petaflops peak performance)
  • 1.66 petabytes (PB) memory, 25PB usable disk
  • Over 5.6 petaflops peak GPU performance (typically ¼ petaflops sustained)
  • Over 1 terabytes/sec (1012 bytes/second) sustained disk bandwidth
  • Extreme scale (22,640 XE6 nodes with over 360,000 core modules or 720,000 integer cores)
  • Large GPU system (4,228 XK7 nodes)

Additional details of the system design can be found at: https://bluewaters.ncsa.illinois.edu/hardware-summary

The Blue Waters allocations are intended for problems that need the unique capabilities of Blue Waters at large scale, which need not be just floating-point speed; it could be total memory, data size or data bandwidth, or even computational scale. Project proposals are expected to demonstrate that no other resource would be suitable for a given problem, as Blue Waters is not merely a large source of compute cycles. For a partial list of alternative resources of compute cycles see Alternate Sources of Compute Cycles. 

Requests for Time on Blue Waters by Illinois Faculty and Staff

Illinois faculty and staff may request time on Blue Waters. This will be the final call for Illinois Blue Waters proposals as this is the final planned year of operations for the system. In this call there will be two types of allocations: general, intended for large-scale research projects, and education for qualifying educational events such as courses and workshops. Unfortunately, exploratory allocations, intended to prepare code for Blue Waters cannot be offered as Blue Waters opperations will be coming to a close.

Proposals for this final round of Illinois General allocations were due November 2, 2020.  Applications for educational allocations are accepted on an ongoing basis—for more details visit: https://bluewaters.ncsa.illinois.edu/education-allocations.

Eligibility

Faculty or staff for whom the University of Illinois at Urbana-Champaign is their home institution by primary appointment affiliation are eligible to submit an Illinois allocation proposal as Principal Investigator. This includes postdoctoral fellows or postdoctoral research associates. Registered graduate or undergraduate students are not eligible to apply as Principal Investigators due to administrative requirements regarding appointment status but are encouraged to apply if their faculty or staff advisor will agree to be Principal Investigator on the proposal.

Visiting faculty or external adjunct faculty for whom Illinois is not their primary home institution are eligible to apply as Principal Investigators if, for the period covered by the proposal request: i) Illinois will be their primary (majority) place of residence; and: ii) and they will hold appointments at Illinois during this period. All proposals can include co-PIs and collaborators from other institutions.

Access to the Blue Waters system will be governed by the Blue Waters Terms of Use: https://bluewaters.ncsa.illinois.edu/terms-of-use.

General Projects

A general project allocation is often used for significant research projects. A typical general proposal will be for 30,000 to 1 million node hours and is required to be 5-8 pages in length and will provide more details than an exploratory proposal. 

Apply for a general project

See an example of a successful proposal

Proposal Guidelines

A project proposal should contain the following:

  1. Name(s) of Illinois PI and Co-PIs and if applicable, name(s) of collaborators and their institutions
  2. Project Abstract
  3. Field of Science   
  4. Project Overview (General proposals only)
  5. Target Problem
  6. Description of Code(s)
  7. Experience, Readiness, Usage Plans and Funding Source(s)
  8. Past Usage and Results
  9. Resources Required
  10. Data Managment at Project End
  11. Requested Start Date and Duration
  12. References

Explanations and guidelines are described in detail below.

Principal Investigator

The Principal Investigator must be a University of Illinois at Urbana-Champaign faculty or staff member. Multiple co-PIs may be listed and they may be students, postdocs, or faculty/staff members from Illinois or other institutions. Young investigators are encouraged to apply. Only one Principal Investigator should be listed for each proposal.  The proposal should include the PI and co-PIs names, title, department, university, and contact information.

Project Abstract

A one-paragraph (about 150 words) project summary including why Blue Waters is necessary for this activity.  This summary may be shared with the community on the Blue Waters website.

Field of Science

Clearly indicate the project's field of science.  A list of possible categories can be found at: https://bluewaters.ncsa.illinois.edu/foslist

Project Overview (for General Proposals)

For general allocations, include a one-page overview of the project that describes the science/engineering problem to be solved and the computational approach, including challenges. Also describe the possible scientific impact to the specific field of science or to the greater scientific community. This overview may be shared with the community on the Blue Waters website.

Target Problem

A description of the specific research question(s) that the resources requested will be used to answer and the scientific and societal impact of the proposed work. Include an explanation of why a petascale resource of the leading-edge capability that Blue Waters represents is necessary to address this research.  

Description of Code(s)

Describe the structure of the application codes that you intend to use. These may either currently exist, might require enhancement, or they may be in development. Include descriptions of any novel computational or data driven approaches. Please include details about the algorithms involved and the approach that you intend to use to ensure that the code scales effectively on the Blue Waters architecture. Describe how your code(s) will use each of the major system elements: the memory hierarchy, the communications network, the computational elements, GPU nodes, and the I/O subsystem. Identify which system element(s) is/are likely to be the main bottlenecks and how the design of your application minimizes the impact of these bottlenecks. Describe how you intend to analyze the output resulting from your use of Blue Waters. IMPORTANT: Please describe any run-time libraries or special system software or program development environment features that you will require and the types of graphics support that you would find most useful.

Experience, Readiness, Usage Plans and Funding Source(s)

Briefly describe your experience with using other HPC systems. Include the current state of readiness of the application codes that you intend to use and your plans for developing these to the point where they are ready to run in production mode on the Blue Waters system. Evidence of suitability for running on Blue Waters may include data on the efficiency of the application and analysis and/or demonstration of scalability.  Projects requiring smaller computational scales, or are not yet ready for Blue Waters should consider applying for one of the XSEDE startup accounts or other resources listed in the Alternate Sources of Compute Cycles.

Provide an estimated Blue Waters usage schedule. The estimate should be per quarter and may be represented as a percent of the requested allocation (e.g. Q1: 10%, Q2: 20%, Q3: 50%, Q4: 20%). It would be best to plan to use most of the time by the end of Q3 and not plan to use time in the final quarter, Q4 since many allocations will be trying to complete thier usage.

The Blue Waters project is tasked with efficient utilization of the Blue Waters system and accordingly reserves the right to age out and scale back allocations if their utilization rate is very low. However, the project understands that research time availability varies across the academic calendar and will consult with the PI before taking such steps.

Please identify the source(s), amount(s) and duration of existing funding that is supporting the development of your petascale application. If not currently funded, please describe how you intend to support any required development work.

Past Usage and Results

If the PI or Co-Ps have received prior Illinois allocations on Blue Waters please describe your utilization of the prior award, published results, presentations given, education and public outreach, and any grants that were obtained as a result of the Blue Waters work. Please include the Allocation title and Account Code (i.e. ILL_bpxy.). As there is a premium on Blue Waters utilization, if the PI or Co-PIs have had prior allocations where less than 75% of the allocation was used in the original allocation period, please provide a detailed summary of why the allocation was under-utilized and why this allocation if award will be fully used.

Resources Required

Describe the Blue Waters resources required to complete research on the Target Problem. This description should include the number and type of system nodes needed for your runs, the anticipated actual memory usage, the expected numbers of each major class of arithmetic and logical operation, the expected numbers of local and remote memory accesses, the total number of node-hours required, the anticipated input and output requirements, the amount of data that you anticipate transferring to or from the Blue Waters enclave, the amount and type of storage required and any other system resource needs that you anticipate. Note that due to the application mix anticipated in the 90% of the system allocation preference will be given to Illinois allocations requesting XK (GPU) node resources. For assistance in computing node hours for Blue Waters see: https://bluewaters.ncsa.illinois.edu/node_core_comparison

For a description of the default storage quotas see: https://bluewaters.ncsa.illinois.edu/storage

Please note that the nearline subsystem will not be avaialble during the final year of operations and the details of the quotas and storage management will change. Those details are not final, but are expected to include default scratch quotas of 100TB without a regular file system purge. If your project will require storage limits that exceed the standard quotas, provide a justification in support of your request.

Data Management at Project End

The Blue Waters storage sub-systems will not be available after the Blue Waters project ends. As a result please provide a description here of your plans for migrating your data from the Blue Waters storage system before the end of your allocation.  Currently, there is no grace period for data movement from Blue Waters so the plan must be convincing that you will be able to transfer your data prior to system shutdown.

Requested Start Date and Duration

For general allocations, the default duration is 12 months. If you believe your project can be completed in less time, please indicate as such (e.g. 6 months). All projects should include the desired start date for the allocation. All projects will end on or before December 31, 2021* - the end of the Blue Waters operations period.

References

Include references related to the work at the end of the proposal as part of the main document.

Related Information

Review Criteria

Projects will be judged on their scientific and/or educational merit, their suitability for Blue Waters, and their demonstrated need for the unique capabilities of Blue Waters. If resources have previously been allocated on Blue Waters for the project the effective utilization of these resources will also be considered.

NCSA Staff Assistance

Blue Waters information is available to guide potential proposals. Much system and programming environment information is available at the Blue Waters website, training material from past workshops, as well as complete documentation about using Blue Waters is available at the Blue Waters website.  See Training: https://bluewaters.ncsa.illinois.edu/training or Documentation: https://bluewaters.ncsa.illinois.edu/documentation. Blue Waters support staff are available during the workday to provide limited assistance for proposal writers such as answering specific questions, discussing algorithms and method approaches, scoping problems, providing general performance metrics, etc. Contacting the Blue Waters Support by submitting a service request via email to help+bw@ncsa.illinois.edu or calling 217-244-6689 between 9 a.m. and 5 p.m. Monday through Friday.

Reporting Requirements

Reports are required for all awards under this program. The report should include accomplishments achieved with the research conducted on Blue Waters—including publications, grants submitted, grants obtained, and talks presented, as well as details of any issues or problems encountered with the use of Blue Waters. Instructions for the final report are available at: https://bluewaters.ncsa.illinois.edu/reports. PIs will be responsible for submitting a final report and a status update mid-allocation. Projects are encouraged to contact Blue Waters support about issues or problems as they occur rather than using these reports to bring issues to the attention of the Blue Waters project. If a PI has had a previous allocation, timely submission of allocation reports will be considered during the review process.

Acknowledging Blue Waters Support

The text found on Acknowledging Support from Blue Waters or something similar must be included in publications of all kinds that report on work performed using an allocation on Blue Waters made under this program. It would be greatly appreciated if you would report publications and presentations to the Blue Waters project as they happen.

Alternate Sources of Compute Cycles

There are a number of sources of compute cycles available to researchers. These include both national and campus resources, and should be considered for projects that do not require the full power of Blue Waters. A partial list follows:

* Blue Waters operations is anticipated to run to December 31, 2021 contingent on continued funding from the National Geospatial-Intelligence Agency

Questions? Contact the Blue Waters Project Office at help+bw@ncsa.illinois.edu.