PAID - Best Practice

William Tang, Princeton University

Usage Details

Based on the significant experiences and lessons learned in developing the modern GTC-Princeton (GTC-P) code, we plan to provide information of interest and usefulness to the Blue Waters science applications community by accumulating, creating, and applying "best practices" for efforts needed to develop portable and efficient science codes across diverse architectures. The GTC-P code is a highly scalable particle-in-cell code used for studying micro-turbulence transport in tokamaks.  It is a representative, discovery-science-capable 3D particle-in-cell code that has been successfully ported and optimized on a wide range of multi-petaflops platforms worldwide at the full or near-to-full capability of leading HPC systems, including NSF's "Blue Waters" and "Stampede."  

Associated benefits for portability come from the fact that GTC-P is not critically dependent on any third-party libraries. Various strategies employed to optimize performance, maximize parallelism, and utilize accelerator technology include multiple levels of decomposition for increasing scalability, choice of data layout for maximizing data reuse, leveraging GPU and Xeon Phi accelerators, and using hybrid programming models (MPI+OpenMP+optionally CUDA).  In particular, we will exploit the trade-off of portability vs. speedup in making effective use of GPU between using CUDA and compiler directives such as OpenACC and OpenMP4.0.