Low Level Benchmark Instructions and 
Result Tables

Introduction

Benchmarks play a critical role in the performance evaluation of lsystems.  The Low Level-2017 (LL-2017) benchmarks serve three purposes:

1. The benchmarks are carefully chosen to represent characteristics of the current Blue Waters and possible future workloads, which consists of solving complex scientific problems using diverse computational techniques at high degrees of parallelism.
2. The benchmarks give the opportunity to provide the concrete data associated with the performance and scalability of the systems.
3. The benchmarks can be used as part of the system acceptance and /or regression testing and as a measurement of performance throughout the operational lifetime of the system.

Observed benchmark performance should be obtained from a system under consideration or a system configured as closely as possible to the target system.  For systems targeted at supporting highly parallel computation, it is critical that the evaluators provide observed benchmark performance using the largest (most parallel) test inputs as well as other sizes.  The largest scale jobs in the benchmark suite should not be interpreted as the limit for the job concurrency for the target system. Performance projections are permissible if they are derived from a similar system that is considered an earlier generation and/or smaller system.  Projections should be rigorously derived, using best practices for application and system performance modeling and be thoroughly documented and easily understood.  In the tables below, the "Projected" column refers to the value projected for the full system target where direct measurements are not possible. 

Submission Guidelines

The benchmark results (or projections including original results) for the target system should be recorded in the tables provided at the end of this document.  Additionally, the evaluator should submit the completed tables, benchmark codes and output files and as well as documentation on any code optimizations or configuration changes.  The submitted source should be in a form that can be readily compiled on the target system.  Please do not include object and executable files, core dump files or large binary data files.  An audit trail should be supplied for any changes made to the benchmark codes.  The audit trail should be sufficient to demonstrate that the changes made conform to the spirit of the benchmark and do not violate the specific restrictions on the various benchmark codes. The compile and run logs should also be submitted.

If performance projections are used, this should be clearly indicated.  The output files on which the projections are based, and a description of the projection method should be included.  In addition, each system used for benchmark projections should be described in Table 2 below.  Each projection in benchmarks results tables should indicate on which system the benchmark was originally run.  Enter the corresponding letter of the "System" column of Table 2 into the "System" column of the benchmark result table. 

Run Rules

The run rules, which are included in the source distribution, supply specific requirements and instructions for compiling, executing, verifying numerical correctness and reporting results for each benchmark.  The benchmark performance should only be accepted from runs that exhibit correct execution.  Only software tools and libraries that will be included for general use in the target system as supported product offerings are permissible to build and execute the benchmarks.

Message passing programs should be built using an implementation that supports 64-bit virtual memory pointers and a thread-safe communication library that implements the MPI standard.  All tests are to be in 64 bit floating point mode unless otherwise allowed.

Benchmark Descriptions

Lower Level Tests

The Lower Level Tests, listed in Table 1, are simple focused tests that are easily compiled and executed.  The results allow a uniform comparison of features and provide an estimation of system balance.  Descriptions and requirements for each test are included in the source distribution.  The results for the target system should be recorded in Table 3 under the column "Target".  In the event that benchmark results are being projected, columns "Benchmarked" and "Target" should be filled out.

Modifications to the Lower Level 2017 Benchmark are only permissible to enable correct execution on the target platform.  No changes related to optimization are permissible except in the case of the NAS FT benchmark where the values for fftblock_default and fftblockpad_default may be changed to suit the target architecture.

Table 1.  Lower Level Tests

All tests are to be run in fully packed mode unless otherwise described below.  In architectures with multiple cores[1] per node, "fully packed" means that the number of instances, threads or MPI tasks per node should at least equal the total number of physical cores available on the node.

The NPB UPC FT Class D benchmark should execute with 256 UPC threads.  The evaluator may choose on how many physical nodes the code will run.

The PSNAP benchmark should execute on all available processors on the benchmark system.  The operating system used for the PSNAP run(s) should be configured as the system would be delivered for regular, production purposes.

Special rules regarding packing apply to the STREAM benchmarks.

Base Case

The base case limits the scope of optimization and the allowable concurrency to prescribed values.  Certain minimal exceptions are allowed for hardware multithreading and if there is insufficient memory per node to execute the application.  The base case also limits the parallel programming model to MPI only.  Each of these points is covered in more detail below.  In the Base Case for all Full Application runs, modifications are permissible only to enable porting and correct execution on the target platform.  No changes related to optimization are permissible.  Library routines may be used as long as they currently exist in an evaluator's supported set of general or scientific libraries, and should be in such a set when the system is delivered.  As well, they should not specialize or limit the applicability of the benchmark nor violate the measurement goals of the particular benchmark.  Source preprocessors, execution profile feedback optimizers, etc. are allowed as long as they are, or will be, available and supported as part of the compilation system for the full-scale systems.  Only publicly available and documented compiler switches should be used.  Compiler optimizations will be allowed only if they do not increase the runtime or artificially increase the delivered FLOP/s rate by performing non-useful work.

If a benchmark will not run on its target number of processors due to memory limitations, the evaluator may use the least number of additional processors necessary.  The evaluator should still solve the same global problem, using the same input files as for the target concurrency when the MPI concurrency is higher than the original target.  For codes where the number of processors to be used is included in the input files the input files may be modified accordingly if a larger number of processors than the target is required. Other than that, no changes to the input files are allowed.

For all Base Case runs the benchmarks should be executed in a fully-packed manner on the computational nodes. In architectures with multiple cores per node, the number of MPI tasks times the number of threads per MPI tasks should equal the total number of physical cores available on the node. 

It is permissible for applications to run with more than one MPI task per core if the target system has the hardware capability to run multiple tasks, and the capability can be activated with a simple environmental setting that would be available to Track-1 users.  To use hardware multithreading, the evaluator should first start with the LL-2017 target concurrency given in the tables and then expand MPI concurrency to occupy hardware threads.  For example, for 2-way hardware multithreading, the evaluator should first start with the target concurrency and then expand to 1024 in order to engage the 2-way hardware threading.  The increase in MPI concurrency should be the minimum needed to exploit the hardware multithreading features.

Optional Optimized Case

An optional optimized case has been added to allow the evaluator to highlight the features and benefits of the target system by submitting benchmarking results obtained through a variety of optimizations. The evaluator may choose to optimize the source code for data layout and alignment or to enable specific hardware or software features that may include (but are not limited to):

·      Using Hybrid OpenMP+MPI for concurrency;

·      Using vendor-specific hardware features to accelerate code;

·      Running the benchmarks at a higher or lower concurrency than the targets;

·      Running at the same concurrency as the targets but in an "unpacked" mode;

·      Any combination of the above.  

For example, if the scheduling unit is a node, all the cores in all the nodes assigned to the job should be counted as being used.  The evaluator should determine if the benchmark performance increases or decreases when running in an unpacked mode before submitting results.

Wholesale changes to the parallel algorithms are also permitted as long as the full capabilities of the code are maintained; the code can still pass validation tests; and the underlying purpose of the benchmark is not compromised.  As many changes to the code may be made as wanted so long as the following conditions are met:

·      All simulation parameters such as grid size, number of particles, etc., should not be changed. 

·      The optimized code execution should still result in correct numerical results.

·      Any code optimizations should be available to the general Track-1 user community, either through a system library or a well-documented explanation of code improvements.

·      Any library routines used should currently exist in an evaluator's supported set of general or scientific libraries, or should be in such a set when the system is delivered, and should not specialize or limit the applicability of the benchmark nor violate the measurement goals of the particular benchmark.

·      Source preprocessors, execution profile feedback optimizers, etc. are allowed as long as they are, or will be, available and supported as part of the compilation system for the full-scale systems. 

·      Only publicly available and documented compiler switches should be used.

·      Finally, the same code optimizations should be made for all runs of a benchmark.  For example, one set of code optimizations may not be made for the smaller concurrency while a different set of optimizations are made for the larger concurrency. 

Any specific code changes and the runtime configuration used should be clearly documented with a complete audit trail and all supporting documentation.

Result Tables

Table 2. System Description

Enter the system details in this table for each system used in benchmarking.  Use the System label to refer to system corresponding to each test in the following tables.

For each application run, enter the run time variation in the column marked COV.

Table 3. Lower Level Test Results