April 18, 2017
Two webinars have been added to the schedule:
- "High-performance workflows with Swift/T" to be held on Wednesday, May 3, 2017
- "HDF Release Cycle and New Features" on Wednesday, August 9, 2017
Swift/T is a workflow system that enables users to combine traditional workflow features like the execution of external programs with calls to in memory functions. Swift/T runs as an MPI program managed by a scalable load balancer, allowing it to handle trillions of tasks on large supercomputers. It offers a concise high-level language for describing data dependencies and iterations. It also allows workflows to call into embedded scripting interpreters, such as Python, R, and JVM scripting languages, allowing scripts and systems (e.g., Numpy) in these languages to be applied at large scale. In this webinar, participants will learn how to use Swift/T with Python functions on Blue Waters.
HDF5 Release Cycle and New Features webinar
The HDF Group will give an overview of the HDF5 development and release cycle in general and the new features in the current family of HDF5 releases. The HDF Group will describe a new storage layout called Virtual Dataset Layout, which allows one to access data stored in multiple HDF5 datasets across HDF5 files as a single (logical) HDF5 dataset. We will present a new way of reading data while it is being written to an HDF5 file (“Single Writer/ Multiple Reader” or SWMR feature). Both features were released in HDF5 version 1.10.0 in March 2016.
Applications’ memory footprint and efficient I/O is the focus of the imminent HDF5 1.10.1 release (April 2017). We will explain how one can reduce application memory usage by taking control of the HDF5 metadata cache (“evict on close” feature) and how to accelerate I/O for applications that use HDF5 files as restart files by invoking the “cache image” feature.
Small-sized and random I/O accesses cause poor performance on many HPC systems. HDF5 release 1.10.1 introduces a new file space management strategy (“paged aggregation” feature) and an additional caching layer (“page buffering” feature) to mitigate the problem. If configured, the HDF5 library aggregates small metadata and raw data allocations into constant-sized well-aligned pages, which are suitable for page caching. Each page in memory corresponds to a page allocated in the file. Access to the file system is then performed on a single page or multiple of pages if they are contiguous. This ensures that small-sized accesses to the file system are avoided.
We will also give an overview of new features beyond the HDF5 1.10.1 release, including the forthcoming support for parallel compression and other I/O optimizations, which will be included in future releases.
February 15, 2017
The Scientific Visualization in Houdini session has moved from May 3 to July 26, 2017.
Due to a scheduling conflict that has arisen for the presenters of the "Scientific Visualization in Houdini" session, the date of the webinar was changed from May 3 to July 26, 2017.