Makeflow and Work Queue: Scaling Up from Laptops to Supercomputers

Photograph of Nate Kremer-Herman

Presenter

Nate Kremer-Herman

Graduate Research Assistant
Cooperative Computing Lab
University of Notre Dame

Presenter Bio


Nate Kremer-Herman is a second year PhD student working in the Cooperative Computing Lab at the University of Notre Dame. Nate is interested in exploring storage constraints and scaling issues in parallel applications. He is currently in charge of outreach in his research group, and he helps introduce new users to the Makeflow workflow specification language and the Work Queue execution engine. These tools are designed to help scientists scale up their research, and making sure that experience is user-friendly and approachable is one of Nate's top priorities.

Abstract


This seminar presentation will introduce attendees to the workflow management system Makeflow along with an associated execution engine called Work Queue. These tools are designed to provide a generic, user-friendly, and scalable interface for workflow management. During this presentation, we will cover the steps of writing a makeflow, executing a makeflow locally, and running a makeflow via batch system using Work Queue. We will make use of a brief, hands-on tutorial. By the end of the seminar, an attendee will be able to get started writing their own makeflows and use Work Queue to effectively scale their research up to their campus- or national-scale infrastructure.


Session details

When: 10:00 CST, March 22, 2017

Length of Session: One hour

Target Audience: Research scientists, graduate students, upperclass undergraduate students of any domain.

Prerequisites: Basic Linux knowledge.


User Base: We currently have users in bioinformatics, high energy physics, computational chemistry. We are always looking to help users in other domains get started.


Software Availability: Our software is open source and available in its totality on GitHub.


Software Requirements: Linux or Mac OS X operating system.


Use cases: We present a generic platform, I will be presenting a fairly generic tutorial. When considering examples in the presentation portion of the seminar, I will cover applications in bioinformatics, high energy physics, and molecular dynamics.


Training Materials: Our most recent tutorial can be found here.

Webinar video


If you don't see the video, reload the page or click here.

 

Webinar slides: Download