Advanced Slurm

Recorded training delivered on 25 February 2025 from the eScience Institute classroom, University of Washington, Seattle, WA, USA. Presenter: Dr. Kristen Finch; HPC Staff Scientist; UW-IT Research Computing; Hyak Team Film and Editing: Sam Han; UW-IT Communications & Marketing Production Specialist Tutorial Materials: https://hyak.uw.edu/docs/hyak101/basi... This tutorial offers a worked example that utilizes a container, computes against publicly available data from a genetic study focused on black cottonwood or Populus trichocarpa [1,2], and uses Slurm to submit interactive, single batch, and array jobs with Slurm (i.e., submitting multiple jobs to be performed in parallel). For this tutorial, we will be using a container made based on a Neural Network called Locator [3]. Locator is a set of python tools [4] that build a neural network with TensorFlow to predict the location of organisms based on their genotype (DNA; or genetic background). The Locator container was built using a python container (version 3.8-slim-buster) with Docker [5] following the Locator installation instructions. The container is publicly available on Docker Hub [6]. Why use Job Scheduling (Slurm) in this case? The neural network is trained on genotypes from a set of organisms with known location (latitude and longitude). The network is then passed a random subset of genotypes from organisms of "unknown origin" as a blind test. The origin of the test set samples is then predicted (latitude and longitude), and model error is determined by calculating the distance as-the-crow-flies between the known and predicted locations. To gain an understanding of the distribution of prediction error, this training/validation and testing protocol may be repeated with additional random subsets of individual trees treated as "unknowns." This case is an example of an "embarrassingly parallel" workload where models for each of these random subsets can be trained and tested independently in parallel. This tutorial will demonstrate these benefits of Slurm and provide you with some template Slurm scripts that you can adapt for your purposes. Acknowledgements and Literature Cited: In this tutorial we use publicly available data and software. Our adaptation of Populus trichocarpa genotype data and locations are licensed under a CC0 1.0 Universal (CC0 1.0) Public Domain Dedication license. [1] Populus trichocarpa Paper Geraldes et al. 2013. https://doi.org/10.1111/1755-0998.12056 [2] Original genotyping results available on DRYAD Geraldes et al. 2013. https://doi.org/10.5061/dryad.1051d Locator Neural Network is a copyright 2019 of C. J. Battey and released under a Non-Profit Open Software License 3.0 (NPOSL-3.0). [3] Locator publication Battey et al. 2020. https://elifesciences.org/articles/54507 [4] Locator GitHub Repository C. J. Battey and University of Oregon. https://github.com/kr-colab/locator.git [5] Repository for the Dockerfile for Locator NN. https://github.com/finchnSNPs/Docker_... [6] DockerHub Repository for the Locator NN container. https://hub.docker.com/repository/doc...