BIG DATA V: Parallel Processing of LARGE Image Datasets - NEUBIAS Academy @Home

NEUBIAS Academy webinar - 9 February, 2021 keywords: Fiji, OpenMPI, HPC, IJ2 Ops commands, ImgLib2, N5, Spark, BDV, Paintera Presenters: Pavel Tomancak, MPI-CBG, Dresden (DE) Stephan Saalfeld, HHMI Janelia campus, Ashburn (VA, USA) Helpers/Moderators: Vladimír Ulman, Ostrava Technical University (CZ) John Bogovic, HHMI Janelia campus, Ashburn (VA, USA) Julien Colombelli, Marion Loveaux, Rocco D'Antuono. ============================================ Contents: PART 1. 30 min Processing large images on a cluster from within Fiji We have developed a brige between Fiji and OpenMPI to enable parallel processing of many, large and many large images on an HPC resource. We will cover three distinct topics: 1) We will show how to realize parallelization of a very specific task, namely multi-view light sheet data registration and fusion, on a HPC resource through a dedicated web interface. 2) We will explain how one can spread an arbitrary Macro-mediated processing task on many relatively small images onto an HPC resource using specialized ImageJ parallel macro commands and monitor the progress and results. 3) We will demonstrate how to deploy specialized (syntactically identical) ImageJ2 Ops commands to parallelise processing of a single large image split into smaller chunks on an HPC resource. Finally, we will show how one can combine 2) and 3) to process many large images by first grouping them with parallel macro and then processing on the groups in parallel using the ImageJ2 Ops. This allows, for example, processing of long-term time-lapse 3D image data. We will discuss the limitations of this approach. PART 2. 1h Large data processing and visualization with ImgLib2, N5, Spark, BigDataViewer, and Paintera (40 + 5 min) Learn to create lazy processing workflows with ImgLib2, using the N5 API for storing and loading large n-dimensional datasets, and how to parallelize such processing workflows on a multi-threaded computer and on compute clusters using Spark. You will see how to use BigDataViewer to visualize and test processing results, and we will prepare projects for manual annotation/ proofreading with Paintera.