CSC4700-Distributed Parallelism with HPX (1st Part)

This lecture introduces distributed parallelism using the HPX framework. Single program multiple data (SPMD) and Communicating Sequential Processes (CSP) models are discussed, then the focus shifts on practical implementation using HPX, including point-to-point and collective communication examples. The lecture also covers data partitioning, serialization, and the use of communicators within HPX. [00:00:00] - Introduction and Course Announcements [00:01:04] - Distributed GPU Programming Recap [00:01:43] - Implementing Distributed Parallelism [00:02:10] - Communicating Sequential Processes (CSP) Model Recap [00:03:14] - Four-Function HPX Code Examples [00:05:23] - Scalability: Weak Scaling and Data Distribution [00:08:03] - Partitioning Data Across Nodes [00:10:01] - Data and Code Distribution in Applications [00:11:39] - The SPMD Model: Code and Data [00:15:00] - Implementing SPMD: Integration Example [00:19:45] - Synchronization and Collective Operations [00:23:46] - Four-Step Pattern in Distributed Applications [00:25:46] - Applying Replicating Process Patterns [00:27:34] - Process Memory and Isolation Revisited [00:30:12] - Synchronization in Collective Operations [00:31:26] - Using CMake and Building HPX Programs [00:34:14] - Running Distributed Applications with HPX [00:35:20] - Verifying Correct Process Launch and IDs [00:36:37] - Communicators: What Do They Transmit? [00:39:24] - Serialization in HPX [00:43:04] - Creating Serialization for Custom Types [00:45:24] - Serialization Across Networks Recap [00:47:05] - Communicators: Source, Target, Peer-to-Peer, Channel-Based [00:48:50] - Details of Communicators Structure and Use [00:52:06] - Using Communicators in Practical Examples [00:53:22] - Channels: Abstraction for Peer-to-Peer Communication [00:55:45] - Synchronous and Asynchronous Send/Receive Functions [00:57:18] - Tags and Explicit Sender/Receiver in Communication [01:00:12] - Ensuring Data Type Matches on Both Ends [01:00:42] - Example: Ping-Pong Communication Pattern [01:05:26] - Importance of Per-Locality Code Paths [01:06:11] - Four Key HPX Functions for Communication [01:07:26] - Distributed Computation Process Summary [01:08:36] - Broadcast and Reduce Visualized and Explained [01:12:36] - Conclusions: DRP (Distributed Replicated Processes) Pattern [01:14:09] - Final Remarks and Wrap-Up