Low-Latency Strix Halo Cluster with RDMA (RoCE/Intel E810) and vLLM, Framework Desktop Boards

In this video, I move beyond Ethernet and llama.cpp and show a 2-node Strix Halo cluster using RDMA and vLLM tensor parallelism. The setup uses two Framework Desktop motherboards with 128 GB of unified memory each, connected directly via Intel E810 cards configured for RoCE. I cover the hardware details that matter for this build: direct-attached RDMA, custom cooling for the E810, and why running a x16 NIC in a PCIe x4 slot isn’t a real problem for inference. I also compare RDMA latency to standard Ethernet and explain why low latency is the key enabler for tensor parallelism here. On the software side, I walk through vLLM with Ray and the main blocker I hit along the way: missing RCCL support for gfx1151 in upstream ROCm. I explain what broke, how I patched RCCL to make multi-node tensor parallelism work on Strix Halo, and how to reproduce the setup using my toolboxes. Timestamps 00:00 – Introduction 01:14 – The Hardware 02:09 – RDMA / RoCE Network Card 03:28 – Custom Cooling for Intel E810 06:22 – PCIe Lane Caveat (x16 to x4) 08:32 – ROCm / RCCL gfx1151 Support 10:31 – Configuration Tutorial 13:06 – Benchmarks 14:03 – Conclusion Links & Resources Strix Halo Toolboxes & Guides: https://strix-halo-toolboxes vLLM Strix Halo Toolboxes (patched RCCL): https://github.com/kyuz0/amd-strix-ha... vLLM Benchmarks: https://kyuz0.github.io/amd-strix-hal... vLLM Tensor Parallelism: https://developers.redhat.com/article...