Advancing Spark - Understanding Low Shuffle Merge

Back in Databricks Runtime 9.0 we saw the introduction of a preview "Low Shuffle Merge" feature, but it seemed to go fairly unnoticed. In DBR 10.4, it's now enabled by default and a fully GA part of the platform... but what does it actually do? In this video, Simon walks through the theory of low shuffle merge, and what you should expect to see happening to both your runtime executions, but also the data layout before and after the change. Make no mistake, it's a real speed boost to many common patterns, so use it if you can! For more info on Low Shuffle Merge, see the docs over at: https://docs.microsoft.com/en-us/azur... And as always, get in touch with Advancing Analytics if you need help on your Lakehouse journey