Scaling Pandas Using Dask: How to Avoid All My Mistakes | Krishan Bhasin | Dask Summit 2021

Dask is a Python package that provides advanced parallelism for analytics, enabling performance at scale for the tools you love. People think it’s magic - drop it in and it scales. This will mostly work, but it will not scale well! We would like to share what we’ve learned about using Dask to scale dataframe and computations, to avoid you making the same mistakes. This is a talk about scaling Pandas using Dask by Krishan Bhasin at Dask Summit 2021. What is the Dask Summit? The Dask Distributed Summit is where users, contributors, and newcomers can share experiences to learn from one another and grow together. The Dask Distributed Summit provides content, information, and learning opportunities for attendees of all levels of Dask familiarity and expertise. What is Dask? Dask is a free and open-source library for parallel computing in Python. Dask is a community project maintained by developers and organizations. Share your feedback with us on this scaling Pandas talk and let us know: Did you find this talk on scaling Pandas using Dask helpful? What is your experience with scaling Pandas? Learn more at summit.dask.org and dask.org KEY MOMENTS 00:00:00 Scaling Pandas Using Dask 00:00:16 About Krishan Bhasin 00:00:59 Why This Talk? 00:01:38 Overview of Session 00:01:59 Dask Recap 00:03:37 A Closer Look at Dask Dataframe 00:04:33 A Closer Look at Distributed Scheduler 00:05:41 Submitting Work to a Cluster 00:08:05 Dask Learnings Part 0 - Just Don't 00:09:16 Dask Learnings Part 1 - You Can't Improve What You Can't See 00:10:34 Use The Dashboard 00:15:01 Dask Learnings Part 2 - Understand and Leverage Dask's Principles 00:23:39 Dask Learnings Part 3 - If It's Broke or Missing, Fix It! 00:29:10 Q & A