Spark Monitoring: Basics
In this video I cover how to use the default UI for monitoring Apache Spark. I use Azure Databricks to demonstrate, but most of the methods are the same in any Spark environment. I focus most on how I use it in my day to day work as a Data Engineer. Topics included: Spark UI, query plan basic, Ganglia Metrics, and driver logs. Monitoring Streams with Custom Streaming Query Listener: • Data Collab Lab: Monitoring Spark Streamin... More from Dustin: Website: dustinvannoy.com Twitter: @dustinvannoy Github: https://github.com/datakickstart
▶︎
Spark Monitoring: Streaming
▶︎
What is Spark? (Visual Explanation)
▶︎
Apache Spark Was Hard Until I Learned These 30 Concepts!
▶︎
Spark + Parquet In Depth: Spark Summit East talk by: Emily Curtin and Robbie Strickland
▶︎
Ex-Google Recruiter Explains Why "Lying" Gets You Hired
▶︎
Apache Spark Architecture - EXPLAINED!
▶︎
How Instagram Scaled Postgres to 2 Billion Users
▶︎
Spark performance optimization Part1 | How to do performance optimization in spark
▶︎
Master Reading Spark DAGs
▶︎
A Deeper Understanding of Spark Internals - Aaron Davidson (Databricks)
▶︎
Spark Basics | Partitions
▶︎
From Query Plan to Performance: Supercharging your Apache Spark Queries using the Spark UI SQL Tab
▶︎
Optimizing Apache Spark SQL Joins: Spark Summit East talk by Vida Ha
▶︎
Monitoring Databricks with System Tables
▶︎
The AI Take Over Has Completely Backfired and I Can't Be Happier
▶︎
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets - Jules Damji
▶︎
Deep Dive: Apache Spark Memory Management
▶︎
Don't waste 2026 learning the wrong tech skills (Meta Engineer's Take)
▶︎