DataFusion Community Showcase Vol. 2: DataFusion Comet & DataFusion Ballista

Recorded live on July 23rd, 2026, this is second DataFusion Community Showcase highlighting two DataFusion projects: ​1) Jordan Epstein, "Accelerating Apache Iceberg Writes and Table Maintenance with Native Code" ​Iceberg writes and table compactions are a compute heavy workload that often goes overlooked. This talk explains the benefit of using DataFusion Comet to get the procedural logic for table compactions that lives in Spark plus the performance benefit of Rust and DataFusion! ​2) Phillip LeBlanc, "​Building an HA, Distributed Query Engine on Apache DataFusion Ballista" Apache DataFusion provides a fast single-node query engine, but larger analytical and agentic workloads need execution to reliably scale beyond one process. At Spice, we built our distributed query layer on Apache DataFusion Ballista. ​This talk covers why we selected Ballista and the extensions required to integrate it into a production query runtime: object-store-based cluster state, mTLS-secured communication, executor discovery and capacity management, remote catalog and UDF synchronization, configurable shuffle storage and encoding, and the reliability problems exposed at production scale. Learn more about the DataFusion Community Showcase and sign-up to present here: https://github.com/apache/datafusion/...