How to Read Any API as a Spark DataFrame (DataSourceV2 Tutorial)

What if you could do spark.read.format("myapi").load() for any data source you want? That's exactly what DataSourceV2 lets you build. In this video, I build a custom PySpark connector from scratch that reads from an external API and walk through every part of the implementation