Using Spark v3 to do bulk loading and migrations - Episode 44

Join Azure Cosmos DB product managers and partners for a weekly webcast on new features, tips, tricks, and more. In this episode, Sergiy Smyrnov joins Mark to talk about how to plan and do large migrations and data loads into Azure Cosmos DB Core (SQL) API using the new Spark 3.x connector. We’ll deep dive into lots of cool new features that make Spark great for moving huge amounts of data including showing off the new throughput control as well as Delete operations from Spark. 00:00:00 - Opening 00:02:02 - Meet Sergiy 00:04:41 - Agenda for this session 00:05:33 - Best practices for large dataset migration/bulk load into Cosmos DB 00:10:57 - Best practices example 00:17:18 - Optimize Write RUs by tuning/customizing Cosmos DB Index 00:18:37 - Spark Load into Cosmos DB container with default index strategy (26 attributes) 00:19:27 - Spark load into Cosmos DB container with an optimized index strategy 00:19:57 - Why Spark for Cosmos DB bulk loading? 00:25:21 - Migrating from Spark 2.4 to new Spark 3 Cosmos DB 00:29:02 - How bulk ingestion works 00:31:00 - Cosmos DB Spark 3 OLTP connector features 00:32:43 - Cosmos Spark 3.x new feature - Catalog API 00:37:58 - Cosmos Spark 3.x change - populate "id"column 00:39:55 - Cosmos Spark 3.x - retry policies and validation 00:42:07 - Cosmos Spark 3.x - JSON serialization settings 00:43:50 - Cosmos Spark 3.x - Throughput Control 00:52:22 - Cosmos Spark 3.x - Bulk Delete 01:07:19 - Closing Quickstart: Manage data with Azure Cosmos DB Spark 3 OLTP Connector for SQL API https://docs.microsoft.com/azure/cosm... Best practices for large data migration/load - https://aka.ms/cosmos-throughput-best... Main page for Cosmos Spark v3 connector - https://aka.ms/azure-cosmos-spark-3 Cosmos Spark best practices - https://aka.ms/CosmosDBSparkBestPract... Reference Notebook repos: https://aka.ms/azure-cosmos-spark-3-s... https://aka.ms/CosmosSparkSynapseNote... Playlist for all episodes of Azure Cosmos DB TV Live -    • Azure Cosmos DB TV   Try Azure Cosmos DB - https://developer.azurecosmosdb.com/ #azurecosmosdb #azure #nosql #cloud