Manage Data Quality at Scale with Dataplex and Gemini

Interested in a solution to Manage Data Quality at Scale with Dataplex and Gemini? Data quality is non-negotiable, especially in the age of GenAI, AI Agents 🤖, and Function Calling. Traditional third-party observability tools often present challenges: increased risk ⚠️, lengthy procurement ⏳, and loss of control over your data 🔒. In my latest video, I dive into a powerful solution: combining Dataplex's Auto DQ with Gemini to manage data quality at scale. Here's what this video covers: 00:29 - Contents & Intro 01:48 - Why Data Quality Is Even More Important Today 03:12 - Problem with Existing Solutions 06:12 - What is Dataplex Auto DQ and Why Use It ✨ 06:12 - The 7 Data Quality Dimensions ✨ 07:12 - Define Data Quality Rules 09:34 - A Powerful Solution Using Dataplex & Gemini ✨ 09:34 - High Level Design ✨ 13:00 - Low Level Design (i.e. Rules naming convention + Terraform module) 15:04 - Code WalkThrough & Live Demo ✨ 15:04 - Code Walkthrough ✨ 22:56 - Rule Specs Documentation (important & hard to find) ✨ 24:09 - Live Demo, DQ Features ✨ 34:40 - Reverse Engineer YAML 38:49 - Outstanding Issues ❗ 42:17 - Summary & Future Thoughts Watch the full video to learn how to build a robust data quality framework and ensure your data foundation is rock-solid! Code Repo: https://github.com/richardhe-fundamen... Slides: https://drive.google.com/file/d/1heEa... NEW: See the new video on how to build a powerful data quality agent to get to these insights in seconds!    • A Powerful Data Quality Agent for Dataplex