[14일차] Spark + Airflow 실전 연동 | 대규모 ETL 파이프라인 구축하기

🏗️ Data Lakehouse Engineering 20-Day Intensive Course DAY 14 — Spark + Airflow Integration and Large-Scale ETL Pipelines In this video, we will learn how to build practical data pipelines by integrating Apache Spark and Apache Airflow. ✔ SparkSubmitOperator ✔ KubernetesPodOperator ✔ Data Quality Gate ✔ Slack Notification Integration ✔ Spark Job Orchestration ✔ Airflow-based ETL Automation ✔ Vault Integration and Security Configuration ✔ Large-Scale Data Processing Architecture Airflow manages workflows, while Spark performs actual data processing. In a practical data platform, these two technologies are combined to build scalable ETL pipelines. This lecture explains the operational-level structure, ranging from Spark Job execution to quality verification, failure notifications, and security configuration. 📌 Recommended for: Data Engineers Platform Engineers DevOps Engineers MLOps Engineers Spark Operators Airflow Operators 🔥 Key Content of This Video How to use SparkSubmitOperator Utilizing KubernetesPodOperator Designing ETL Pipelines Automating Data Quality Validation Slack-based Fault Notifications Airflow + Spark Integration 🧪 Hands-on Content Running Airflow → SparkSubmitOperator Building Bronze → Silver → Gold Pipelines Implementing Data Quality Gates Setting up Slack Fault Notifications Spark Job Monitoring Automating Iceberg Loading 🔐 Security Content Vault Agent Sidecar Integration Kubernetes ServiceAccount Authentication NetworkPolicy-based Communication Control Masking Sensitive Information Personal Information in Fault Notifications Protection #ApacheSpark #ApacheAirflow #Spark #Airflow #ETL #DataEngineering #Iceberg #Kubernetes #DataPipeline #DevOps #MLOps #DataEngineer #AIInfrastructure