Как правильно настроить Triggers в Apache Airflow: TriggerDagRunOperator,logical_date, reset_dag_run

🔥 Do you want your DWH storefronts to launch strictly after ODS, without unnecessary restarts and chaos in DAG dependencies? In this video, we'll discuss how to properly configure Triggers in Apache Airflow: TriggerDagRunOperator, working with logical_date, reset_dag_run, and skip_when_already_exists, so that DAG chains are idempotent, predictable, and don't snowball. Links: Data Engineering Mentoring/Consulting — https://korsak0v.notion.site/Data-Eng... TG Channel — https://t.me/DataLikeQWERTY Instagram —   / i__korsakov   Habr — https://habr.com/ru/users/k0rsakov/pu... 🔍 What's in the video: 🧠 What are Triggers in Airflow and how are they different from Sensors in real DWH pipelines? 🧱 Basic pipeline: ods_dag_users_to_dwh_pg → dm_dag_without_sensors ⚙️ First step: DAG showcases without sensors and without triggers — what's it all about? Bad and where data is lost 🚀 Adding a trigger: ods_dag_users_to_dwh_pg_with_triggers and the trigger_dag_dm_dag_without_sensors task 📆 Working with dates: logical_date, execution_date, Date of start — how a trigger affects the DAGRun of a dependent DAG ♻️ Idempotency: why does dm_dag_without_sensors have schedule_interval=None and how to achieve "clean" restarts 🧨 reset_dag_run and skip_when_already_exists: • how to safely restart DAGRuns for past dates • how to avoid creating duplicates and not dazing your storefronts with restarts 📊 dag_run and xcom tables: • how triggers record launch information • how to use tables to understand that a DAG is already running Worked for the required date 🧬 Dashboard chains: • another_dm_dag_without_sensors and another_another_dm_dag_without_sensors • What does a DAG → DAG → DAG dependency look like via triggers • How wait_for_completion works and why it can freeze a scheduler slot 🕒 Poking without mode: Why do triggers with wait_for_completion always occupy a slot, unlike sensors in reschedule mode 💥 What will go wrong if: • Change the dag_id of a dependent DAG • Delete/rename a DAG but leave triggers in the "main" DAG • Trigger from a DAG that itself depends on the same main DAG (cyclic dependency) 🧵 Practical recommendations: When Triggers are OK, and when is it better to leave Sensors and leave the stable one alone DAG 🗂️ GitHub repository with code: https://github.com/k0rsakov/pet_proje... ✉️ Questions, training, and consultations – PM me: https://korsak0v.notion.site/Data-Eng... 💡 At the end of the video, I'll explain why I still favor the Sensors approach in production and why I use triggers sparingly: we'll cover typical anti-patterns, issues with multiple sources (multiple ODSs on a single storefront), and cases where restarting the "main" DAG suddenly triggers the entire DWH. Timestamps: 00:00 – Start 00:17 – Disclaimer before we begin 01:02 – Reviewing the project infrastructure 01:47 – Looking at our DWH 02:05 – The first DAG for populating the DWH 03:18 – Looking at the first DWH showcase 05:38 – Reviewing a DAG with a trigger 08:55 – Reviewing how a trigger works under the hood in Airflow 10:00 – How to properly work with schedule in Airflow 10:48 – Restarting a DAG with triggers and reviewing the idempotency setup 13:47 – Reviewing how execution_date works under the hood in Airflow 15:48 – Reviewing DAG dependencies with triggers 16:30 – Adding a new trigger to the main DAG 18:45 – New trigger and reviewing the wait_for_completion attribute 21:27 – Shift The name of the DAG we're triggering 24:20 – Recommendations #apacheairflow #airflow #triggers #triggerdagrunoperator #logicaldate #dataengineering #etl #elt #dwh #python #xcom #dagrun #dataengineer #tutorial