THE ULTIMATE GUIDE FOR DATA ENGINEERS USING AI (Ep. 1)

"The Ultimate Guide for Data Engineers Using AI" EP1 — Data Ingestion I built a production-grade data ingestion platform from scratch — using nothing but Claude Code and VS Code. No boilerplate, no Stack Overflow. Just prompts, reviews, and a real-world pipeline. In this episode: Airflow DAGs pulling from Stripe, HubSpot, and Mixpanel, landing Parquet files in S3, partitioned by date. Every DAG is incremental and idempotent. Runs in Docker locally, deploys to EC2. What we build: — Apache Airflow 2.9 on Docker — 3 DAGs (Stripe charges, HubSpot contacts + deals, Mixpanel events) — S3 with Hive-style date partitioning — Parquet + Snappy compression — Incremental state management (no duplicate data) — Unit tests with moto + responses mocks — EC2 deploy with Nginx + Let's Encrypt Timestamps 0:00 — Series intro & what we're building across 3 episodes 1:00 — Episode intro: the ingestion layer 1:45 — Prerequisites (API keys, AWS, Docker, VS Code) 2:30 — Architecture walkthrough 3:30 — Claude Code: project scaffold prompt 6:00 — Results: Airflow UI, S3, Parquet check, tests 7:30 — What to verify before you call it done 8:30 — Outro + what's coming in EP. 2 Links 🤖 Replication prompt/skills.md → https://www.askcostafordata.com/episo... (click View resources ↓) 📦 EP2 — Data Warehouse & Transformation → [EP2 link] Prerequisites — Stripe, HubSpot, Mixpanel API keys (or use the dummy data mode in the prompt) — AWS account with an S3 bucket — Docker Desktop — VS Code + Claude Code extension