Design a Web Crawler — System Design Interview (Staff-Level)

Write a web crawler in five lines and you've built a denial-of-service attack. A real planet-scale crawler was never about downloading — it's about politeness, deduplication, and prioritization. Here's the full staff-level breakdown. In this system design deep-dive: • Why the naive crawl loop gets your IP banned • Capacity estimation done live (~400 pages/s, ~100TB, the seen-set that forces a Bloom filter) • The crawl pipeline + the DNS bottleneck nobody mentions + backpressure • The URL frontier (Mercator): front queues for priority, back queues + a timing table for per-domain politeness • Bloom filters for URL dedup (and the false-positive tradeoff), content canonicalization, SimHash near-dupes, crawler traps • A full trace-and-break walkthrough of one URL through every stage • Domain-hash sharding, adaptive freshness, JS rendering, and the AI-era robots.txt shift (GPTBot/ClaudeBot) Practice this exact interview — with an AI interviewer that pushes back — free at https://craqit.io ⏱ Chapters 0:00 Your crawler is a DoS attack 1:35 Requirements & the core tension 3:05 Capacity estimation (live math) 4:45 The crawl pipeline (DNS, stores, backpressure) 6:40 The URL frontier — front & back queues 8:15 Politeness: the timing table 10:05 Dedup: the Bloom filter 11:40 Content dedup & crawler traps 13:30 Trace it & break it 15:25 Scaling & freshness 17:20 The AI-era shift 18:55 Recap System Design Interview series — more in this playlist #systemdesign #systemdesigninterview #webcrawler #distributedsystems #softwareengineering #faang