Design a web crawler
Design a crawler that discovers and downloads pages across the web for a search index.
SDE-1Beta
Clarify
Step 1 of 5Real rounds start with questions, not boxes. Write what you would ask the interviewer, then see what they would answer.
What the interviewer tells you
- Start from seed URLs, follow links, store page content
- Be polite: honour robots rules and per-site rate limits
- Avoid re-crawling the same page and avoid traps
- Refresh pages over time
Step 1 of 5