Why BFS is the default
BFS gives you the homepage, then every category page, then every listing page, before drilling into individual items. For a site with depth-3 content, BFS surfaces meaningful structure within the first few hundred requests. DFS might spend the first thousand requests inside one tag's pagination before touching another category. Most crawl objectives ("get a sample of every section") favor BFS.
When DFS makes sense
DFS wins when you have a specific deep target and broad coverage does not matter — scraping every product in a single category, or extracting every page in one documentation section. It also has a memory advantage on very wide sites: DFS keeps a stack proportional to depth (small); BFS keeps a queue proportional to fanout × depth (potentially huge).
The hybrid that wins in practice
The strategy most production crawlers actually use: BFS with bounded depth, then per-pattern DFS for targeted extraction. The first pass discovers the site's structure (BFS, depth 3). The second pass targets specific subtrees you identified (DFS, unbounded depth within scope). This gives you both broad situational awareness and the deep coverage your pipeline needs.
