Why 503s happen
There are two distinct sources. The first is genuine origin trouble: the server is overloaded, restarting, deploying, or hitting a database timeout. These 503s are transient and clear on their own within seconds or minutes. The second — the one scrapers hit constantly — is bot detection. Cloudflare historically returned 503 with its "Checking your browser…" interstitial; modern Cloudflare uses 403 for explicit blocks and reserves 503 for capacity protection, but the legacy pattern still appears. Akamai, AWS WAF, and Imperva all return 503 in various managed-block configurations. The response body usually tells you which one you're dealing with — a vendor-branded page is bot detection; a plain "Service Unavailable" or your own server's error template is real overload.
How scrapers should respond to 503
Treat 503 as retryable, but never blindly. Honor `Retry-After` if present. Use exponential backoff with jitter — 2s, 4s, 8s, 16s — capping at a few minutes. If the body shows a bot-detection interstitial, retrying from the same identity won't help; rotate proxy and refresh fingerprint before the next attempt. If you see 503s in bursts followed by recovery, that's likely real origin overload and your retry strategy will eventually succeed. If 503s are constant from your scraper but not from a browser, you're being blocked, not throttled.
Avoiding 503s in the first place
Spread your request rate. Concurrency limits should be set per-domain, not globally — hitting one site with 50 parallel workers reliably triggers 503s even on robust origins. Reuse TCP/TLS connections (a single HTTP/2 connection can carry many requests; spinning up a new connection per request is both slower and more bot-shaped). Cache aggressively — if you've already scraped a page in the last 24 hours, don't refetch it. And monitor: a sudden spike in 503s from a previously stable target usually means the target rolled out new rate limits or a new bot-detection rule, and your strategy needs to catch up.