What triggers a 429
Servers enforce rate limits to protect themselves from abuse and accidental denial-of-service. The trigger is usually one of: too many requests per second from a single IP, too many requests per minute or hour against a specific endpoint, or an exhausted per-account quota on an authenticated API. The threshold varies wildly — public APIs might allow 60 requests per minute, while protected pages might cut off at 10 per minute or fewer. Cloudflare, AWS WAF, and most CDNs ship rate-limiting rules out of the box, so even sites that don't write custom logic can serve 429s. Scrapers most often hit them when they spin up many parallel workers without coordinating across them.
How to read a 429 response
Always check the `Retry-After` header first. It's either a number of seconds (`Retry-After: 30`) or an HTTP-date (`Retry-After: Wed, 26 May 2026 14:30:00 GMT`). Honor it. If it's missing, fall back to exponential backoff: wait 1s, then 2s, then 4s, capping at a few minutes. Look at the response body too — some APIs include a JSON object with the current limit, remaining quota, and reset time, which is more useful than guessing. Logs of when you saw 429s, against which endpoint, from which IP, are the data you'll need to tune your concurrency.
How scrapers handle 429 correctly
The wrong answer is to retry immediately or switch IPs and keep hammering — that escalates a soft rate limit into a hard ban. The right answer has three parts. First, slow the request rate per IP — if you're seeing 429s at 5 req/s, drop to 2 and back off further if they persist. Second, distribute load across more IPs via proxy rotation, but make each IP's request rate look human (one request per few seconds, not one per millisecond). Third, queue and retry with backoff: failed requests go to the back of the queue with a delay derived from `Retry-After` plus jitter. Production scrapers treat 429 as routine and budget for it, not as an error to alert on.