What triggers a 429
How to read a 429 response
Retry-After header first. It's either a number of seconds (Retry-After: 30) or a specific date and time (Retry-After: Wed, 26 May 2026 14:30:00 GMT). Honor whatever it says. If the header is missing, fall back to exponential backoff — wait 1s, then 2s, then 4s, doubling each time, capping at a few minutes. Look at the response body too: some APIs include a JSON object listing the current limit, how much quota you have left, and when it resets, which is far more useful than guessing. Keep logs of when you got 429s, against which endpoint, and from which IP — that record is the data you'll need to tune how many requests you send at once (your concurrency).How scrapers handle 429 correctly
Retry-After plus a little randomness (jitter), so your retries don't all fire at the same instant. Production scrapers treat 429 as routine and plan for it, rather than treating it as an error worth alerting on.How to fix a 429 in Python (requests)
The fix has three layers: honor Retry-After, back off exponentially with jitter when it's missing, and only then spread load across proxies. The function below does all three. It parses Retry-After whether the server sends seconds (Retry-After: 30) or an HTTP date, waits the right amount, and caps total attempts so a hard block doesn't loop forever.
The most common mistake — and the reason a lot of "too many requests python" searches end in frustration — is a bare time.sleep(5) retry loop that retries instantly on the next 429 and gets the IP banned. Adaptive backoff plus a realistic User-Agent (the default python-requests/2.x header is itself a rate-limit trigger on many sites) fixes the majority of cases. See the full snippet in the code example below.
If 429s persist after backoff the limit is probably per-IP, not per-account. At that point you need rotating proxies so each IP stays under the threshold — or a managed API that pools IPs and paces requests for you.
