How rate limiting works
A server tracks how many requests have come from a given key - most often an IP address or an API token - and compares that count against an allowance. The two most common implementations are the token bucket and the sliding window. A token bucket hands each client a bucket that refills at a steady rate (say, one token per second up to a cap); each request spends a token, and when the bucket is empty, requests are rejected until it refills. A sliding window counts requests over the trailing N seconds. When you exceed the allowance, the server returns a 429 status, often with a Retry-After header telling you how many seconds to wait. Well-behaved clients read that header and pause; clients that ignore it and keep hammering frequently earn a longer, harder block.
Why rate limiting matters for web scraping
Rate limiting is the wall most scrapers hit first, before any CAPTCHA or fingerprint check. Fire requests as fast as your code can loop and you will trip a per-IP limit within seconds, and from then on the site sees a stream of 429s instead of data. The naive fix - add a sleep between requests - works for small jobs but does not scale: one IP at a polite rate is still one IP, and the whole job runs at that single IP's allowance. The real fix is to spread the load. Distribute requests across a pool of rotating proxies so no single IP exceeds the per-IP cap, while keeping each individual IP's rate plausibly human.
Staying under the limit
A robust scraper handles rate limits in layers. Respect the Retry-After header when you get a 429 and back off exponentially - wait 1s, then 2s, then 4s - instead of retrying instantly. Cap your concurrency so you are not sending fifty parallel requests to a site that allows ten. Spread traffic across many IPs via a rotating proxy pool, and add small random jitter to your timing so the pattern looks organic rather than a metronome. A managed scraping API folds all of this together: it pools IPs, paces requests per target site, and retries on 429 automatically, so you send one call and get the data without tuning backoff by hand. See also throttling and request retries.