How retry logic works
A retry wrapper sits around your HTTP call. When a request fails, it inspects the failure to decide whether retrying could help. Transient errors - a network timeout, a connection reset, or a 5xx server error - are worth retrying because the cause is temporary. A 429 rate-limit is retriable too, after waiting for the limit to reset. Most 4xx client errors are not: a 404 means the page is not there, and a 401 means you are not authenticated, so re-sending the identical request just wastes attempts. When a retry is warranted, the wrapper waits, then re-sends, up to a maximum number of attempts before it gives up and raises the error.
Exponential backoff and jitter
Retrying instantly is the wrong move - if a server is overloaded, an immediate retry adds to the load that caused the failure. The standard pattern is exponential backoff: wait 1 second, then 2, then 4, then 8, doubling the delay each attempt so a struggling server gets room to recover. On top of that, add jitter - a small random offset to each wait - so that many clients retrying at once do not all fire again at the exact same instant (the "thundering herd" problem). When the server sends a Retry-After header, honor it: it tells you exactly how long to wait, which beats any guess. This same backoff logic underpins polite rate-limit handling.
Why retries matter for web scraping
At scale, transient failures are not the exception - they are constant. A job fetching a million pages will hit thousands of timeouts and temporary blocks purely by volume, and a scraper without retries simply loses that data. Good retry logic recovers it automatically. The one thing to watch is idempotency: a GET is safe to retry because re-fetching a page has no side effect, but retrying a POST that places an order could place it twice, so non-idempotent requests need care (an idempotency key, or no retry). A managed scraping API builds retries in - it detects soft failures like a block page returned with a 200 status, rotates to a fresh IP, and retries transparently, so you get the data instead of an error. See also self-healing scrapers.