Why 403s happen in scraping
On a normal API, a 403 means you authenticated but lack permission. In scraping, 403 almost always means an anti-bot system fired. The trigger could be any of: a known datacenter IP range, a missing or suspicious User-Agent, a TLS fingerprint that doesn't match the User-Agent, a request rate that crossed a threshold, a geo block on your country, or a behavioral signal collected on an earlier page. Cloudflare in particular returns 403 with its branded challenge page as the response body — if you see "Cloudflare" and a ray ID in the HTML, that's the source. The 403 itself is the symptom; the actual decision happened a layer above.
How to diagnose a 403
Open the response body — that's where the truth is. A short "Forbidden" page usually means a simple WAF rule; a Cloudflare/Akamai/DataDome branded page means a managed bot detection service. Check the response headers for `cf-ray`, `x-amzn-waf-action`, `server: AkamaiGHost`, or `x-datadome` to identify the vendor. Then check what you're sending: User-Agent realistic? Accept-Language present? TLS fingerprint matching the browser you're claiming to be? Try the same URL from a residential proxy in the target's primary geo — if that works, the block is IP-level. If it still fails, the block is fingerprint-level.
How to recover from a 403
The fix depends on the cause. IP-level 403s clear with residential or mobile proxy rotation. Header-level 403s clear with realistic headers — copy a real browser's headers from DevTools verbatim. Fingerprint-level 403s require a real-browser stack: Playwright with stealth patches, or a managed scraping API that maintains fingerprint pools. CAPTCHA-redirect 403s require a solver. Geo-403s require a proxy in the right country. Never retry a 403 without changing something — repeat 403s from the same identity reinforce the block.