Proxies

How to Rotate Proxies in Python

By the Scrappey Research Team

How to Rotate Proxies in Python — conceptual illustration
On this page

To rotate proxies in Python you keep a pool of proxy URLs and switch which one you send through on each request (or each session), so the target site sees traffic spread across many IPs instead of one. A proxy is a relay that forwards your request, so the site logs the proxy's IP, not yours. Rotation means picking a different proxy from the pool per call, which keeps any single IP under the site's per-IP rate limit (the cap on how many requests one address may send) and means a flagged IP is just one to drop and replace. In practice you do this either by hand-rolling a pool over a list of proxies, or by pointing every request at a single rotating-residential gateway that picks the exit IP for you.

Quick facts

Rotation granularityPer-request (stateless) or per-session (sticky)
Common librariesrequests, curl_cffi, httpx, aiohttp
proxies dict format{"http": "http://user:pass@host:port", "https": ...}
Sticky sessionPin one IP via username token, e.g. user-session-abc123
Pool hygieneHealth-check IPs, retire on timeout/403/429, re-add on cooldown

Per-request vs. per-session rotation

The first decision is how often you change IPs, and it follows the workflow, not a preference. Per-request rotation picks a fresh proxy for every HTTP call. It's the right default for stateless scraping, where each URL stands alone and nothing carries over between requests (a product catalog, a list of independent pages). Spreading every call across the pool keeps any one IP well under the per-IP rate limit.

Per-session (sticky) rotation pins one IP to a sequence of requests. You need this whenever state carries forward: a login, a cart, paginated results behind a cookie, or anything where the server expects the same client across steps. A logged-in user keeps the same IP across a flow, so a sticky session keeps the cart, cookies, and session state consistent across the sequence of requests. With a list of your own proxies you implement stickiness by keeping the same proxy for a whole worker's run; with a rotating-residential gateway you append a session token to the username (commonly user-session-abc123) and the gateway holds that exit IP for a window like 1, 10, or 30 minutes.

A subtle trap with requests.Session: HTTP keep-alive pools a TCP connection, so once a session opens a socket to a rotating gateway, later requests can ride the same tunnel and hit the same exit node even though you configured per-request rotation. If you want a truly new IP per call against a gateway, send each request without reusing a keep-alive connection, or use one short-lived session per IP.

Building a pool and rotating with requests or curl_cffi

The minimal pattern is a list of proxy URLs plus a chooser. Round-robin (cycle the list in order) is predictable and easy to reason about; random selection avoids any IP-ordering pattern. For most jobs random is the safer default. Each proxy is passed as a proxies dict with http and https keys; the scheme on the value is the proxy's own protocol, so for an HTTP proxy both values start with http:// even for HTTPS targets.

  • requests: requests.get(url, proxies={"http": p, "https": p}, timeout=10). Pure Python, ubiquitous, but it sends a default Python TLS fingerprint, which differs from a browser's handshake.
  • curl_cffi: a drop-in-style client built on curl-impersonate that reproduces a real browser's TLS/JA3 handshake via impersonate="chrome", useful when a server expects a browser-style handshake. Same proxies dict shape. A very common mistake is putting https:// on the value of the https key; keep it http:// for an HTTP proxy.
  • httpx / aiohttp: for async pools, where you fan out many requests across the proxy list concurrently.

Rotating IPs reduces per-IP rate-limit hits (a 429 means you sent too many requests from one address) and IP-reputation blocks (a 403 once an IP looks suspicious), because the next request comes from a clean IP. It only affects per-IP limits and IP reputation, not the TLS handshake or request patterns; for sites that expect a browser-style handshake, a client like curl_cffi sends one.

Health-checking and retiring dead proxies

A static list rots fast: proxies time out, get blocked, or return garbage. A production rotator treats the pool as live state. Before a run, probe each proxy against a cheap endpoint (an IP-echo service such as https://httpbin.org/ip or https://api.ipify.org) and keep only the ones that answer quickly with a valid response. During the run, score outcomes: a connection timeout, a 407 (proxy auth failed), repeated 403s, or a sudden 429 are signals to pull that IP out of rotation. Don't delete it permanently for a single 429 — put it on a cooldown timer and re-admit it later, since rate limits are temporary by design.

Practical rules that mirror good rotating-proxy hygiene: keep rotation inside one geography so a session doesn't jump countries mid-flow; size the pool so the request rate per IP stays well within a polite, sustainable pace for the target; and log which IPs succeed against which targets so you can tune instead of guessing. Once you're past roughly 50,000-100,000 requests a day, maintaining and health-checking your own pool starts costing more engineering time than it saves. At that point a single rotating-residential endpoint that picks and retires exit IPs for you is usually the cleaner move, and a managed web-data API goes further by handling proxy rotation, browser rendering, and retries behind one request so your code just asks for the page.

Code example

python
import random
import time
import requests
# pip install curl_cffi  for the browser-TLS variant
from curl_cffi import requests as cffi

# A pool of your own proxy URLs (scheme is the PROXY's protocol -> http://)
PROXIES = [
    "http://user:[email protected]:8000",
    "http://user:[email protected]:8000",
    "http://user:[email protected]:8000",
]

# Dead/blocked proxies get parked here with a cooldown timestamp.
COOLDOWN = {}
COOLDOWN_SECONDS = 300


def live_pool():
    """Proxies not currently on cooldown."""
    now = time.time()
    return [p for p in PROXIES if now >= COOLDOWN.get(p, 0)]


def retire(proxy):
    COOLDOWN[proxy] = time.time() + COOLDOWN_SECONDS


def pick():
    pool = live_pool() or PROXIES  # fall back if everything is cooling down
    return random.choice(pool)


def health_check():
    """Drop proxies that can't reach a cheap IP-echo endpoint."""
    for p in list(PROXIES):
        try:
            r = requests.get(
                "https://api.ipify.org?format=json",
                proxies={"http": p, "https": p},
                timeout=8,
            )
            r.raise_for_status()
            print("ok", p, "->", r.json()["ip"])
        except Exception:
            retire(p)
            print("retired", p)


def fetch(url, attempts=4):
    """Per-request rotation with retry + proxy retirement."""
    for _ in range(attempts):
        proxy = pick()
        try:
            # curl_cffi impersonates a real Chrome TLS/JA3 fingerprint.
            r = cffi.get(
                url,
                proxies={"http": proxy, "https": proxy},
                impersonate="chrome",
                timeout=15,
            )
            if r.status_code == 429:   # rate-limited: cool this IP off
                retire(proxy)
                continue
            if r.status_code in (403, 407):  # blocked / proxy auth
                retire(proxy)
                continue
            r.raise_for_status()
            return r.text
        except Exception:
            retire(proxy)  # timeout / connection error -> park it
    raise RuntimeError(f"all attempts failed for {url}")


if __name__ == "__main__":
    health_check()
    html = fetch("https://httpbin.org/headers")
    print(html[:300])

# Sticky session against a rotating-residential gateway: pin one exit IP
# by adding a session token to the username, so a login/cart flow keeps
# the same address across steps.
STICKY = "http://user-session-abc123:[email protected]:8000"
with cffi.Session(impersonate="chrome") as s:
    s.proxies = {"http": STICKY, "https": STICKY}
    s.get("https://example.com/login")
    s.get("https://example.com/cart")  # same IP as the login above

Related terms

What Is a Rotating Proxy?
A rotating proxy is a proxy service that automatically gives each request — or each new session — a different outbound IP address, picked fr…
What Is a Residential Proxy?
A residential proxy sends your web traffic through a real home internet connection — a regular broadband or fiber line — instead of through …
What Is Proxy Web Scraping?
Proxy web scraping means sending your scraper's traffic through proxy servers — middleman machines that forward your requests for you — so t…
What Is the 429 Status Code (429 Error)?
HTTP 429 Too Many Requests is the status code a server returns when a client has sent more requests in a given window than the server's rate…
What Is curl_cffi?
curl_cffi is a Python HTTP client whose TLS fingerprint looks exactly like real Chrome, Firefox, or Safari. TLS is the encryption layer behi…

Concept map

How How to Rotate Proxies in Python connects

The terms most directly tied to this one. Hover a node to see its neighbours, click to preview, drag to rearrange.

0 terms · 0 connections
You are here · Proxies
Building map…

Tools & solutions for this topic

Frequently asked questions

Should I rotate on every request or keep one IP per session?

Match it to the workflow. Use per-request rotation for stateless scraping where each URL is independent and nothing carries over, because spreading every call across the pool keeps each IP under the rate limit. Use a sticky session, where one IP stays pinned across steps, whenever state carries forward — a login, a cart, or cookie-based pagination — since a real user's IP does not change on every click.

Why does my requests.Session keep using the same IP even though I rotate?

HTTP keep-alive pools a TCP connection, so once a Session opens a socket to a rotating gateway, later requests can reuse that same tunnel and exit through the same node. To force a new IP per call against a gateway, avoid reusing a keep-alive connection or use a fresh short-lived session per IP. With a list of distinct proxy URLs this is not an issue because each URL points at a different endpoint.

How do I detect and remove a dead proxy?

Probe each proxy against a cheap IP-echo endpoint such as https://api.ipify.org before a run and keep only the ones that respond quickly with a valid result. During the run, treat connection timeouts, 407 proxy-auth failures, and repeated 403s as signals to pull an IP from rotation. For a 429, park the IP on a cooldown timer and re-admit it later rather than deleting it, since rate limits are temporary.

Do I need rotating residential proxies, or is a datacenter pool enough?

It depends on the target. Datacenter proxies are cheaper and faster and are fine for unprotected sites and open APIs. Rotating residential proxies route through real consumer connections and hold up better where datacenter IP ranges are blocked, at higher cost. Rotation itself is a property of the gateway, not the IP type, so you can rotate either — choose residential when the site blocks datacenter ASNs.

Last updated: 2026-06-16 · Facts last verified: 2026-06-16