Set a User-Agent in Python Requests

There are two ways to set the User-Agent, and the choice comes down to how many requests you are making. For a single call, pass a headers dict:

import requests

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                         "AppleWebKit/537.36 (KHTML, like Gecko) "
                         "Chrome/126.0.0.0 Safari/537.36"}
r = requests.get("https://httpbin.org/user-agent", headers=headers)
print(r.json())   # {"user-agent": "Mozilla/5.0 ..."}

For anything more than one request, use a Session. Setting the header once on session.headers applies it to every request the session makes, and the session also persists cookies and reuses TCP connections, which is both faster and more realistic:

session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 ... Chrome/126.0.0.0 Safari/537.36"})

session.get("https://example.com/")          # warm up, collect cookies
data = session.get("https://example.com/api")  # same UA + cookies reused

Two practical notes. Header keys are case-insensitive in requests, so "User-Agent" and "user-agent" behave the same. And a per-request headers dict is merged on top of the session headers, so you can override the UA for a single call without changing the session default.

The reason the default fails is that python-requests/2.32.3 is an honest self-identification: no human browser ever sends it, so a single string match flags the request. Swapping in a real Chrome or Firefox string clears that specific check and is enough for many simple, unprotected sites. See what a User-Agent is for the full anatomy of the string.

The catch is that real browsers do not send only a User-Agent. They send a consistent set of headers in a specific order: Accept, Accept-Language, Accept-Encoding, Sec-Ch-Ua client hints, Sec-Fetch-* metadata, and so on. A request that carries a Chrome User-Agent but omits the headers a real Chrome sends is internally inconsistent, and servers may treat a self-identified Chrome client differently from one whose headers do not match. So the right move is to mirror a full browser header set, not just the one field:

browser_headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,"
              "image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Ch-Ua": '"Chromium";v="126", "Google Chrome";v="126", "Not.A/Brand";v="24"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Upgrade-Insecure-Requests": "1",
}

Keep the values internally consistent: a UA claiming Windows should pair with Sec-Ch-Ua-Platform: "Windows", and the Chrome version in the UA should match the version in Sec-Ch-Ua. Internally inconsistent header sets are an obvious mismatch, so keep the platform and version values aligned with the User-Agent.

If you send many requests, rotating the User-Agent lets you test or represent several browser/OS combinations rather than a single fixed one. Use a small pool of current strings, since outdated browser versions can cause sites to serve different markup and, ideally, swap the whole header set together so the UA and client hints always agree:

import random

UA_POOL = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.4 Safari/605.1.15",
]
session.headers["User-Agent"] = random.choice(UA_POOL)

Here is the limit that surprises people. You can perfect every header and still get a 403 error, because anti-bot systems also read the TLS fingerprint — a JA3/JA4 hash computed from the encrypted handshake that happens before any header is sent. Python's requests negotiates TLS through OpenSSL with a fixed cipher list, producing one well-known signature that no browser emits. The HTTP headers identify Chrome while the TLS handshake is the one Python's requests produces, so the two describe different clients even though your code never sees the handshake directly.

Plain requests cannot change its TLS fingerprint. To make the handshake match the browser in your User-Agent, switch to curl_cffi, which wraps curl-impersonate and replicates a browser's exact cipher list, extension order, and HTTP/2 settings with a single impersonate="chrome" argument. When even that is not enough because the page runs JavaScript challenges, a managed web data API like Scrappey handles the headers, TLS fingerprint, proxies, and browser rendering in one call.