What Is a User Agent?

The user agent travels in the User-Agent request header, one of the standard headers sent with every HTTP request. A server can branch on it - serving a lighter page to an old browser, a mobile layout to a phone, or a block page to something it does not recognize. The string is purely advisory: nothing stops a client from sending whatever it wants, which is exactly why a scraper can set a Chrome user agent even though no Chrome is involved. The default user agents that HTTP libraries send are the problem. Out of the box, Python's requests library sends python-requests/2.31.0, curl sends curl/8.4.0, and Go sends Go-http-client/1.1. Any of those is a one-line rule for a site to block, because no human visitor ever sends them.

Setting a realistic user agent is the single cheapest improvement you can make to a scraper, and the most common rookie mistake is forgetting it. But a believable string is necessary, not sufficient. Modern bot detection treats the user agent as one claim among dozens and checks whether the rest of the request agrees with it. If you claim to be Chrome 126 on Windows but your TLS handshake fingerprint matches a Python library, your header order is alphabetical (browsers use a fixed, non-alphabetical order), and your Client Hints are missing, the user agent becomes evidence against you rather than cover. A coherent identity beats a fancy user agent every time.

Two rules cover most real-world use. First, rotate user agents from a pool of current, real strings so a long job does not send ten thousand requests from the exact same fingerprint - but keep the pool fresh, because a Chrome 90 string in 2026 is as suspicious as a default one. Second, and more important, match every other signal to the user agent you send: the Sec-CH-UA Client Hints, the Accept-Language header, the navigator properties a headless browser exposes, and the TLS fingerprint all have to tell the same story. Mismatched signals are what lie-detection systems look for.