HTTP headers checker

What are HTTP headers?

HTTP headers are key/value metadata that browsers and servers exchange on every request and response. Request headers tell the server who the client is (User-Agent), what content it understands (Accept, Accept-Language) and what state to maintain (Cookie). Response headers carry back the same kind of metadata about the document: its content type, caching rules, cookies to store, security policies, and clues about which CDN or WAF the request hit.

Why HTTP headers matter for web scraping

Almost every modern anti-bot system makes its first decision based on headers alone — before a single byte of HTML is rendered. A request with a stock python-requests/2.x User-Agent and no Accept-Language is flagged as a bot in microseconds. Sites use Accept-Language to infer geo, Sec-CH-UA-* client hints to cross-check the User-Agent, Referer to verify the click came from a real page, and Cookie to keep a session continuous. On the response side, Set-Cookie: __cf_bm, cf-ray, Server: cloudflare, or _abck tell you immediately whether you're up against Cloudflare Bot Management, Akamai, or DataDome. Reading both sides of the header conversation is the fastest way to debug a scraper that suddenly stopped working.

Common headers a scraper needs to send

Common headers a scraper sees in responses

How to set custom headers in Python, cURL, and Node.js

# Python (requests)
import requests
headers = {
    "User-Agent": "Mozilla/5.0 ...",
    "Accept-Language": "en-US,en;q=0.9",
    "Referer": "https://google.com/"
}
r = requests.get("https://example.com", headers=headers)
print(r.status_code, r.headers)

# cURL
curl -sI \
  -H "User-Agent: Mozilla/5.0 ..." \
  -H "Accept-Language: en-US,en;q=0.9" \
  https://example.com

// Node.js (built-in fetch)
const res = await fetch("https://example.com", {
  headers: {
    "User-Agent": "Mozilla/5.0 ...",
    "Accept-Language": "en-US,en;q=0.9"
  }
});
console.log(res.status, Object.fromEntries(res.headers));

Frequently asked questions