How headless browsers work
Under the hood, a headless browser is the same binary as the regular browser — Chromium ships a `--headless` flag and Firefox ships `-headless`. The browser starts a debugging protocol (Chrome DevTools Protocol for Chromium, WebDriver BiDi as the new cross-browser standard) and listens on a local port. Your automation library connects to that port and sends commands: navigate to a URL, wait for a selector, click this button, evaluate this JavaScript, return the HTML. The browser executes them inside its full rendering pipeline — same network stack, same V8/SpiderMonkey JavaScript engine, same DOM. The only thing missing is the window.
Why scrapers need headless browsers
Modern sites do not ship a complete HTML document anymore. They ship a thin shell and a JavaScript bundle that fetches data from internal APIs and renders the content client-side — React, Vue, Next.js, and Angular all work this way. A plain HTTP request gets you the shell, not the content. A headless browser executes the JavaScript, waits for the API calls to resolve, and gives you the final DOM. Headless browsers also handle the other moving parts of a real session: cookies, localStorage, redirects, form submissions, and WebSocket connections — all things a request library either doesn't do or doesn't do convincingly.
Headless browser detection
Anti-bot vendors actively look for headless browsers. The default Chrome headless mode leaks signals — the `navigator.webdriver` property is `true`, the User-Agent contains "HeadlessChrome", the window has no dimensions, plugins are missing, the WebGL renderer is generic. Stealth libraries (puppeteer-extra-plugin-stealth, playwright-stealth) patch the obvious ones; bot-detection vendors find new ones; the cat-and-mouse continues. Chrome's newer `--headless=new` mode closes most of the legacy leaks and is what serious scrapers use today. For very protected sites, even "new" headless is detectable — you need a real headful browser with display virtualization, which is what scraping APIs run internally.
When not to use a headless browser
Headless browsers are expensive — hundreds of MB of RAM per instance, seconds per page load. If the target site has an obvious internal API endpoint (open DevTools, Network tab, look for XHR/fetch calls returning JSON), call that directly with a plain HTTP request. If the site renders content server-side and ships complete HTML, a request library and an HTML parser will be 10–50x faster. Reach for the headless browser when the page genuinely needs JavaScript to render, when you need to interact with elements, or when the site fingerprints you before serving useful data.