SERP scraping in 2026
Google SERPs in 2026 mix organic results, AI overviews, knowledge panels, product carousels, and ad blocks — all rendered with JavaScript and personalized by user signals. A good SEO scraping API normalizes this into structured output: ranked organic positions, featured snippets, AI overview text, paid placements, and competitor citations. Country and device targeting are non-optional — Google's SERP differs significantly between desktop US and mobile DE.
On-page extraction
Once you have the URLs that rank, the on-page pass extracts: title, meta description, canonical, robots directive, hreflang alternates, all h1-h6 in document order, structured data (JSON-LD, microdata), Open Graph and Twitter cards, image alt counts, internal vs external link counts, and word count. For technical SEO add render-blocking JS, CSS file count, and the rendered vs source DOM diff.
Core Web Vitals require real browsers
Lighthouse-style metrics (LCP, INP, CLS) cannot be measured from a plain HTTP fetch. You need a real browser running on a network profile that matches Google's field data — usually a slow 4G simulation. Most scraping APIs offer this as a premium feature; budget for it on the pages that matter (homepage, top landing pages) rather than the full site.