Web Scraping APIs Glossary

A CAPTCHA solver is software that automatically completes CAPTCHA challenges for an automated client.

Web scraping is the automated extraction of structured data from websites.

A web scraping API is a hosted HTTP service that visits a web page for you and hands back the result — rendered HTML, JSON, or already-parsed data.

A headless browser is a real web browser — Chrome, Firefox, or WebKit — that runs without a visible window, driven entirely by code instead of by a person clicking.

Browser fingerprinting is a technique that identifies and tracks a visitor by combining dozens of small, observable characteristics of their browser and device into a single distin.

curl_cffi is a Python HTTP client whose TLS fingerprint looks exactly like real Chrome, Firefox, or Safari.

Camoufox is a fork of Firefox with anti-fingerprinting patches applied at the C++ build level.

AI web scraping is an approach that replaces CSS selectors with natural-language prompts, LLM-based extraction, and Markdown-first output.

Mobile API scraping means watching the traffic a vendor's phone app sends to its servers, then making those same requests yourself from Python or any HTTP client.

CloakBrowser is a Chromium build with 49 C++ binary patches that give it a consistent browser configuration.

PatchRight is a browser-automation library that edits Playwright's own Python code before Chrome launches, instead of injecting JavaScript into the page after it loads.

Firecrawl is a web-scraping API built for AI: you hand it a URL and it hands back clean Markdown or JSON — no CSS selectors, no XPath, no HTML parsing on your end.

Schema-validated LLM extraction is the standard production pattern for AI scraping: you describe the data you want as a Pydantic schema (a Python class that defines field names and.

Botasaurus is a free, open-source (MIT-licensed) Python framework for building web scrapers.

Crawl4AI is the most-starred open-source LLM-friendly web crawler on GitHub — 66.3k stars under Apache 2.0 license, maintained by UncleCode.

The Burp Suite MCP Server is an official PortSwigger extension (released 3 April 2025) that exposes Burp's HTTP history, Repeater, Intruder, Collaborator, and proxy controls as Mod.

The web scraping decision flow is a six-step checklist, ordered cheapest-first, that experienced engineers run through on every new target they are permitted to access.

A Computer Use Agent (CUA) is an AI agent that acts like a person at a keyboard: it logs into a portal as the user, clicks through the screens, deals with MFA (multi-factor login c.

A self-healing scraper is a scraper that notices, while it is running, that the rules it uses to find data on a page have stopped working — and then fixes those rules on its own.

The best web scraping API for JavaScript-rendered sites runs a real headless browser per request, executes the page's JavaScript, waits for dynamic content to load, and returns the.

The best web scraping API for e-commerce price monitoring is one that reliably pulls accurate, location-correct product data from major retailers (large marketplaces and hosted-pla.

The best web scraping API for SEO audits combines reliable SERP scraping (Google, Bing, regional engines) with on-page extraction — title, meta, headings, schema, internal links, r.

The best web scraping API for LLM training data delivers clean, deduplicated, license-aware text at the scale training pipelines need — boilerplate stripped, main content extracted.

The best web scraping API for competitor research covers the full surface a strategy team needs to monitor — pricing pages, product detail, content marketing, ad copy, review platf.

Getting all links from a webpage means downloading the page, reading every <a href> attribute (the URL inside each link tag), turning relative URLs into full ones, cleaning them up.

Infinite scroll is the page design where new content keeps loading on its own as you scroll down (like a social feed that never ends).

Reverse-engineering API requests for scraping means watching the network traffic a website makes, spotting the JSON endpoints that feed its visible UI, and calling those endpoints .

Synchronous web scraping sends one request at a time and waits ("blocks") until each one finishes before starting the next; asynchronous scraping fires off many requests at once an.

Batch web scraping means handing a whole list of URLs to a service as one job, letting it work through them in the background, and collecting the results once they are ready — inst.

Stateful web scraping means keeping the same identity across many requests - the same cookies, session tokens, browser fingerprint, and proxy IP - so the site sees one consistent v.

The Chrome DevTools Protocol (CDP) is the low-level interface for instrumenting and controlling Chromium-based browsers.

An MCP server for scraping is a Model Context Protocol endpoint that exposes scraping tools (fetch, screenshot, parse, search) as callable functions an AI agent can invoke.

The Scrapy + Go TLS sidecar architecture is the most common production pattern for scraping Akamai- and Cloudflare-protected sites at scale.

"Web scraping tools" is the whole family of software you use to pull data off websites — and in 2026 that family is big but neatly sorted into roles.

Playwright is a cross-browser automation framework from Microsoft that drives Chromium, Firefox, and WebKit through a single API.

Puppeteer is Google's Node.js library for driving a Chromium browser from code, over the Chrome DevTools Protocol (CDP) - the same channel Chrome's own DevTools use to talk to the .

Selenium is the original cross-browser automation framework — the W3C WebDriver standard predates Puppeteer by a decade.

Scrapy is the industry-default crawler framework for Python.

mitmproxy is a free tool that sits between an app and the internet so you can read and change the HTTPS traffic passing through it.

SeleniumBase is a Python framework for automating and testing browsers, built on top of Selenium 4.

XDriver is a browser-automation tool for Playwright (a browser-automation library): one command swaps Playwright's internal driver files for versions that reduce common automation .

Scrapling is an all-in-one Python scraping framework that bundles fetching, parsing, anti-detection, and crawling behind one API — it is a layer above the other tools, not a compet.

Obscura is an open-source headless browser engine written from scratch in Rust — not a fork or patch of Chrome or Firefox.

Anti-detect browser tools aim to present a consistent, real-looking browser configuration so that automated sessions render the same fingerprint signals a normal browser would — th.

jsoup is a free Java library that reads HTML and lets you pull data out of it.

Data parsing is the process of taking raw, messy data and turning it into a clean, structured format your program can use.

Web scraping as a service (WSaaS) is a managed, cloud-based offering that handles web data extraction for you through an API or dashboard - including the proxies, browsers, and ant.

PyQuery is a Python library for parsing and manipulating HTML and XML using a jQuery-like syntax.

A browser-automation-engine benchmark drives several automation stacks through the same set of targets and records, side by side, how often each one reaches real page content, how .

Choosing an anti-detect browser tool comes down to matching the tool's strengths to the detection layer you actually face - no single tool is best at everything, and none is truly .

A user agent is a short text string a client sends in the User-Agent HTTP header to tell a server what software is making the request.

Rate limiting is a control that caps how many requests a single client can make to a server within a fixed time window.

A CAPTCHA is a challenge a website uses to tell a human visitor apart from an automated script.

Request retries are the practice of automatically re-sending an HTTP request that failed, instead of giving up on the first error.

A web unblocker is a managed service that sits between your scraper and a target site, automatically handling the proxies, browser rendering, and verification needed to retrieve a .

A CSS selector is a pattern that picks out specific elements in an HTML document by matching their tag, class, id, attributes, or position.

XPath (XML Path Language) is a query language for navigating the tree structure of an HTML or XML document to select elements by their path, attributes, or text content.

JavaScript rendering is the process of executing a page's JavaScript in a real browser engine so that content built on the client side appears before you extract it.

A regular expression (regex) is a compact pattern that describes a set of strings, used to find, match, and extract text.

OCR (optical character recognition) is technology that converts text shown inside an image into machine-readable text characters.

Scraping publicly available data is generally legal, but legality depends on what you collect, how you collect it, and what you do with it — not on web scraping as an activity in i.

To scrape website data into Excel, fetch the page through a scraping API that returns structured JSON, load the rows into a Python list of dictionaries, then write them to an .xlsx.

Claude Skills are reusable capability packages - a folder containing a SKILL.md file plus optional scripts and reference files - that Claude discovers and loads on demand to perfor.

AI agent tools are the callable functions an autonomous LLM agent uses to act on the world - searching, fetching web pages, running code, querying APIs - rather than only generatin.

llms.txt is a proposed web standard - a Markdown file published at a site's root (/llms.txt) that gives large language models a curated, clean map of the site's most important cont.

Web scraping for LLMs is the process of fetching web pages and converting them into clean, chunkable text (usually Markdown) that can be embedded into a vector store for retrieval-.

To get scraped data into Google Sheets you either write rows from code with the gspread library and a Google service account, or pull a published feed into a cell with the built-in.

Export scraped data to CSV when you need flat, spreadsheet-ready rows, and to JSON when you need to preserve nested structure.

To scrape prices reliably you fetch each product page through a residential proxy in the right country, parse the current price out of the page (or let a scraping API return it as .