Why MCP changed scraping for AI agents
Before MCP, every AI-agent integration was built from scratch. Claude's tool-use format and OpenAI's function-calling format were different; LangChain, LlamaIndex, and CrewAI each had their own wiring. To give an agent the ability to scrape, you had to write the scraping client, a JSON schema describing each function (its inputs and outputs), the error-handling, and the rate-limit logic - then copy all of it into every agent framework you wanted to support.
MCP collapsed this down to one server per tool, which any MCP-capable client can use. Firecrawl, Browserbase, and Apify shipped MCP servers in early 2025; by late 2025 most managed scraping APIs offer one. On the agent side, the code is now just a single MCP connection string in a config file. The scraping vendor handles the hard parts - fingerprinting, proxies, and CAPTCHA - and presents a clean set of tools.
What tools an MCP scraping server typically exposes
The conventional surface across major scraping MCPs:
| Tool | What it does | Used when the agent… |
|---|---|---|
scrape(url) | Fetches and returns clean markdown or text | …knows the URL and just needs content |
search(query) | SERP scrape — returns ranked URLs + snippets | …needs to find a page first |
crawl(url, depth) | Recursive scrape with budget | …wants the whole site or section |
extract(url, schema) | LLM-extraction against a Pydantic-style schema | …needs structured data, not text |
browser.{click, type, ...} | Stateful browser session for interactive flows | …needs login, multi-step forms, infinite scroll |
screenshot(url) | Returns PNG for vision-model inspection | …needs to verify visual state |
A quick read of the table: most tools are single-shot (give a URL, get content back), while browser.{click, type, ...} keeps a live, stateful session open so the agent can interact step by step - useful for logins or multi-page forms. The extract tool is the one that returns structured data shaped to a schema rather than raw text.
Burp Suite's MCP server is the outlier - it exposes the security-research surface (intercept, modify, replay) rather than scraping primitives. It is included here because the recon workflows it enables overlap with mobile API discovery.
When MCP wins and when it doesn't
MCP wins when the work is driven by an agent and the timing is unpredictable: research assistants, customer-support bots that look things up, code agents that read documentation, content-generation pipelines that need fresh source material. The agent decides which URLs to scrape; the MCP server handles the how.
MCP does not win when the work is a known, repeating batch job: scrape this 10k-product list every 12 hours, or monitor these 500 SKUs every minute. For those, a traditional REST scraping API - a plain HTTP endpoint you call on a fixed schedule - is cheaper, more predictable, and easier to monitor. MCP's value is the agent-orchestration glue, not the scraping itself.
The other catch is cost. MCP servers from managed vendors charge per tool call. An agent that scrapes 1000 URLs per task at $0.005 each costs $5 per task - fine for occasional research, expensive for production. Self-hosting your own MCP server (Firecrawl's open-source variant, Crawl4AI's MCP wrapper, the webclaw Rust server) avoids that per-call fee, but you take on the work of running the infrastructure yourself.