Why MCP changed scraping for AI agents
Before MCP, every AI-agent integration was bespoke. Claude's tool-use schema and OpenAI's function-calling schema differed; LangChain, LlamaIndex, and CrewAI each had their own glue. To give an agent scraping ability you wrote the scraping client, a JSON schema for each function, error-handling, and rate-limit logic, then pasted it into every agent framework you targeted.
MCP collapsed this to one server per tool, consumed by every MCP-capable client. Firecrawl, Browserbase, and Apify shipped MCP servers in early 2025; by late 2025 most managed scraping APIs expose one. The agent-side code is now a single MCP connection string in a config file. The scraping vendor handles fingerprinting, proxies, and CAPTCHA, and exposes a clean tool surface.
What tools an MCP scraping server typically exposes
The conventional surface across major scraping MCPs:
| Tool | What it does | Used when the agent… |
|---|---|---|
scrape(url) | Fetches and returns clean markdown or text | …knows the URL and just needs content |
search(query) | SERP scrape — returns ranked URLs + snippets | …needs to find a page first |
crawl(url, depth) | Recursive scrape with budget | …wants the whole site or section |
extract(url, schema) | LLM-extraction against a Pydantic-style schema | …needs structured data, not text |
browser.{click, type, ...} | Stateful browser session for interactive flows | …needs login, multi-step forms, infinite scroll |
screenshot(url) | Returns PNG for vision-model inspection | …needs to verify visual state |
Burp Suite's MCP server is the outlier — it exposes the security-research surface (intercept, modify, replay) rather than scraping primitives. It is included here because the recon workflows it enables overlap with mobile API discovery.
When MCP wins and when it doesn't
MCP wins when the workflow is agent-driven and the schedule is unpredictable: research assistants, customer-support bots that look things up, code agents that read documentation, content-generation pipelines that need fresh source material. The agent decides which URLs to scrape, the MCP server handles the how.
MCP does not win when the workflow is a known batch pipeline: scrape this 10k-product list every 12 hours, monitor these 500 SKUs every minute. For those, a traditional REST scraping API is cheaper, more predictable, and easier to monitor. MCP's value is the agent-orchestration glue, not the scraping itself.
The other catch is cost. MCP servers from managed vendors charge per-tool-invocation. An agent that scrapes 1000 URLs per task at $0.005 each costs $5 per task — fine for occasional research, expensive for production. Self-hosted MCP (Firecrawl's open-source variant, Crawl4AI's MCP wrapper, the webclaw Rust server) avoids this but reintroduces the operational burden.