PathScraping for AI agents and LLMs

Step 1 of 7

Web Scraping for LLMs and RAG: Reliable Context Pipelines

Pim · Scrappey Research

June 8, 2026 4 min read

Paste into ChatGPT, Claude, or any LLM

Web Scraping for LLMs and RAG: Reliable Context Pipelines — conceptual illustration

On this page

Web scraping for LLMs is the process of fetching web pages and converting them into clean, chunkable text (usually Markdown) that can be embedded into a vector store for retrieval-augmented generation (RAG) or passed directly into an agent's context. The pipeline is: fetch, clean (strip navigation, ads, and scripts), convert to Markdown, chunk, embed, store. The quality of every later step is capped by the first: if the fetch is blocked or the page never renders, retrieval has nothing real to work with. That is why fetch reliability - anti-bot handling and JavaScript rendering - rather than chunk size, is the limiting factor for most real-world RAG quality.

Pipeline	fetch -> clean -> Markdown -> chunk -> embed -> vector store
Output format	Markdown - preserves headings/lists/tables, strips nav/scripts, token-efficient
Typical chunk	300-800 tokens with 10-20% overlap; split on headings where possible
Limiting factor	Fetch reliability (anti-bot, JS rendering), not chunk tuning
Freshness	Re-crawl on a schedule; dedupe by content hash

The pipeline: from URL to vector store

A retrieval pipeline turns a list of URLs into searchable context in five steps. Fetch the page (the hard step, covered below). Clean and convert to Markdown, dropping navigation, footers, cookie banners, and scripts while keeping headings, lists, tables, and code. Chunk the Markdown into passages. Embed each chunk into a vector. Store the vectors with their source metadata in a vector database.

Markdown is the preferred intermediate format because it preserves document structure that raw text loses and strips the markup noise that raw HTML carries - a page that is 60 KB of HTML is often 4 KB of Markdown, which means more real content per token. Most managed web-data APIs can return Markdown directly, so the fetch and convert steps collapse into a single call (see the code example).

Chunking and embedding without losing structure

Chunking is where most retrieval quality is won or lost. A few rules that hold up in practice:

Split on structure, not character count. Break on Markdown headings first, then on paragraphs, so a chunk is a coherent idea rather than an arbitrary slice. This is exactly why you convert to Markdown before chunking.
Keep chunks at roughly 300-800 tokens with 10-20% overlap. Too large and the embedding blurs multiple topics; too small and it loses context. Overlap stops an answer from being cut in half at a boundary.
Attach metadata to every chunk - source URL, page title, heading path, fetch date. You need it for citations, for filtering, and for re-crawl freshness.
Dedupe by content hash. Boilerplate (the same footer or sidebar across a site) otherwise floods your index with near-identical chunks and crowds out real answers.

None of this matters if the text you chunked was wrong - which is the failure mode the next section covers.

Why pipelines that work in dev break in production

A RAG pipeline tested against a handful of friendly URLs almost always works. The same pipeline pointed at real targets degrades, and the cause is rarely the chunking - it is the fetch. Two things go wrong. First, many pages render their content with JavaScript, so a plain HTTP fetch returns an near-empty shell and you embed nothing useful. Second, sites behind bot protection (such as Cloudflare or DataDome) return a verification or challenge page instead of content.

The insidious part is that both failures return a 200 OK with a body, so a naive pipeline embeds the challenge page as if it were the article. Now your vector store is poisoned: retrieval surfaces "please enable JavaScript" and "verify you are human" as context, and the model answers from garbage. Retrieval accuracy is capped by scrape reliability, full stop.

The fix is to fetch through infrastructure that renders the page in a real browser, then returns clean Markdown - so the text entering your pipeline is the text a human would see. A managed web-data API does this in one call and, on pay-per-success pricing, you are not billed for the requests that fail, which pairs well with the retry loops a production crawler needs. For the broader context on agent-driven access, see AI agent tools and choosing a scraping API for LLM data.

Code example

python

import requests, hashlib

ENDPOINT = "https://publisher.scrappey.com/api/v1"
API_KEY = "YOUR_API_KEY"

def fetch_markdown(url: str) -> str:
    """Fetch a URL as clean Markdown via a real browser with verification handling."""
    resp = requests.post(
        ENDPOINT,
        params={"key": API_KEY},
        json={"cmd": "request.get", "url": url, "markdown": True},
        timeout=180,
    )
    resp.raise_for_status()
    return resp.json()["solution"]["markdown"]

def chunk(md: str, size: int = 600, overlap: int = 80) -> list[str]:
    """Split on blank lines first, then pack into ~size-token-ish chunks."""
    paras = [p for p in md.split("\n\n") if p.strip()]
    chunks, cur = [], ""
    for p in paras:
        if len(cur) + len(p) > size * 4:   # ~4 chars/token heuristic
            chunks.append(cur.strip())
            cur = cur[-overlap * 4:]
        cur += "\n\n" + p
    if cur.strip():
        chunks.append(cur.strip())
    return chunks

for url in ["https://example.com/docs/quickstart"]:
    md = fetch_markdown(url)
    for c in chunk(md):
        cid = hashlib.sha256(c.encode()).hexdigest()[:16]   # dedupe key
        # embed(c) -> vector_store.upsert(id=cid, text=c, metadata={"url": url})
        print(cid, len(c), "chars")

Next in Scraping for AI agents and LLMs · 2 of 7

The convention sites use to describe themselves to models.

What Is llms.txt?

The best web scraping API for LLM training data delivers clean, deduplicated, license-aware text at the scale training pipelines need — boil…

What Is AI Web Scraping?

AI web scraping is an approach that replaces CSS selectors with natural-language prompts, LLM-based extraction, and Markdown-first output. N…

What Is an MCP Server for Scraping?

An MCP server for scraping is a Model Context Protocol endpoint that exposes scraping tools (fetch, screenshot, parse, search) as callable f…

What Are AI Agent Tools?

AI agent tools are the callable functions an autonomous LLM agent uses to act on the world - searching, fetching web pages, running code, qu…

What Is llms.txt?

llms.txt is a proposed web standard - a Markdown file published at a site's root (/llms.txt) that gives large language models a curated, cle…

What Are Claude Skills?

Claude Skills are reusable capability packages - a folder containing a SKILL.md file plus optional scripts and reference files - that Claude…

Concept map

How Web Scraping for LLMs and RAG connects

The terms most directly tied to this one. Hover a node to see its neighbours, click to preview, drag to rearrange.

0 terms · 0 connections

You are here · Web Scraping APIs

Tools & solutions for this topic

Frequently asked questions

What format is best for feeding web pages to an LLM?

Markdown. It preserves document structure (headings, lists, tables) that plain text loses, while stripping the markup noise that raw HTML carries. The result is far more real content per token, which matters directly for both context-window cost and retrieval quality.

Why does my RAG pipeline return empty or garbage chunks?

Almost always the fetch step, not the chunking. JavaScript-rendered pages return a near-empty shell to a plain HTTP request, and bot-protected sites return a verification page - both as a 200 OK with a body. The pipeline then embeds that shell or challenge page as if it were content. Render in a real browser so the text you embed is the text a human sees.

How often should I re-scrape sources for RAG?

It depends on how fast the source changes - docs and pricing pages monthly or on change, news daily, reference content rarely. Store a fetch date and a content hash per chunk so you can re-crawl on a schedule and skip pages whose hash has not changed, which keeps both cost and index churn down.

Last updated: 2026-06-08