What Is Link Extraction in Web Crawling?

Paste into ChatGPT, Claude, or any LLM

On this page

Link extraction is the step in a crawl where you pull every URL out of a fetched page so you can decide which to follow next. The output is a deduplicated, normalized list of absolute URLs. The implementation looks simple but the edge cases (relative URLs, fragment-only anchors, JS-only links, data-href attributes, links inside event handlers) make it the source of most "missing pages" bugs in crawlers.

Source	<a href> primarily; also <link>, <area>, <iframe src>
Steps	Parse → resolve relative URLs → strip fragments → normalize → dedupe
Easy to miss	JS-rendered links, data-* attributes, onclick handlers
Normalize	Lowercase host, sort query params, strip trailing slashes consistently
Output	Set of absolute, normalized URLs for the frontier

The basic algorithm

Parse the HTML. Select every <a> with an href attribute. Resolve each href against the page's base URL (handle <base> tag if present). Strip the fragment (#section). Drop javascript:, mailto:, tel:, and empty hrefs. Normalize: lowercase the host, decode percent-encoding, sort query params alphabetically. Dedupe against the seen-set.

JS-rendered links

Modern sites load many links via JavaScript: lazy-rendered cards, "load more" buttons, infinite scroll. A static HTML parser misses them entirely. Either render the page in a browser before extracting, or identify the underlying XHR endpoint and crawl from its JSON response directly. Both are valid; the XHR path is usually cheaper if the endpoint is accessible.

Edge cases that cause missing pages

Links in data-href, data-url, or other custom attributes — most parsers ignore them. Links inside JSON-LD structured data — same. Links built dynamically from React state — only visible after rendering. PDF/document URLs in <embed> and <object> tags — easy to skip. For a thorough crawl, audit one rendered page by hand and compare against your extractor's output.

Code example

python

from urllib.parse import urljoin, urldefrag, urlparse, parse_qsl, urlencode
from bs4 import BeautifulSoup

def extract_links(html, base_url):
    soup = BeautifulSoup(html, 'html.parser')
    base = soup.find('base', href=True)
    base_url = urljoin(base_url, base['href']) if base else base_url
    links = set()
    for a in soup.select('a[href]'):
        href = a['href'].strip()
        if not href or href.startswith(('javascript:', 'mailto:', 'tel:')):
            continue
        clean = urldefrag(urljoin(base_url, href)).url
        p = urlparse(clean)
        sorted_q = urlencode(sorted(parse_qsl(p.query)))
        links.add(p._replace(query=sorted_q).geturl())
    return links

Related terms

How to Get All Links From a Webpage

Getting all links from a webpage means fetching the page, parsing every <a href> attribute, resolving relative URLs against the base, normal…

What Is a Web Crawler?

A web crawler is software that systematically discovers and fetches web pages by following links from a starting set (seed URLs), building u…

What Is a Sitemap?

A sitemap is an XML (or sometimes plain-text) file that lists a site's canonical URLs along with optional metadata: last-modified date, chan…

What Is Crawl Depth Limit?

Crawl depth limit is the maximum number of link hops a crawler will follow from a seed URL. Depth 0 fetches only the seed; depth 1 fetches t…

What Is a 404 Error?

HTTP 404 Not Found means the server understood the request but has no resource at that URL. In normal web traffic 404 is honest — the page i…

Best Web Scraping API for SEO Audits

The best web scraping API for SEO audits combines reliable SERP scraping (Google, Bing, regional engines) with on-page extraction — title, m…

Concept map

How Link Extraction connects

The terms most directly tied to this one. Hover a node to see its neighbours, click to preview, drag to rearrange.

0 terms · 0 connections

You are here · Crawling

Tools & solutions for this topic

Frequently asked questions

Should I normalize trailing slashes?

Yes, but pick one rule and apply it consistently. Trailing-slash and no-trailing-slash of the same path are usually the same page; treating them as different doubles your crawl budget for nothing.

Do I extract links from PDFs?

Only if your scope requires it. PDF link extraction is its own problem (pdfplumber or similar) and most crawls do not need it.

What about links in noindex pages?

For SEO crawls, follow them but tag them as "from-noindex" — search engines do not credit them for ranking. For data crawls, follow them like any other link.

Last updated: 2026-05-26