HTML is a tree
Before parsing, it helps to see HTML as what it is: a nested tree of elements. <html> contains <body>, which contains <div>s, which contain <p>s and <a>s. A parser reads the raw markup string and builds this tree (the DOM) in memory, so instead of hunting through text with fragile string operations or regex, you navigate: "find every <a> inside the element with class product."
You select nodes in that tree two ways: CSS selectors (div.product > a) which every library below supports, or XPath (//div[@class="product"]/a) which lxml and parsel support and which can do things CSS cannot, like selecting by visible text or walking back up to a parent. See our XPath for web scraping guide for the full syntax.
Do not parse HTML with regular expressions. HTML is not a regular language; nested tags, optional attributes, and broken markup will break any regex eventually. Use a real parser.
BeautifulSoup — the beginner-friendly default
BeautifulSoup wraps a parser (use lxml as the backend for speed) in a famously forgiving API. Install with pip install beautifulsoup4 lxml.
from bs4 import BeautifulSoup
html = '''
<div class="product">
<h2>Wireless Mouse</h2>
<span class="price">$24.99</span>
<a href="/p/123">details</a>
</div>
'''
soup = BeautifulSoup(html, 'lxml')
# By tag, by class, by CSS selector:
print(soup.h2.text) # Wireless Mouse
print(soup.find('span', class_='price').text) # $24.99
print(soup.select_one('.product a')['href']) # /p/123
# Loop over many elements:
for a in soup.select('a[href]'):
print(a['href'], a.text.strip())
Key methods: find() / find_all() for tag-and-attribute matching, select() / select_one() for CSS selectors, .text for inner text, and ['attr'] for attributes. It tolerates broken, real-world HTML gracefully, which is why it is the most common starting point.
lxml — fast parsing with XPath
lxml is a C-backed library that is both fast and the standard way to use XPath in Python. Install with pip install lxml.
from lxml import html as lxml_html
doc = lxml_html.fromstring(html) # same markup as above
# CSS selectors:
print(doc.cssselect('.product h2')[0].text_content()) # Wireless Mouse
# XPath — selecting by attribute, text, or relationship:
print(doc.xpath('//span[@class="price"]/text()')[0]) # $24.99
print(doc.xpath('//a/@href')[0]) # /p/123
# XPath can select by visible text (CSS cannot):
link = doc.xpath('//a[contains(text(), "details")]/@href')
print(link) # ['/p/123']
Reach for lxml when you need XPath’s power (selecting by text, axes like parent:: / following-sibling::) or when BeautifulSoup is too slow. Many people use both: BeautifulSoup with features='lxml' gives the friendly API on top of the fast parser.
selectolax — the fastest parser
When you are parsing millions of pages, parser speed matters. selectolax wraps the C-based Modest/Lexbor engine and is typically several times faster than lxml and an order of magnitude faster than BeautifulSoup. Install with pip install selectolax.
from selectolax.parser import HTMLParser
tree = HTMLParser(html) # same markup as above
print(tree.css_first('h2').text()) # Wireless Mouse
print(tree.css_first('.price').text()) # $24.99
print(tree.css_first('.product a').attributes['href']) # /p/123
for node in tree.css('a[href]'):
print(node.attributes['href'], node.text(strip=True))
The API is CSS-selector based (css() / css_first()). It does not support XPath, but for the common case of CSS-selecting at high volume it is the fastest option in Python. Most competitor guides never mention it — it is the easiest performance win at scale.
Which parser should you use?
| Parser | Speed | Selectors | Best for | Install |
|---|---|---|---|---|
| BeautifulSoup | Slowest | CSS, find() | Learning, readable code, messy HTML | beautifulsoup4 |
| lxml | Fast | CSS + XPath | XPath power, speed | lxml |
| selectolax | Fastest | CSS | High-volume parsing | selectolax |
| html.parser | Slow | via BeautifulSoup | Zero dependencies | stdlib |
| parsel | Fast | CSS + XPath | Scrapy projects | parsel |
Rule of thumb: start with BeautifulSoup (on the lxml backend) for readability, switch to lxml directly when you need XPath, and to selectolax when speed at scale matters.
One caveat that no parser fixes: you can only parse HTML you actually received. If the page is JavaScript-rendered, the data will not be in the HTML at all, and if the site blocks you, you will be parsing a CAPTCHA or 403 page. A scraping API like Scrappey returns the fully rendered HTML so your parser always has real markup to work with.