Your first R scraper with rvest
rvest reads a page and lets you select nodes with CSS or XPath, then pull out text or attributes into a vector — which drops straight into a data frame. Install with install.packages("rvest").
library(rvest)
library(dplyr)
page <- read_html("https://books.toscrape.com/")
books <- page %>% html_elements("article.product_pod")
titles <- books %>% html_element("h3 a") %>% html_attr("title")
prices <- books %>% html_element(".price_color") %>% html_text2()
df <- tibble(title = titles, price = prices)
print(df)
write.csv(df, "books.csv", row.names = FALSE)Note the current rvest API: html_elements() (plural) returns all matches, html_element() (singular) returns one per parent, html_text2() reads cleaned text, and html_attr() reads an attribute. Older tutorials use the retired html_nodes()/html_node() names — the new ones are the way to go.
Requests and politeness with httr2 + polite
For custom headers, query parameters, or POST requests, use httr2 — the modern successor to httr. It builds requests with a readable pipeline:
library(httr2)
library(rvest)
page <- request("https://books.toscrape.com/") %>%
req_user_agent("Mozilla/5.0 (compatible; research-bot)") %>%
req_perform() %>%
resp_body_html()
page %>% html_elements(".price_color") %>% html_text2()To scrape responsibly, the polite package wraps rvest to honour robots.txt, identify your scraper, and rate-limit automatically — a best practice almost every competitor guide omits:
library(polite)
library(rvest)
session <- bow("https://books.toscrape.com/",
user_agent = "polite R scraper ([email protected])")
prices <- scrape(session) %>%
html_elements(".price_color") %>%
html_text2()bow() introduces you to the host and checks robots.txt; scrape() then fetches within those rules.
Scraping JavaScript-rendered pages
rvest only reads the HTML the server returns — it does not run JavaScript. For client-side-rendered pages, RSelenium drives a real browser:
library(RSelenium)
driver <- rsDriver(browser = "chrome", chromever = "latest", verbose = FALSE)
remote <- driver$client
remote$navigate("https://quotes.toscrape.com/js/")
quotes <- remote$findElements(using = "css", ".quote .text")
texts <- sapply(quotes, function(q) q$getElementText()[[1]])
print(texts)
remote$close()
driver$server$stop()chromote is a newer, lighter alternative that controls headless Chrome directly without a separate Selenium server — worth preferring for new projects. As always, you can also find the page's JSON API and fetch it directly with httr2.
Which R package should you use?
| Package | Role | Runs JS? | Best for |
|---|---|---|---|
| rvest | Download + parse | No | The standard — CSS/XPath to data frames |
| httr2 | HTTP client | No | Custom headers, POST, APIs |
| polite | Politeness wrapper | No | robots.txt + rate-limiting |
| RSelenium | Browser automation | Yes | JavaScript-rendered pages |
| chromote | Headless Chrome | Yes | Lighter JS rendering, no Selenium server |
Start with rvest, add httr2 for request control and polite for good manners; reach for RSelenium or chromote only when JavaScript is involved.
The hard part: handling anti-bot blocking
R is excellent for analysing scraped data, but rvest's requests carry a TLS fingerprint and headers that Cloudflare, DataDome, and Akamai flag as non-browser. You cannot turn a 403 or CAPTCHA page into a data frame.
Handling serious anti-bot protection — residential proxies and a real browser fingerprint — is awkward to build in R. A scraping API handles it server-side; you POST the URL with httr2 and parse the returned HTML with rvest: