Web Scraping With curl

The basics: follow redirects with -L, stay quiet with -s, request gzip with --compressed, and always set a realistic User-Agent with -A (curl's default UA is instantly blocked).

# GET a page, follow redirects, set a browser User-Agent, save to file:
curl -sL --compressed \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" \
  https://books.toscrape.com/ -o page.html

# Send extra headers (repeat -H as needed):
curl -s \
  -H "Accept-Language: en-US,en;q=0.9" \
  -H "Referer: https://www.google.com/" \
  https://example.com/

# Show response headers and timing without the body:
curl -sI https://example.com/
curl -s -o /dev/null -w "time_total: %{time_total}s\n" https://example.com/

-o file saves the body; -I fetches headers only; -w prints timing and status metrics. These cover inspecting and downloading almost any page.

Real scraping needs logins, form posts, and proxies. curl handles all of them:

# POST a form (login) and save the session cookies:
curl -s -c cookies.txt \
  -d "username=jane&password=secret" \
  https://example.com/login

# Reuse those cookies on a protected page:
curl -s -b cookies.txt https://example.com/account

# POST JSON to an API endpoint:
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d '{"query":"laptops","page":1}' \
  https://api.example.com/search

# Route the request through a proxy (rotate by changing -x):
curl -s -x http://user:[email protected]:8000 https://example.com/

-c writes a cookie jar, -b sends it back, so you can keep a session across requests. -x sends the request through an HTTP or SOCKS proxy, which is how you rotate IPs from the shell.

Here is what most curl tutorials skip: you can extract data without leaving the shell. Pipe HTML to pup (CSS selectors) or htmlq, and JSON to jq.

# Extract every product link (CSS selector -> href attribute) with pup:
curl -sL https://books.toscrape.com/ | pup 'h3 a attr{href}'

# Pull the text of each price into a list:
curl -sL https://books.toscrape.com/ | pup '.price_color text{}'

# Parse a JSON API response with jq:
curl -s https://api.example.com/items | jq -r '.items[].name'

# Combine: titles + prices into a quick CSV with htmlq + paste:
curl -sL https://books.toscrape.com/ | htmlq -t 'article.product_pod h3 a' > /tmp/t.txt
curl -sL https://books.toscrape.com/ | htmlq -t '.price_color'        > /tmp/p.txt
paste -d',' /tmp/t.txt /tmp/p.txt

This terminal-native pipeline — curl | pup | ... — is fast, scriptable, and needs no programming language. Install pup, htmlq, and jq from your package manager.

curl has two hard limits. First, it cannot run JavaScript — on a client-side-rendered page it only sees the empty HTML shell, so the data is not there to extract. Second, curl sends a TLS fingerprint and header set that no real browser produces, so anti-bot systems (Cloudflare, DataDome, Akamai) flag it immediately — and no combination of flags changes that.

You can address part of this with curl-impersonate (a build that reproduces a browser's TLS handshake), but JavaScript rendering and CAPTCHA challenges are out of reach for curl alone. A scraping API keeps the simple curl workflow but renders JavaScript server-side — you still pipe the result to pup or jq: