What Is Web Scraping?

Every scraper has three stages: fetch, parse, store. Fetch sends an HTTP request to a URL and receives the response — usually HTML, sometimes JSON from an internal API endpoint. Parse extracts the fields you care about using CSS selectors, XPath, or regex; for JavaScript-rendered pages, the parser runs inside a headless browser that executes the page's scripts first. Store writes the cleaned data to a destination: a CSV, a Postgres table, a S3 bucket, or directly into an application. Each stage has its own failure modes — fetch fails on blocks, parse fails on layout changes, store fails on duplicates — and a production scraper is mostly the code that handles those failures gracefully.

The dominant use cases are commercial: e-commerce sites track competitor prices, travel aggregators pull flight and hotel inventory, recruiters build candidate lists from public profiles, and SEO teams audit SERPs and backlinks. Research and AI use cases are growing fast — large language models are trained on scraped web crawls, and academic researchers use scrapers to study everything from misinformation to housing markets. Internally, companies scrape their own public-facing sites for QA, monitoring, and content audits. The common thread is that the data is visible to anyone with a browser, but pulling it at scale requires automation.

For small jobs, Python's requests + BeautifulSoup or Node's axios + cheerio are still the default. For dynamic sites, Playwright and Puppeteer drive real browsers. For large crawls, Scrapy adds queues, retries, and pipelines on top. The next step up — when you start fighting Cloudflare, rotating thousands of proxies, or solving CAPTCHAs — is a managed scraping API like Scrappey, which handles the infrastructure layer so you only write the parsing logic. The right choice depends on volume, site difficulty, and how much of your time you want to spend on anti-bot defense rather than on the data itself.

Scraping public data is generally legal in the US (hiQ v. LinkedIn established this) and in most of Europe, but with caveats: you should respect robots.txt where possible, not scrape personal data without a lawful basis under GDPR, not bypass technical access controls in ways that could trigger CFAA or equivalent laws, and not republish copyrighted content as your own. Rate-limit yourself so you don't degrade the target site. When in doubt — especially for logged-in pages, paywalled content, or personal data — get a lawyer's opinion before shipping.