What Is Web Scraping?

Every scraper has three stages: fetch, parse, store. Fetch sends an HTTP request to a URL and receives the response — usually HTML, sometimes JSON returned by a site's internal API (a hidden data endpoint the page itself calls). Parse picks out the fields you care about using locators like CSS selectors, XPath, or regex (pattern-matching for text). When a page builds itself with JavaScript after loading, the parser first runs that code inside a headless browser — a real browser with no visible window. Store writes the cleaned data to a destination: a CSV file, a Postgres table, an S3 bucket, or directly into an application. Each stage has its own way of breaking — fetch fails when you get blocked, parse fails when the site changes its layout, store fails on duplicate records — so in practice most of a production scraper is the code that handles those failures gracefully.

Most scraping is commercial. E-commerce sites track competitor prices, travel aggregators pull flight and hotel availability, recruiters build candidate lists from public profiles, and SEO teams audit search results (SERPs) and backlinks. Research and AI uses are growing fast: large language models are trained on scraped web crawls, and academics use scrapers to study everything from misinformation to housing markets. Companies also scrape their own public sites for QA, monitoring, and content audits. The common thread is that the data is already visible to anyone with a browser — but collecting it at scale takes automation rather than manual effort.

For small jobs, the defaults are still Python's requests + BeautifulSoup or Node's axios + cheerio — lightweight libraries that fetch a page and pick fields out of the HTML. For dynamic sites that need JavaScript to run, Playwright and Puppeteer drive real browsers. For large crawls, Scrapy adds queues, retries, and pipelines on top. The next step up — once you're fighting Cloudflare, rotating thousands of proxies, or solving CAPTCHAs — is a managed scraping API like Scrappey, which runs that infrastructure for you so you only write the parsing logic. The right choice depends on volume, how hard the site is to access, and how much of your time you want to spend on anti-bot defense rather than on the data itself.

Scraping public data is generally legal in the US (the landmark hiQ Labs case established this) and across most of Europe — but with caveats. You should respect robots.txt (the file where a site states which paths bots may visit) where possible; avoid scraping personal data without a lawful basis under GDPR (the EU's data-protection law); don't bypass technical access controls in ways that could trigger the CFAA (a US computer-access law) or its equivalents; and don't republish copyrighted content as your own. Rate-limit yourself — space out your requests — so you don't slow down the target site. When in doubt, especially for logged-in pages, paywalled content, or personal data, get a lawyer's opinion before shipping.