Web Scraping for Research: A Practical End-to-End Guide

Your crawler has finished, the spreadsheet opens, and the row count looks reassuring. Then you sort by publication date and find blank cells, repeated headlines, and prices that don't match the pages in your browser. The download worked. The measurement didn't.

Web scraping for research means turning web content into observations you can defend, not just collecting HTML. That requires a defined sampling frame, a recorded capture environment, and checks that distinguish a retrieved page from a valid record. If your question concerns what people saw, you also need to establish whether your crawler saw the same thing.

Map the Research Workflow Before Writing Code

Say a graduate student starts coding immediately, then discovers during analysis that a layout change left required fields empty. Without saved responses, they can't establish whether the information was absent from the website or lost during extraction. Rerunning the crawler produces today's pages, not the missing historical evidence.

Start with the research question and unit of analysis. A product, listing, seller, and daily price observation aren't interchangeable units. If you mix them, even perfectly parsed fields can produce misleading totals.

The University of Chicago's web-scraping guidance supports checking access instructions, identifying the crawler, and avoiding excessive load; its conservative default when a site supplies no limit is no more than one request every 10 seconds. Build those constraints into the project design rather than attaching them after collection.

Six phases of a research scraping project

Treat each done-when test as a gate, not an aspiration.

For a hypothetical pilot, you might select 50 to 200 pages across the templates you expect to encounter. That's a planning choice, not a statistically justified sample size. Include unusual pages deliberately, such as unavailable products and listings with missing dates.

Tie collection to the analysis

For marketplace research, Trendlytic's high-demand niches on Etsy can help generate candidate categories to investigate. It doesn't replace a sampling frame: define which listings qualify, how you discover them, and whether search ranking affects inclusion.

Compare research data collection methods before committing to scraping. An official export or API may answer the question with fewer moving parts.

Completion test: A research scrape isn't finished when the crawler stops. It's finished when someone else can reconstruct the accepted dataset from its documented inputs.

Legal and Ethical Checks Before the First Request

A publicly reachable page isn't automatically fair game for automated collection or redistribution. Reachability answers a technical question. Permission, contractual obligations, privacy, copyright, and institutional requirements need separate consideration.

Use the following checklist to assign an action to every unresolved issue.

Resolve access and reuse separately

• Public access: Confirm the target material can be reached without authentication. Keep login-only and paywalled material outside this workflow; if essential evidence sits there, seek an authorized access arrangement.
• Terms of Service: Save the terms and identify clauses governing automation, storage, and reuse. If the wording conflicts with your intended method, pause for permission or qualified legal advice.
• Publisher crawling rules: Archive robots.txt with its retrieval date. Treat disallow directives as a publisher signal requiring a decision, not an obstacle to route around.
• Personal information: Remove names, email addresses, profile images, and persistent identifiers unless your question needs them. Document why retained categories are necessary.
• Operational limits: Agree on request pacing and stop conditions before collection. Don't treat unused server capacity as permission to increase traffic.
• Institutional review: Ask the relevant ethics or legal office whether your design requires review, especially when records concern identifiable or vulnerable people.
• Accountability: Record the collection purpose, responsible researcher, access date, retention policy, and a contact route for complaints or takedown requests.

Read the actual service terms, not just the target website's homepage. For example, a project investigating podcast discovery should examine Podzone's podcast search usage terms before designing collection or reuse. The practical question is whether the intended activity fits the current terms, not whether similar services allow it.

When technical access and publisher preference conflict

The 2025 Global Bot Security Report, analyzing more than 16,900 websites, reported that 88.9% of observed robots.txt files explicitly disallowed GPTBot. That finding concerns a particular crawler, not a universal ban on research, but it illustrates why bot identity and directive scope matter.

robots.txt isn't an access-control mechanism. A server may deliver content despite a published objection to automated access. Successful retrieval doesn't resolve the ethical or legal question.

If a public site signals no, narrow the proposed collection, request permission, choose another source, or pause. Record that reasoning in the methods documentation, including any approved exception. This checklist is a risk-management process, not a legal determination.

Choosing Proxies, Headless Browsers and APIs

Open View Page Source and search for a required field. If the value appears in the server response, start with an HTTP client and HTML parser. If it appears only in the rendered page, investigate JavaScript execution, interaction, or an authorized structured endpoint.

Check both representations before selecting infrastructure.

Choosing a scraping architecture by page type

An API deserves inspection before adoption. Check pagination, update timing, deleted-record behavior, and whether search results represent the population you need. Resources such as Grantlas documentation for developers illustrate where to begin evaluating a documented interface rather than guessing how an application retrieves its content.

For publicly available website data, our Scrappey API accepts an HTTP POST containing a URL at a single endpoint. We run a real browser through a residential IP and return JSON containing status, full HTML, cookies, and LLM-ready Markdown. That replaces infrastructure work, not the researcher's obligation to validate the result.

Proxies change the observation context

Datacenter proxies are generally the simpler, lower-cost option to operate, but a site may treat hosting-network traffic differently from ordinary visitor traffic. Residential proxies offer a residential-network context, with additional sourcing, cost, and governance considerations. Neither guarantees acceptance or representative content.

Evaluate block rate on your authorized pilot, rather than assuming an address category will solve failures. A guide to backconnect residential proxies can help explain rotating connections, but rotation shouldn't become a way to disregard a site's limits.

Location routing controls apparent network location. It doesn't reproduce device type, cookies, or login state. Record browser version, locale, viewport, timezone, consent state, session persistence, and routing configuration wherever they may affect fields.

The video provides another format for reviewing the collection setup.

Choose the simplest architecture that retrieves the complete required fields under the approved capture conditions.

A full browser isn't more scientific. It is useful when its rendering or interaction capabilities are necessary to produce the observation your study defines.

A Reproducible Scraping Workflow in Python

Build the static-page pipeline around saved inputs, not live requests embedded throughout parsing code. Keep dependencies, selector configuration, and export logic in a version-controlled repository. The following fragments build on each other in the same Python process; install requests and beautifulsoup4 first.

The retrieval path differs for static and rendered pages, but both should produce an auditable snapshot.

Retrieve and preserve the evidence

1. Identify the crawler. Set RESEARCH_CONTACT to a monitored contact address; don't ship an anonymous default.

   import os import requests

   session = requests.Session() session.headers["User-Agent"] = ( f"ResearchCollector ({os.environ['RESEARCH_CONTACT']})" )

2. Bound the retries. This function requires explicit project settings and waits before each attempt, including retries.

   import time

   def fetch(url, delay, timeout, attempts): for attempt in range(attempts): time.sleep(delay * (attempt + 1)) response = session.get(url, timeout=timeout) if response.status_code not in (429, 503): response.raise_for_status() return response if attempt + 1 == attempts: response.raise_for_status()

These status codes distinguish rate limiting and temporary unavailability from ordinary success. In production, honor a valid Retry-After header and route repeated failures to the domain scheduler. Don't retry access denials indefinitely.

3. Save before parsing. A URL hash and UTC timestamp make each response traceable without placing URL characters in filenames.

   import hashlib from pathlib import Path from datetime import datetime, timezone

   def save_snapshot(url, response): stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") key = hashlib.sha256(url.encode()).hexdigest() path = Path("raw") / f"{key}_{stamp}.html" path.parent.mkdir(exist_ok=True) path.write_bytes(response.content) return path, stamp

Save a metadata sidecar containing response headers, declared encoding, final URL, status, and capture settings. The common mistake is saving only successful extracted rows, which removes the evidence needed to diagnose failures.

Parse deterministically and test the rerun

4. Version the selectors. Store them in selectors.json; this runnable setup creates an example configuration that must be adapted to the target template.

   import json

   config = { "version": "pilot", "record": "article", "title": "h2" } Path("selectors.json").write_text(json.dumps(config))

5. Attach provenance to every row. Parse the snapshot rather than issuing another request.

   from bs4 import BeautifulSoup

   def parse_snapshot(path, url, stamp, status): config = json.loads(Path("selectors.json").read_text()) soup = BeautifulSoup(path.read_bytes(), "html.parser") for item in soup.select(config["record"]): title = item.select_one(config["title"]) yield { "title": title.get_text(" ", strip=True) if title else None, "source_url": url, "fetched_at": stamp, "http_status": status, "parser_version": config["version"] }

6. Cache deliberately and log failures. Maintain a manifest mapping each capture to its snapshot and metadata. A parser rerun should use that manifest, while a new observation gets a new capture identifier.

   import logging

   logging.basicConfig(filename="failures.log", level=logging.INFO)

   def logged_fetch(url, **settings): try: return fetch(url, **settings) except requests.RequestException: logging.exception("Retrieval failed for %s", url) raise

7. Compare repeat outputs. Run extraction against the same frozen manifest, sort by stable record keys, and compare serialized outputs. Preserve the original capture timestamps; generating new ones during parsing creates artificial differences.

For dynamic pages, install Playwright and its Chromium browser, then substitute a browser retrieval function. Wait for the actual field, not an arbitrary sleep.

from playwright.sync_api import sync_playwright

def rendered_html(url, selector):
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch()
        page = browser.new_page()
        try:
            page.goto(url, wait_until="domcontentloaded")
            page.locator(selector).first.wait_for(state="visible")
            return page.content()
        finally:
            browser.close()

Save this rendered HTML with its browser context metadata. For longer runs, reuse a controlled browser context rather than launching a browser for every URL.

Validating, Storing and Exporting Clean Data

A crawler can return HTTP 200 while delivering a challenge page, an empty template, or unrelated content. Transport success belongs in operational logs. It isn't a record acceptance criterion.

Validate at field, sample, and dataset levels. The UCSB Library Carpentry guidance on scraping ethics and legality also emphasizes testing recursive collection on a smaller dataset and pacing requests to avoid harming the source.

Validation checks before a record is published

For a longitudinal study, URL alone isn't a unique key. Include the observation time or capture identifier so legitimate repeated measurements survive deduplication.

Create a human-labeled ground-truth set covering templates, locales, missing values, and edge cases. Measure precision as the share of extracted values that are correct, and completeness against the values expected in that labeled set. Freeze retrieval inputs when comparing parsers, then rerun the tests after layout changes.

Use a separate stratified audit of accepted records to catch errors outside the labeled fixtures. Practical data validation methods should produce rejection reasons, not merely a pass flag.

Store provenance and measure capture bias

Keep storage straightforward:

• Raw evidence: Immutable snapshots and capture metadata, with restricted access where appropriate.
• Clean tables: Accepted records plus a separate rejection table containing reasons.
• Data dictionary: Column types, units, selectors, missing-value meanings, and transformations.
• Exports: CSV for straightforward interchange or Parquet when typed, larger analytical tables are more useful.

Correct extraction doesn't establish correct exposure measurement. A 2025 study comparing browsing-session capture with later URL retrieval found that static, ex-situ scraping failed to determine at least 33.8% of participants' news exposure, with the collection environment causing most of the discrepancy.

For questions about what readers saw, compare a consented sample of in-session observations with your retrieved pages where feasible. Check headline, article body, overlays, and personalization-sensitive elements, not just URL agreement. If direct comparison isn't possible, describe the dataset as content retrieved under your capture conditions, not an exact reconstruction of audience exposure.

Scaling Safely With Rate Limits and Documentation

Scaling research collection means processing more observations under a stable method, not maximizing simultaneous requests. Assign each domain a shared queue and request budget so independent workers can't accidentally multiply its traffic.

Spread large jobs across collection windows consistent with the research question. A slow crawl protects the source, but it can introduce temporal differences across observations. Record ordering and timing so analysts can distinguish a real change from a collection-window effect.

Control the scheduler, not just individual scripts

• Coordinate by domain: Make all workers use the same limiter rather than running unrelated crawlers against the same site.
• Respond to pressure: Honor retry instructions and reduce traffic when rate-limit responses increase.
• Pause on denial: Investigate repeated access-denied responses instead of automatically changing identities.
• Isolate failures: Keep a blocked domain from consuming the worker pool or stalling unrelated sources.
• Reuse snapshots: Reprocess cached inputs without network retrieval; schedule fresh captures only when the study needs new observations.

Cache reuse avoids repeat network access, but parsing, storage, and managed infrastructure can still incur costs. Track cost per accepted record, not just cost per delivered response.

Methods note: If you change the capture environment during collection, record the boundary and test whether the resulting records remain comparable.

Make the README an executable record

A reviewer should be able to inspect the project without reading every function. Keep these entries current:

• Run identity: Capture window, timezone, code commit, parser version, and configuration file.
• Population definition: Discovery method, inclusion and exclusion rules, and stopping conditions.
• Request policy: Domain pacing, concurrency controls, timeouts, retries, and pause rules.
• Quality results: Missingness by field, rejected records, deduplication method, and changes after the pilot.
• Coverage limitations: Retrieval outcomes grouped by relevant page type, topic, country, or device context.
• Reproduction command: A repository command such as python pipeline.py manifest.json, with dependency setup and expected output filenames.

Use that command as the actual entry point, not illustrative documentation that drifts away from the code. Preserve configuration alongside each run so later analysts don't unknowingly apply today's settings to yesterday's evidence.

Decide Go or No-Go After Your First Pilot

Choose a single source, page type, and research question this week. Keep the pilot small enough to inspect directly, but diverse enough to include the templates and failure cases your full collection will encounter.

Make the decision explicit

• Define: Write the minimum field list and connect each field to an analysis variable.
• Authorize: Check for an official API and preserve the dated access rules before retrieval.
• Collect: Run with pacing, snapshots, cache controls, and failure logs enabled.
• Inspect: Compare a predefined manual sample with the rendered pages and label expected values.
• Score: Calculate field completeness and precision against acceptance thresholds chosen before inspecting results.

Go when the access basis is documented, essential fields meet the acceptance criteria, and manual checks support the capture method. Keep unresolved optional fields visibly missing rather than filling them with guesses.

No-go when you can't establish an acceptable access basis, essential observations remain unavailable, or systematic mismatches undermine the research question. An access block isn't automatically an engineering challenge to overcome. Sometimes the correct decision is to change sources or redesign the study.

After a go decision, increase collection gradually while preserving the tested configuration. Schedule the next expansion only after reviewing accepted records, failures, and coverage gaps from the preceding run.

We offer 150 free scrapes to start, so you can test a publicly accessible source before committing to a larger collection. Visit Scrappey to run a documented research pilot, then judge the output against your labeled sample and go or no-go criteria.