The web protocols and primitives every scraper developer should understand — HTTP, cookies, and REST APIs.
HTTP (HyperText Transfer Protocol) is the set of rules browsers and servers use to talk to each other on the web.
An HTTP cookie is a small piece of data a website asks your browser to store and then send back on every later request to that site.
A REST API is a standard way for programs to read and change data over the web using ordinary HTTP requests.
IPv4 and IPv6 are the two versions of the Internet Protocol that give every device online an address.
To send HTTP Basic Authentication with curl, use the -u (or --user) flag: curl -u username:password https://example.com.
To POST JSON with curl, set the content type and pass the body: curl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://example.com/api.
To make curl ignore SSL certificate errors, add the -k (or --insecure) flag: curl -k https://example.com.