The basic algorithm
Parse the HTML (turn the raw text into a structure you can search). Select every <a> tag that has an href attribute. Resolve each href against the page's base URL - that is, combine a relative path like /about with the page's address to get a full URL (and respect the <base> tag if the page has one). Strip the fragment (the #section part, which only scrolls within a page). Drop javascript:, mailto:, tel:, and empty hrefs, since none of those are pages to crawl. Normalize so equivalent URLs look identical: lowercase the host, decode percent-encoding (turn %20 back into a space), and sort query parameters alphabetically. Finally, dedupe against the seen-set - the running list of URLs you have already collected.
JS-rendered links
Modern sites load many links via JavaScript: lazy-rendered cards, "load more" buttons, infinite scroll. A static HTML parser - one that only reads the originally downloaded HTML and never runs scripts - misses them entirely. You have two options. Either render the page in a real browser before extracting, so the JavaScript runs and the links appear; or find the underlying XHR endpoint (the background request the page makes to fetch its data) and crawl from its JSON response directly. Both are valid; the XHR path is usually cheaper if the endpoint is accessible.
Edge cases that cause missing pages
Links in data-href, data-url, or other custom attributes — most parsers ignore them because they only look at standard href. Links inside JSON-LD structured data (machine-readable metadata embedded in the page) — same problem. Links built dynamically from React state — only visible after the page renders. PDF/document URLs in <embed> and <object> tags — easy to skip. For a thorough crawl, audit one rendered page by hand and compare against your extractor's output to see what it is dropping.