The basic pattern (static HTML)
The plan is straightforward: download the page, parse the HTML, walk through every <a> tag that has an href, and turn each href into a full URL by resolving it against the document's base URL (a relative link like /page is just shorthand for the complete address). Strip the fragment - everything after the # - unless you specifically care about anchored links. Normalize the host to lowercase and put the path in a canonical (consistent) form. Drop the junk: empty hrefs, javascript: pseudo-links, and mailto: addresses. Finally, dedupe so the same URL is not listed twice.
When you need a real browser
Modern SPAs (single-page apps - sites that build the page in your browser with JavaScript) and infinite-scroll feeds add links to the DOM (the live, in-memory version of the page) only after the initial HTML loads. A plain static fetch never runs that JavaScript, so it misses those links. Use Playwright (or a JS-rendering scraping API), wait for the page to settle, then run document.querySelectorAll('a[href]') in the browser context to read the finished page. For infinite scroll, scroll to the bottom in steps and collect links after each scroll until no new ones appear.
Filtering for crawl pipelines
For a focused crawler you usually want fewer links, not more, so filter aggressively: same-domain only (or a domain allowlist), URL patterns that match real content paths (skip /login, /cart, and asset paths), and respect rel="nofollow" if you care about the crawled site's signal. rel="nofollow" is a hint a site adds to a link to say "do not pass ranking credit through here." For SEO link extraction, keep the rel attributes as metadata rather than filtering on them.