Web Scraping With Go (Golang)

The simplest Go scraper uses the standard library's net/http to fetch a page and goquery to parse it. Install goquery with go get github.com/PuerkitoBio/goquery.

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/PuerkitoBio/goquery"
)

func main() {
	res, err := http.Get("https://books.toscrape.com/")
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()

	doc, err := goquery.NewDocumentFromReader(res.Body)
	if err != nil {
		log.Fatal(err)
	}

	doc.Find("article.product_pod").Each(func(i int, s *goquery.Selection) {
		title, _ := s.Find("h3 a").Attr("title")
		price := s.Find(".price_color").Text()
		fmt.Printf("%s | %s\n", title, price)
	})
}

goquery mirrors jQuery: Find() takes a CSS selector, Each() iterates matches, Text() reads text, and Attr() returns an attribute plus an "exists" boolean. It is the most idiomatic way to parse HTML in Go.

For anything beyond a single page, Colly is the framework of choice. It gives you a collector with event callbacks, automatic link-following, and built-in concurrency and rate-limiting. Install: go get github.com/gocolly/colly/v2.

package main

import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector(
		colly.AllowedDomains("books.toscrape.com"),
		colly.UserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)"),
	)

	// Extract each product.
	c.OnHTML("article.product_pod", func(e *colly.HTMLElement) {
		title := e.ChildAttr("h3 a", "title")
		price := e.ChildText(".price_color")
		fmt.Printf("%s | %s\n", title, price)
	})

	// Follow the "next" pagination link.
	c.OnHTML("li.next > a", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Request.AbsoluteURL(e.Attr("href")))
	})

	c.Visit("https://books.toscrape.com/")
}

Colly's callback model — OnHTML, OnRequest, OnError — keeps crawlers readable as they grow. Enable parallelism with colly.Async(true) and a LimitRule to control concurrency and delay per domain.

Go's superpower for scraping is cheap concurrency. To fetch many URLs in parallel, launch goroutines and bound them with a semaphore channel so you do not overwhelm the target:

func fetchAll(urls []string) {
	sem := make(chan struct{}, 10) // max 10 in flight
	var wg sync.WaitGroup

	for _, u := range urls {
		wg.Add(1)
		go func(url string) {
			defer wg.Done()
			sem <- struct{}{}        // acquire
			defer func() { <-sem }() // release

			res, err := http.Get(url)
			if err != nil {
				return
			}
			defer res.Body.Close()
			// ...parse res.Body with goquery...
		}(u)
	}
	wg.Wait()
}

This pattern scrapes hundreds of pages concurrently while keeping a hard cap on simultaneous requests. Colly offers the same via its async mode and limit rules if you prefer the framework to manage it.

Use net/http + goquery for quick jobs, Colly for real crawlers, and chromedp or go-rod only when the page needs JavaScript.

Go is fast, but raw speed does not help when a site's anti-bot defenses reject the request. A net/http request has a TLS and header signature anti-bot vendors flag immediately, and even chromedp leaks headless signals. You cannot parse a Cloudflare or DataDome challenge page.

The fix — residential proxies, a real browser TLS fingerprint, and CAPTCHA challenges — is a lot to maintain in Go. A scraping API handles it server-side; your Go code POSTs the URL and parses the returned HTML with goquery: