Try 25 MB of US mobile proxy data free for 30 days.Start free trial
November 6, 202612 min read

Golang Proxy Scraping: HTTP, SOCKS5 and Rotation in Go

Isometric forms representing Go proxy configuration

To scrape through a proxy in Go, set the Proxy field on an http.Transport (HTTP or HTTPS proxies) or build a dialer with golang.org/x/net/proxy (SOCKS5), then reuse one client across goroutines. Add rotation by returning a different proxy URL per request, and wrap every call in timeouts and capped retries. This guide gives you working code for each step, the errors you will meet, and how real mobile carrier IPs fit in.

TL;DR

  • Use http.ProxyURL on a shared http.Transport for HTTP/HTTPS proxies. Credentials go in the URL's user info.
  • For SOCKS5, proxy.SOCKS5 from golang.org/x/net/proxy handles username/password auth and plugs into DialContext.
  • Rotate inside the Transport.Proxy function. Never swap the field while goroutines are running.
  • Set a client timeout, retry with backoff and jitter, and cap total time per URL.
  • Mobile carrier IPs from MaskLabs give your requests the profile of real mobile traffic, with sticky or rotating sessions in 46 US cities.

Table of Contents

How Go sends a request through a proxy

Every http.Client in Go sends requests through an http.Transport. The transport decides, per request, whether to go direct or through a proxy. That decision lives in one field: Transport.Proxy, a function that takes the *http.Request and returns a proxy URL (or nil for direct).

The Go team has been explicit about how to use it. According to the net/http discussion in Go issue 31952, Transport.Proxy must be set before any concurrent use, and it should read the request it receives to make routing decisions. It cannot be changed safely from two goroutines at once. For a scraper, that means one rule: configure the transport once, and put any per-request logic (rotation, per-domain routing) inside the function.

Two practical consequences follow:

  1. Build one client and share it. Creating a new transport per request throws away connection reuse and leaks sockets.
  2. If you replace http.DefaultTransport with your own, set Proxy: http.ProxyFromEnvironment yourself if you still want HTTP_PROXY and HTTPS_PROXY to be honored. A custom transport with no Proxy field goes direct.

Route a Go scraper through an HTTP proxy

The smallest working version uses a fixed proxy URL with credentials:

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func newClient(proxyURL string) (*http.Client, error) {
	u, err := url.Parse(proxyURL)
	if err != nil {
		return nil, err
	}
	tr := &http.Transport{
		Proxy:                 http.ProxyURL(u),
		MaxIdleConnsPerHost:   10,
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ResponseHeaderTimeout: 20 * time.Second,
	}
	return &http.Client{Transport: tr, Timeout: 30 * time.Second}, nil
}

func main() {
	client, err := newClient("http://USER:PASS@PROXY_HOST:PORT")
	if err != nil {
		panic(err)
	}
	resp, err := client.Get("https://example.com/")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)
	fmt.Println(resp.Status, len(body))
}

Replace USER, PASS, PROXY_HOST and PORT with the values from your proxy dashboard. If the password contains characters such as @, : or /, percent-encode it or build the URL with url.UserPassword so parsing does not break.

For HTTPS targets, Go sends a CONNECT request to the proxy and tunnels TLS through it, so the proxy never sees page content. If you are unsure whether you need an HTTP or HTTPS proxy endpoint, our decision rules for HTTP vs HTTPS proxies lay out the difference.

Setting the proxy per request

Because the field is a function, you can route by hostname without touching the transport again:

tr.Proxy = func(r *http.Request) (*url.URL, error) {
	if r.URL.Hostname() == "internal.example.com" {
		return nil, nil // direct
	}
	return proxyU, nil
}

Assign it before the first request and leave it alone.

Using environment variables

If you leave Proxy as http.ProxyFromEnvironment, Go reads HTTP_PROXY, HTTPS_PROXY and NO_PROXY. That is convenient for containers and CI, where you want to change the proxy without rebuilding. For a scraper that needs rotation or per-job credentials, an explicit function is easier to reason about.

Configure a SOCKS5 proxy in Go

The standard library transport speaks HTTP proxies out of the box. For SOCKS5 you use the golang.org/x/net/proxy package. Its SOCKS5 implementation follows RFC 1928 and RFC 1929 and supports optional username/password authentication through proxy.Auth, as shown in the package source.

import (
	"context"
	"net"
	"net/http"
	"time"

	"golang.org/x/net/proxy"
)

func socksClient(addr, user, pass string) (*http.Client, error) {
	dialer, err := proxy.SOCKS5("tcp", addr,
		&proxy.Auth{User: user, Password: pass}, proxy.Direct)
	if err != nil {
		return nil, err
	}
	cd, ok := dialer.(proxy.ContextDialer)
	if !ok {
		return nil, fmt.Errorf("dialer has no DialContext")
	}
	tr := &http.Transport{
		DialContext: func(ctx context.Context, network, address string) (net.Conn, error) {
			return cd.DialContext(ctx, network, address)
		},
		TLSHandshakeTimeout: 10 * time.Second,
	}
	return &http.Client{Transport: tr, Timeout: 30 * time.Second}, nil
}

Notes that save debugging time:

  • Use DialContext, not the older Dial. The context version respects cancellation and deadlines, which matters when a scraper worker is shut down mid-request.
  • Hostname resolution. When you hand the dialer a hostname, the name is passed to the SOCKS5 server to resolve, so lookups happen on the proxy side rather than on your machine. That keeps DNS consistent with the exit location.
  • Auth failures surface as a dial error, not an HTTP status. If you see an error from the SOCKS handshake, check credentials first.

Curious how the handshake negotiates credentials? We wrote up how each SOCKS5 authentication method works.

A shorter route: Go also accepts a socks5:// URL in http.ProxyURL on the standard transport, which is handy when you want one code path for both proxy types. Test it against your provider before relying on it in production.

Rotate proxies without breaking concurrency

Rotation is the part most scrapers get wrong, usually by mutating tr.Proxy while workers are running. The pattern the Go team describes works better: one function, set once, that decides each time it is called.

A round-robin version with an atomic counter:

import (
	"net/http"
	"net/url"
	"sync/atomic"
)

type Rotator struct {
	urls []*url.URL
	n    atomic.Uint64
}

func (r *Rotator) Next(_ *http.Request) (*url.URL, error) {
	i := r.n.Add(1) - 1
	return r.urls[i%uint64(len(r.urls))], nil
}

// usage
// rot := &Rotator{urls: parsedURLs}
// tr := &http.Transport{Proxy: rot.Next}

atomic.Uint64 needs Go 1.19 or later. On older versions, use a mutex around an integer.

Two things to keep in mind with any rotating setup:

  • Connection reuse can hide your rotation. The transport keeps idle connections per proxy, so if your function returns the same URL for a stretch of requests, they share a connection. That is fine and efficient. If you want a new exit IP for every request, either use a rotating endpoint that changes the IP on its side or set DisableKeepAlives: true on the transport.
  • Match session type to the job. Crawling a catalog page by page often works better with a sticky session, where one IP stays pinned for a few minutes. Collecting independent pages (search results across many queries, price checks by city) suits rotation.

Rotating versus sticky on mobile IPs

With a mobile carrier network, many real users share each IP, so sites generally treat these addresses with more tolerance than datacenter ranges. MaskLabs offers both modes: sticky sessions that hold one IP, and rotating sessions that change it. You pick the mode through your proxy credentials and dashboard settings, so your Go code stays the same either way. If you want to see how the same idea looks in other stacks, our walkthroughs for Python scraping with proxies and rotation and Java proxy scraping configs use the same session logic.

Timeouts, retries and failed connections

A scraper that runs for hours will see resets, slow responses and rate-limit replies. Three settings keep that manageable.

Timeouts at every layer. Client.Timeout bounds the whole exchange including reading the body. Add TLSHandshakeTimeout and ResponseHeaderTimeout on the transport so a stuck proxy hop fails quickly. Use http.NewRequestWithContext with a per-job context so you can cancel a whole batch.

Retries with backoff and jitter. Retry on network errors, 429, and 502/503/504. Wait longer after each attempt, add random jitter so parallel workers do not retry in lockstep, honor a Retry-After header when the server sends one, and stop after a fixed number of attempts. Only retry idempotent requests (GET, HEAD) unless you have safeguards.

Timeout and retry safeguards for scraper requests

import (
	"context"
	"math/rand"
	"net/http"
	"strconv"
	"time"
)

func fetch(ctx context.Context, c *http.Client, u string) (*http.Response, error) {
	ctx, cancel := context.WithTimeout(ctx, 60*time.Second) // global deadline
	defer cancel()

	base := 500 * time.Millisecond
	var lastErr error
	for attempt := 0; attempt < 4; attempt++ {
		req, _ := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
		resp, err := c.Do(req)
		if err == nil && resp.StatusCode < 500 && resp.StatusCode != 429 {
			return resp, nil
		}
		wait := base << attempt
		if err != nil {
			lastErr = err
		} else {
			if s, e := strconv.Atoi(resp.Header.Get("Retry-After")); e == nil {
				wait = time.Duration(s) * time.Second
			}
			lastErr = fmt.Errorf("status %d", resp.StatusCode)
			resp.Body.Close()
		}
		wait += time.Duration(rand.Int63n(int64(wait) / 2))
		select {
		case <-time.After(wait):
		case <-ctx.Done():
			return nil, ctx.Err()
		}
	}
	return nil, lastErr
}

The global deadline matters as much as the attempt cap: without it, four attempts with long waits can hold a worker for minutes. Our post on scraper retry logic with three jitter options and one global deadline compares jitter strategies in more depth.

Always close response bodies, including on error paths you retry. Leaked bodies keep connections open and eventually look like proxy failures when they are really local resource exhaustion.

Common proxy errors in Go and how to diagnose them

Symptom in GoLikely causeWhat to check
proxyconnect tcp: dial tcp ... connection refusedWrong host or port, or the proxy is unreachable from your networkConfirm the endpoint and port from your dashboard, and test with curl -x
Proxy Authentication Required / status 407Missing or wrong credentials, or unescaped characters in the URLRebuild the URL with url.UserPassword, and check the plan or account is active
context deadline exceeded or Client.Timeout exceededSlow target, tight timeout, or a stalled connectionRaise ResponseHeaderTimeout, and log which phase is slow
EOF or connection reset by peerTarget closed the connection, or a stale idle connection was reusedRetry with backoff, and lower IdleConnTimeout
x509: certificate signed by unknown authoritySomething intercepting TLS, or a missing CA bundle in a minimal containerInstall CA certificates in the image
Status 429 or 403 on every requestRequest rate or fingerprint is too aggressive for the targetSlow down, spread load across sessions, send realistic headers
SOCKS handshake errorBad credentials or wrong protocol on the portConfirm the port is the SOCKS5 port, not the HTTP one

A fast triage routine:

  1. Run the same request with curl -x http://USER:PASS@HOST:PORT https://example.com. If curl works and Go does not, the issue is in your transport.
  2. Fetch an IP-echo endpoint through the proxy to confirm the exit address and region.
  3. Log the error string, the attempt number and the proxy used for every failure. Patterns show up quickly.
  4. Turn on GODEBUG=http2debug=1 only when you suspect HTTP/2 stream problems.

For Java teams hitting the same 407 symptoms, our guide to fixing 407s and rotating proxies with Apache HttpClient maps almost one-to-one onto these checks.

App proxies versus the Go module proxy

The name causes confusion. The proxy your scraper uses at runtime is configured in your program, through the transport or the environment variables above. The Go module proxy is a separate thing: it is the server the go command contacts to download dependencies, controlled by GOPROXY, with GOPRIVATE marking modules that should skip the proxy and checksum database.

So if go get golang.org/x/net fails behind a corporate network, that is a module-proxy problem and your scraper's proxy settings will not help. Likewise, setting GOPROXY does nothing for HTTP requests your scraper makes. Keep the two configurations separate in your Dockerfile and CI.

Scrape responsibly: policy, rate limits and legal basics

Proxies change where a request appears to come from. They do not change your obligations. Before you point a Go scraper at a site:

  • Read the site's terms and its robots.txt. Respect disallowed paths and any stated crawl rules, and skip areas behind logins unless you have permission.
  • Throttle. Cap concurrency per host, add delays, and back off when you see 429. A scraper that is gentle on the target also fails less often.
  • Collect only what you need. Avoid personal data unless you have a lawful basis, and store it securely.
  • Get legal advice for sensitive projects. If you scrape at scale, resell data or collect personal information, talk to a lawyer who works in data and IP law in the countries involved.

The rules differ by country and by use case, so treat this as a checklist for good practice, not legal advice.

Our perspective on proxies for Go scrapers

Most Go scraping problems that look like proxy problems are really three things: a transport built per request, retries with no ceiling, and an IP type the target already distrusts.

We built MaskLabs around the third one. Datacenter ranges are easy for sites to recognize, so requests from them tend to meet throttling early. Real US carrier IPs look like the mobile traffic the site already serves every day. That is why we focus on genuine mobile IPs, city-level targeting across 46 US cities, and a choice between sticky and rotating sessions.

For Go developers, we keep the integration plain. Standard net/http with HTTP or HTTPS, or x/net/proxy with SOCKS5, and credentials you manage yourself in the dashboard. No sales call is needed to start, and billing is metered per byte, so you see what a crawl costs as it runs. Our goal is that the proxy layer becomes the least interesting part of your scraper.

Jon

Start with real mobile IPs

If you are building a Go scraper for price monitoring, ad verification, geotargeted research or AI data collection, you can plug MaskLabs into the clients above in a few minutes. You get real US carrier IPs, HTTP, HTTPS and SOCKS5 endpoints, API access, and team credential management. A free trial with limited data lets you run your own Go code first.

Masklabs

Plans are billed monthly: Starter is $30 per month, Basic is $100 per month, Advanced is $500 per month and Scale is $1,000 per month, with prepaid data add-ons billed per GB. See the current plans on our pricing page or read more about US mobile proxies on real carrier IPs.

FAQ

Does Go's net/http support SOCKS5 proxies?

The standard transport is built around HTTP proxies, so SOCKS5 is normally handled with golang.org/x/net/proxy. It supports username/password authentication through proxy.Auth. You pass the resulting dialer's DialContext to http.Transport.

Can I change the proxy on an existing http.Client?

Not safely while requests are in flight. Transport.Proxy should be set before concurrent use. The reliable approach is a single proxy function that picks a URL for each request, or a separate client per proxy that you store in a pool.

Why does my Go program ignore HTTP_PROXY?

Usually because a custom http.Transport was created without Proxy: http.ProxyFromEnvironment. A transport with no Proxy function connects directly. Add the field, or set the proxy explicitly in code.

Should I use sticky or rotating sessions for scraping in Go?

Use sticky sessions when a sequence of requests must look like one visitor, such as paging through results or a multi-step flow. Use rotating sessions for many independent requests, such as price checks across cities. With MaskLabs you choose the mode through your credentials, so the Go code does not change.

Sources

Recommended