Try 25 MB of US mobile proxy data free for 30 days.Start free trial
November 16, 202613 min read

Kubernetes Proxy Settings: Route Scraper Pods Through Mobile IPs

Geometric route from pods to mobile IPs

For a scraper, price monitor or ad-verification job running on Kubernetes, "proxy settings" almost always means one thing: the pods need an outbound proxy so target sites see a real mobile carrier IP instead of your cloud provider's datacenter range. You set that with HTTP_PROXY, HTTPS_PROXY and NO_PROXY on the workload, keep the credentials in a Secret, and leave node-level and cluster-level proxy settings alone. We built Masklabs for this setup: real US carrier IPs in 46 cities, reachable over HTTP, HTTPS or SOCKS5, so the only thing your cluster needs is one proxy URL.

This guide covers that workload path in full, then shows where the other "Kubernetes proxies" fit so you don't edit the wrong layer.

TL;DR

  • Kubernetes has several unrelated things called a proxy. Your scraper needs an outbound proxy for pods, set through environment variables, not kube-proxy or kubectl proxy.
  • Set HTTP_PROXY, HTTPS_PROXY and NO_PROXY (upper and lowercase) on the Deployment. Put the proxy URL with credentials in a Secret.
  • NO_PROXY must cover .svc, .cluster.local, and your pod and service ranges, or in-cluster traffic will break.
  • Proxy only the pods that scrape. Node and container runtime proxies are for image pulls and cluster components.
  • Pods read environment variables at start, so roll the Deployment after every change.

Table of Contents

The four things "Kubernetes proxy" can mean

The Kubernetes documentation lists several distinct proxies: kubectl proxy, the API server proxy, kube-proxy, a load balancer in front of the API servers, and cloud load balancers. It also notes that end users mostly deal with the first two, while cluster administrators handle the rest.

None of those carry your scraper's traffic to the internet. Here is how they differ:

LayerWhat it doesDoes it affect scraper egress?
kubectl proxyProxies localhost HTTP to the HTTPS API server and adds authenticationNo
API server proxyLets users reach Node, Pod or Service IPs through the API serverNo
kube-proxyRuns on each node, forwards TCP, UDP and SCTP to Services, does not understand HTTPNo
Load balancersSit in front of the API servers or expose ServicesNo
Workload proxy variablesTell the application's HTTP client to go through an upstream proxyYes

The last row is not a Kubernetes feature at all. It is the standard proxy environment variables that HTTP libraries read, injected into your containers by the manifest. That is why it is easy to get right and also easy to get subtly wrong.

What kubectl proxy is actually for

If you landed here from a kubectl question: kubectl proxy opens a local listener, port 8001 on 127.0.0.1 by default, that forwards to the API server. By default it rejects pod exec and attach paths, and the reference flags --disable-filter as dangerous because of XSRF risk. The task page shows the pattern: start it with kubectl proxy --port=8080, then curl http://localhost:8080/api/. It gets you into the cluster. It does not get your pods out.

Why scraper pods need an upstream proxy

Pods in a managed cluster egress through the cloud provider's NAT or public IPs. Those are datacenter addresses, and sites that serve different content by location, or that rate-limit non-residential traffic, treat them accordingly. A mobile carrier IP looks like an ordinary phone user.

For the people who run these jobs, the practical consequences are:

  • Price comparison across regions: you need the request to originate in the city whose price you want.
  • Ad verification and local search testing: the geography of the IP decides what the page shows.
  • AI training data collection: blocked or throttled requests leave gaps in the dataset.

Masklabs gives you real carrier IPs in 46 US cities, with rotating sessions for broad collection and sticky sessions for flows that must stay on one IP, such as a login followed by a few page loads. Both work from a plain proxy URL, which is what Kubernetes needs.

Step 1: Store the proxy URL in a Secret

A proxy URL with credentials is a secret. Do not paste it into a Deployment manifest that lives in Git. Create a Secret instead. Use the host, port, username and password from your Masklabs dashboard:

apiVersion: v1
kind: Secret
metadata:
  name: mobile-proxy
  namespace: scrapers
type: Opaque
stringData:
  PROXY_URL: "http://USERNAME:PASSWORD@PROXY_HOST:PORT"

Two details trip people up:

  • Special characters in the password must be percent-encoded inside the URL, or the client will parse the URL wrongly and you will see 407 or connection errors that look like a bad password. Our guide to HTTP proxy authentication and 407 errors walks through that failure.
  • The scheme must match the proxy. Use http:// for an HTTP proxy endpoint, and socks5:// only if the client supports it. A mismatch produces errors such as "tls: oversized record received", which the minikube handbook traces to an HTTPS proxy setting that points at an HTTP proxy. If you are unsure which to pick, our HTTP vs HTTPS proxy decision rules cover it.

Secrets are only base64-encoded by default, so limit who can read them with RBAC, and keep one Secret per namespace that needs it. Teams with several projects can give each its own credentials in the Masklabs account, which keeps usage and revocation separate.

Step 2: Define NO_PROXY in a ConfigMap

The non-secret part of the configuration is the bypass list. Get this wrong and the proxy swallows your own cluster traffic.

apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-bypass
  namespace: scrapers
data:
  NO_PROXY: "localhost,127.0.0.1,.svc,.cluster.local,10.0.0.0/8"
  no_proxy: "localhost,127.0.0.1,.svc,.cluster.local,10.0.0.0/8"

What belongs in it:

  • localhost and 127.0.0.1, so sidecars and local ports are never proxied.
  • .svc and .cluster.local, so calls to Services such as your queue, database or results API stay inside the cluster. Kong's operator documentation recommends exactly these so in-cluster communication isn't routed through the proxy. A Broadcom knowledge-base article describes a deployment where the API server became unreachable and object creation failed until these suffixes were added to no_proxy.
  • Your pod and service CIDRs, and any VPC endpoints or internal hostnames your job calls. Replace 10.0.0.0/8 above with the ranges your cluster actually uses.

NO_PROXY is not standardized

The format is a convention, not a spec, and clients differ. SUSE's Rancher documentation warns that the accepted format differs between applications. Microsoft's AKS documentation notes that curl and Python do not support CIDR notation in no_proxy, while Ruby does. Kong's page adds that when both cases are set, the uppercase variable takes precedence.

The practical rules:

  1. Set both NO_PROXY and no_proxy, and give them the same value.
  2. Prefer domain suffixes (.svc, .cluster.local) over CIDRs, because Python's requests and curl will ignore a CIDR.
  3. If a Python scraper must skip an internal IP, list the exact IP or hostname.

Step 3: Inject both into the Deployment

Now wire the Secret and the ConfigMap into the scraper pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: price-scraper
  namespace: scrapers
spec:
  replicas: 3
  selector:
    matchLabels:
      app: price-scraper
  template:
    metadata:
      labels:
        app: price-scraper
    spec:
      containers:
        - name: scraper
          image: registry.example.com/price-scraper:1.4
          envFrom:
            - configMapRef:
                name: proxy-bypass
          env:
            - name: HTTP_PROXY
              valueFrom:
                secretKeyRef:
                  name: mobile-proxy
                  key: PROXY_URL
            - name: HTTPS_PROXY
              valueFrom:
                secretKeyRef:
                  name: mobile-proxy
                  key: PROXY_URL
            - name: http_proxy
              valueFrom:
                secretKeyRef:
                  name: mobile-proxy
                  key: PROXY_URL
            - name: https_proxy
              valueFrom:
                secretKeyRef:
                  name: mobile-proxy
                  key: PROXY_URL

The ConfigMap and Secret must exist in the same namespace as the pods, so a second namespace needs its own copies.

Vendor documentation follows the same pattern. SmoothGlue recommends setting both lowercase and uppercase variables for legacy compatibility and notes that Java applications may need the variables injected in a way their Helm chart documents. That matters because the JVM does not read these variables on its own. If your scraper is on the JVM, set the proxy in code or through system properties, as in our OkHttp proxy guide.

Confirm your HTTP client honors the variables

Not every library reads them. Quick reference for the stacks our users run most:

ClientReads the env vars by default?Where to look
Python requests, curlYesSet NO_PROXY with hostnames, not CIDRs
Python httpxYes, unless trust_env=FalseHTTPX proxy setup
Node.js fetch, node-fetchNot automatically in all versionsNode fetch proxy
AxiosReads them, with its own proxy optionAxios proxy for Node.js
Chrome and Chromium automationNo, needs a flagChrome proxy flags
Java, OkHttpNo, set explicitlyOkHttp proxy

For headless browsers in pods, pass --proxy-server in the container args rather than relying on the environment.

Step 4: Roll the pods and verify the exit IP

Environment variables are read when a container starts. Microsoft's AKS documentation states that after a proxy configuration changes, pods must be rotated for applications to pick up the new values. In your own manifests the same applies:

kubectl -n scrapers rollout restart deployment/price-scraper
kubectl -n scrapers rollout status deployment/price-scraper

Then test from inside a pod, not from your laptop:

kubectl -n scrapers exec deploy/price-scraper -- \
  curl -s https://api.ipify.org

The address returned should be the carrier IP, not your cluster's NAT address. Repeat it a few times: with a rotating session you expect different addresses, with a sticky session you expect the same one for the session's duration. To check a specific city, request that city's endpoint from your dashboard and compare against an IP geolocation lookup.

Carrier IP verification for scraper pods

Troubleshooting by symptom

Match the error to the layer before you change anything. Our proxy error troubleshooting guide goes deeper; these are the Kubernetes-specific cases.

SymptomLikely causeFix
proxyconnect tcp: ... connection refusedHost or port wrong, or proxy unreachableRecheck the URL and egress network policy. The minikube handbook lists this signature
407 Proxy Authentication RequiredBad or unencoded credentialsPercent-encode the password, recreate the Secret, restart pods
In-cluster calls time out or failNO_PROXY missing .svc or .cluster.localAdd the suffixes and the cluster CIDRs
x509: certificate signed by unknown authorityA TLS-rewriting proxy in the pathThe minikube handbook says its certificate must be trusted by the runtime
tls: oversized record receivedHTTPS proxy variable points at an HTTP proxyCheck the scheme
Changes have no effectPods still running old envkubectl rollout restart

Keep node and runtime proxies separate

It is tempting to set one proxy at the node and be done. For scraping, that is the wrong layer. Node-level settings exist so the container runtime can pull images and system components can reach their endpoints. Amazon EKS's hybrid nodes documentation configures containerd through a systemd drop-in with NO_PROXY=localhost, and adds proxy variables to the kube-proxy DaemonSet because it needs to resolve the cluster endpoint.

Three reasons not to push scraper traffic through that layer:

  • It applies to everything. One configuration covers system pods and every workload, so you cannot give one job a Dallas IP and another a Chicago IP.
  • Credentials are limited. Microsoft lists user and password authentication among the unsupported scenarios for the AKS node proxy feature, and also different proxy configurations per node pool.
  • It can break the cluster. A Rancher RKE2 issue shows that CNI plugins launched by containerd inherit its proxy environment. A CONTAINERD_NO_PROXY without the cluster CIDR sent API-server traffic through the proxy and produced certificate errors. The maintainer also explained that CONTAINERD_-prefixed variables replace the defaults rather than merge with them.

Keep the node proxy, if your network requires one, for image pulls and cluster plumbing. Put the Masklabs proxy on the pods that scrape.

Scaling the setup across jobs and regions

Once one Deployment works, the pattern extends cleanly:

  • One Secret per target region. Store a Dallas URL in one, a Chicago URL in another, and point each Deployment at the Secret for its region. Price-comparison jobs then map one workload to one city.
  • Sticky for sessions, rotating for breadth. Crawlers that fetch many independent pages suit rotating IPs. Checkout or login flows suit sticky ones.
  • Watch usage in one place. Masklabs bills metered per byte against your plan's data pool, so a runaway job shows up on the usage dashboard. Add resource limits and a rate limit in the scraper so a retry loop doesn't spend your data on repeated failures.
  • Headers matter too. Proxies and clients add or strip headers. Review HTTP proxy headers if responses differ between your local test and the cluster.

Perspective

Most Kubernetes proxy documentation is written for administrators who need their cluster to survive a corporate firewall. Scraping teams have the opposite problem: the cluster has open internet access, and the quality of the exit IP decides whether the data is usable.

We think the cleanest design is the simplest one. Treat the proxy as an application dependency, a single URL in a Secret, and keep it out of node and runtime configuration. That way, platform engineers change cluster networking without touching scrapers, and data teams change cities or session types without opening a ticket.

This is also why we built Masklabs around real US carrier IPs with city targeting, sticky and rotating sessions, and standard HTTP, HTTPS and SOCKS5 endpoints. Credential management is self-serve, so a team can issue a credential per project and wire it into a Secret in minutes.

Jon

Try it on one Deployment

The fastest way to see the difference is to point one scraper Deployment at a mobile carrier IP and compare success rates against your current datacenter egress. Masklabs offers a free trial with limited data, so you can do that before committing. Paid plans are billed monthly: Starter is $30 per month, Basic $100, Advanced $500 and Scale $1,000, with prepaid data add-ons billed per GB. There are no sales calls or long-term commitments.

Masklabs

Start with the US mobile proxies on real carrier IPs, copy the endpoint from your dashboard into the Secret above, and roll out the Deployment.

FAQ

How do I set a proxy for a single pod in Kubernetes?

Add HTTP_PROXY, HTTPS_PROXY and NO_PROXY to the container's env or envFrom in the pod or Deployment spec, and source the proxy URL from a Secret. Include the lowercase variants as well. Restart the pods afterward, because containers read environment variables only at start.

Does kube-proxy have anything to do with HTTP proxy settings?

No. According to the Kubernetes documentation, kube-proxy runs on each node, forwards TCP, UDP and SCTP traffic to Services, and does not understand HTTP. It is not where you configure an outbound proxy for your applications.

What should NO_PROXY contain in a cluster?

At minimum localhost, 127.0.0.1, .svc, .cluster.local and your pod and service ranges. Prefer domain suffixes to CIDRs where possible, since curl and Python don't support CIDR in no_proxy. Set both upper and lowercase versions with the same value.

Can I use a SOCKS5 proxy from Kubernetes pods?

Yes, if your client supports it. Many HTTP libraries need an extra package or an explicit setting for SOCKS5, and the standard HTTP_PROXY variables are most reliably used with http:// URLs. Masklabs supports HTTP, HTTPS and SOCKS5, so you can choose the protocol your stack handles best.

Why do my pods still use the old proxy after I changed the Secret?

Environment variables are fixed when a container starts, and Secret changes are not re-injected into running env vars. Run kubectl rollout restart on the Deployment. AKS's documentation likewise says pods must be rotated to pick up proxy changes.

Sources

Recommended