Kubernetes Proxy Settings: Route Scraper Pods Through Mobile IPs

For a scraper, price monitor or ad-verification job running on Kubernetes, "proxy settings" almost always means one thing: the pods need an outbound proxy so target sites see a real mobile carrier IP instead of your cloud provider's datacenter range. You set that with HTTP_PROXY, HTTPS_PROXY and NO_PROXY on the workload, keep the credentials in a Secret, and leave node-level and cluster-level proxy settings alone. We built Masklabs for this setup: real US carrier IPs in 46 cities, reachable over HTTP, HTTPS or SOCKS5, so the only thing your cluster needs is one proxy URL.
This guide covers that workload path in full, then shows where the other "Kubernetes proxies" fit so you don't edit the wrong layer.
TL;DR
- Kubernetes has several unrelated things called a proxy. Your scraper needs an outbound proxy for pods, set through environment variables, not
kube-proxyorkubectl proxy. - Set
HTTP_PROXY,HTTPS_PROXYandNO_PROXY(upper and lowercase) on the Deployment. Put the proxy URL with credentials in a Secret. NO_PROXYmust cover.svc,.cluster.local, and your pod and service ranges, or in-cluster traffic will break.- Proxy only the pods that scrape. Node and container runtime proxies are for image pulls and cluster components.
- Pods read environment variables at start, so roll the Deployment after every change.
Table of Contents
- The four things "Kubernetes proxy" can mean
- Why scraper pods need an upstream proxy
- Step 1: Store the proxy URL in a Secret
- Step 2: Define NO_PROXY in a ConfigMap
- Step 3: Inject both into the Deployment
- Step 4: Roll the pods and verify the exit IP
- Troubleshooting by symptom
- Keep node and runtime proxies separate
- Scaling the setup across jobs and regions
- Perspective
- Try it on one Deployment
- FAQ
- Sources
The four things "Kubernetes proxy" can mean
The Kubernetes documentation lists several distinct proxies: kubectl proxy, the API server proxy, kube-proxy, a load balancer in front of the API servers, and cloud load balancers. It also notes that end users mostly deal with the first two, while cluster administrators handle the rest.
None of those carry your scraper's traffic to the internet. Here is how they differ:
| Layer | What it does | Does it affect scraper egress? |
|---|---|---|
kubectl proxy | Proxies localhost HTTP to the HTTPS API server and adds authentication | No |
| API server proxy | Lets users reach Node, Pod or Service IPs through the API server | No |
kube-proxy | Runs on each node, forwards TCP, UDP and SCTP to Services, does not understand HTTP | No |
| Load balancers | Sit in front of the API servers or expose Services | No |
| Workload proxy variables | Tell the application's HTTP client to go through an upstream proxy | Yes |
The last row is not a Kubernetes feature at all. It is the standard proxy environment variables that HTTP libraries read, injected into your containers by the manifest. That is why it is easy to get right and also easy to get subtly wrong.
What kubectl proxy is actually for
If you landed here from a kubectl question: kubectl proxy opens a local listener, port 8001 on 127.0.0.1 by default, that forwards to the API server. By default it rejects pod exec and attach paths, and the reference flags --disable-filter as dangerous because of XSRF risk. The task page shows the pattern: start it with kubectl proxy --port=8080, then curl http://localhost:8080/api/. It gets you into the cluster. It does not get your pods out.
Why scraper pods need an upstream proxy
Pods in a managed cluster egress through the cloud provider's NAT or public IPs. Those are datacenter addresses, and sites that serve different content by location, or that rate-limit non-residential traffic, treat them accordingly. A mobile carrier IP looks like an ordinary phone user.
For the people who run these jobs, the practical consequences are:
- Price comparison across regions: you need the request to originate in the city whose price you want.
- Ad verification and local search testing: the geography of the IP decides what the page shows.
- AI training data collection: blocked or throttled requests leave gaps in the dataset.
Masklabs gives you real carrier IPs in 46 US cities, with rotating sessions for broad collection and sticky sessions for flows that must stay on one IP, such as a login followed by a few page loads. Both work from a plain proxy URL, which is what Kubernetes needs.
Step 1: Store the proxy URL in a Secret
A proxy URL with credentials is a secret. Do not paste it into a Deployment manifest that lives in Git. Create a Secret instead. Use the host, port, username and password from your Masklabs dashboard:
apiVersion: v1
kind: Secret
metadata:
name: mobile-proxy
namespace: scrapers
type: Opaque
stringData:
PROXY_URL: "http://USERNAME:PASSWORD@PROXY_HOST:PORT"
Two details trip people up:
- Special characters in the password must be percent-encoded inside the URL, or the client will parse the URL wrongly and you will see 407 or connection errors that look like a bad password. Our guide to HTTP proxy authentication and 407 errors walks through that failure.
- The scheme must match the proxy. Use
http://for an HTTP proxy endpoint, andsocks5://only if the client supports it. A mismatch produces errors such as "tls: oversized record received", which the minikube handbook traces to an HTTPS proxy setting that points at an HTTP proxy. If you are unsure which to pick, our HTTP vs HTTPS proxy decision rules cover it.
Secrets are only base64-encoded by default, so limit who can read them with RBAC, and keep one Secret per namespace that needs it. Teams with several projects can give each its own credentials in the Masklabs account, which keeps usage and revocation separate.
Step 2: Define NO_PROXY in a ConfigMap
The non-secret part of the configuration is the bypass list. Get this wrong and the proxy swallows your own cluster traffic.
apiVersion: v1
kind: ConfigMap
metadata:
name: proxy-bypass
namespace: scrapers
data:
NO_PROXY: "localhost,127.0.0.1,.svc,.cluster.local,10.0.0.0/8"
no_proxy: "localhost,127.0.0.1,.svc,.cluster.local,10.0.0.0/8"
What belongs in it:
localhostand127.0.0.1, so sidecars and local ports are never proxied..svcand.cluster.local, so calls to Services such as your queue, database or results API stay inside the cluster. Kong's operator documentation recommends exactly these so in-cluster communication isn't routed through the proxy. A Broadcom knowledge-base article describes a deployment where the API server became unreachable and object creation failed until these suffixes were added tono_proxy.- Your pod and service CIDRs, and any VPC endpoints or internal hostnames your job calls. Replace
10.0.0.0/8above with the ranges your cluster actually uses.
NO_PROXY is not standardized
The format is a convention, not a spec, and clients differ. SUSE's Rancher documentation warns that the accepted format differs between applications. Microsoft's AKS documentation notes that curl and Python do not support CIDR notation in no_proxy, while Ruby does. Kong's page adds that when both cases are set, the uppercase variable takes precedence.
The practical rules:
- Set both
NO_PROXYandno_proxy, and give them the same value. - Prefer domain suffixes (
.svc,.cluster.local) over CIDRs, because Python'srequestsand curl will ignore a CIDR. - If a Python scraper must skip an internal IP, list the exact IP or hostname.
Step 3: Inject both into the Deployment
Now wire the Secret and the ConfigMap into the scraper pods:
apiVersion: apps/v1
kind: Deployment
metadata:
name: price-scraper
namespace: scrapers
spec:
replicas: 3
selector:
matchLabels:
app: price-scraper
template:
metadata:
labels:
app: price-scraper
spec:
containers:
- name: scraper
image: registry.example.com/price-scraper:1.4
envFrom:
- configMapRef:
name: proxy-bypass
env:
- name: HTTP_PROXY
valueFrom:
secretKeyRef:
name: mobile-proxy
key: PROXY_URL
- name: HTTPS_PROXY
valueFrom:
secretKeyRef:
name: mobile-proxy
key: PROXY_URL
- name: http_proxy
valueFrom:
secretKeyRef:
name: mobile-proxy
key: PROXY_URL
- name: https_proxy
valueFrom:
secretKeyRef:
name: mobile-proxy
key: PROXY_URL
The ConfigMap and Secret must exist in the same namespace as the pods, so a second namespace needs its own copies.
Vendor documentation follows the same pattern. SmoothGlue recommends setting both lowercase and uppercase variables for legacy compatibility and notes that Java applications may need the variables injected in a way their Helm chart documents. That matters because the JVM does not read these variables on its own. If your scraper is on the JVM, set the proxy in code or through system properties, as in our OkHttp proxy guide.
Confirm your HTTP client honors the variables
Not every library reads them. Quick reference for the stacks our users run most:
| Client | Reads the env vars by default? | Where to look |
|---|---|---|
Python requests, curl | Yes | Set NO_PROXY with hostnames, not CIDRs |
Python httpx | Yes, unless trust_env=False | HTTPX proxy setup |
Node.js fetch, node-fetch | Not automatically in all versions | Node fetch proxy |
| Axios | Reads them, with its own proxy option | Axios proxy for Node.js |
| Chrome and Chromium automation | No, needs a flag | Chrome proxy flags |
| Java, OkHttp | No, set explicitly | OkHttp proxy |
For headless browsers in pods, pass --proxy-server in the container args rather than relying on the environment.
Step 4: Roll the pods and verify the exit IP
Environment variables are read when a container starts. Microsoft's AKS documentation states that after a proxy configuration changes, pods must be rotated for applications to pick up the new values. In your own manifests the same applies:
kubectl -n scrapers rollout restart deployment/price-scraper
kubectl -n scrapers rollout status deployment/price-scraper
Then test from inside a pod, not from your laptop:
kubectl -n scrapers exec deploy/price-scraper -- \
curl -s https://api.ipify.org
The address returned should be the carrier IP, not your cluster's NAT address. Repeat it a few times: with a rotating session you expect different addresses, with a sticky session you expect the same one for the session's duration. To check a specific city, request that city's endpoint from your dashboard and compare against an IP geolocation lookup.

Troubleshooting by symptom
Match the error to the layer before you change anything. Our proxy error troubleshooting guide goes deeper; these are the Kubernetes-specific cases.
| Symptom | Likely cause | Fix |
|---|---|---|
proxyconnect tcp: ... connection refused | Host or port wrong, or proxy unreachable | Recheck the URL and egress network policy. The minikube handbook lists this signature |
407 Proxy Authentication Required | Bad or unencoded credentials | Percent-encode the password, recreate the Secret, restart pods |
| In-cluster calls time out or fail | NO_PROXY missing .svc or .cluster.local | Add the suffixes and the cluster CIDRs |
x509: certificate signed by unknown authority | A TLS-rewriting proxy in the path | The minikube handbook says its certificate must be trusted by the runtime |
tls: oversized record received | HTTPS proxy variable points at an HTTP proxy | Check the scheme |
| Changes have no effect | Pods still running old env | kubectl rollout restart |
Keep node and runtime proxies separate
It is tempting to set one proxy at the node and be done. For scraping, that is the wrong layer. Node-level settings exist so the container runtime can pull images and system components can reach their endpoints. Amazon EKS's hybrid nodes documentation configures containerd through a systemd drop-in with NO_PROXY=localhost, and adds proxy variables to the kube-proxy DaemonSet because it needs to resolve the cluster endpoint.
Three reasons not to push scraper traffic through that layer:
- It applies to everything. One configuration covers system pods and every workload, so you cannot give one job a Dallas IP and another a Chicago IP.
- Credentials are limited. Microsoft lists user and password authentication among the unsupported scenarios for the AKS node proxy feature, and also different proxy configurations per node pool.
- It can break the cluster. A Rancher RKE2 issue shows that CNI plugins launched by containerd inherit its proxy environment. A
CONTAINERD_NO_PROXYwithout the cluster CIDR sent API-server traffic through the proxy and produced certificate errors. The maintainer also explained thatCONTAINERD_-prefixed variables replace the defaults rather than merge with them.
Keep the node proxy, if your network requires one, for image pulls and cluster plumbing. Put the Masklabs proxy on the pods that scrape.
Scaling the setup across jobs and regions
Once one Deployment works, the pattern extends cleanly:
- One Secret per target region. Store a Dallas URL in one, a Chicago URL in another, and point each Deployment at the Secret for its region. Price-comparison jobs then map one workload to one city.
- Sticky for sessions, rotating for breadth. Crawlers that fetch many independent pages suit rotating IPs. Checkout or login flows suit sticky ones.
- Watch usage in one place. Masklabs bills metered per byte against your plan's data pool, so a runaway job shows up on the usage dashboard. Add resource limits and a rate limit in the scraper so a retry loop doesn't spend your data on repeated failures.
- Headers matter too. Proxies and clients add or strip headers. Review HTTP proxy headers if responses differ between your local test and the cluster.
Perspective
Most Kubernetes proxy documentation is written for administrators who need their cluster to survive a corporate firewall. Scraping teams have the opposite problem: the cluster has open internet access, and the quality of the exit IP decides whether the data is usable.
We think the cleanest design is the simplest one. Treat the proxy as an application dependency, a single URL in a Secret, and keep it out of node and runtime configuration. That way, platform engineers change cluster networking without touching scrapers, and data teams change cities or session types without opening a ticket.
This is also why we built Masklabs around real US carrier IPs with city targeting, sticky and rotating sessions, and standard HTTP, HTTPS and SOCKS5 endpoints. Credential management is self-serve, so a team can issue a credential per project and wire it into a Secret in minutes.
Jon
Try it on one Deployment
The fastest way to see the difference is to point one scraper Deployment at a mobile carrier IP and compare success rates against your current datacenter egress. Masklabs offers a free trial with limited data, so you can do that before committing. Paid plans are billed monthly: Starter is $30 per month, Basic $100, Advanced $500 and Scale $1,000, with prepaid data add-ons billed per GB. There are no sales calls or long-term commitments.

Start with the US mobile proxies on real carrier IPs, copy the endpoint from your dashboard into the Secret above, and roll out the Deployment.
FAQ
How do I set a proxy for a single pod in Kubernetes?
Add HTTP_PROXY, HTTPS_PROXY and NO_PROXY to the container's env or envFrom in the pod or Deployment spec, and source the proxy URL from a Secret. Include the lowercase variants as well. Restart the pods afterward, because containers read environment variables only at start.
Does kube-proxy have anything to do with HTTP proxy settings?
No. According to the Kubernetes documentation, kube-proxy runs on each node, forwards TCP, UDP and SCTP traffic to Services, and does not understand HTTP. It is not where you configure an outbound proxy for your applications.
What should NO_PROXY contain in a cluster?
At minimum localhost, 127.0.0.1, .svc, .cluster.local and your pod and service ranges. Prefer domain suffixes to CIDRs where possible, since curl and Python don't support CIDR in no_proxy. Set both upper and lowercase versions with the same value.
Can I use a SOCKS5 proxy from Kubernetes pods?
Yes, if your client supports it. Many HTTP libraries need an extra package or an explicit setting for SOCKS5, and the standard HTTP_PROXY variables are most reliably used with http:// URLs. Masklabs supports HTTP, HTTPS and SOCKS5, so you can choose the protocol your stack handles best.
Why do my pods still use the old proxy after I changed the Secret?
Environment variables are fixed when a container starts, and Secret changes are not re-injected into running env vars. Run kubectl rollout restart on the Deployment. AKS's documentation likewise says pods must be rotated to pick up proxy changes.
Sources
- Proxies in Kubernetes
- kubectl proxy reference
- Use an HTTP Proxy to Access the Kubernetes API
- Proxies and VPNs, minikube handbook
- HTTP proxy configuration, Kong Operator
- Broadcom knowledge base: proxy configured on Kubernetes nodes
- Configure AKS nodes with an HTTP proxy, Microsoft Learn
- SUSE Rancher Manager: HTTP proxy Kubernetes install
- SmoothGlue: Send Traffic through a Web Proxy
- Configure proxy for hybrid nodes, Amazon EKS
- RKE2 issue 4924: kubelet affected by CONTAINERD_* environment