Proxy Rate Limits: What They Are and How to Handle Them

A proxy rate limit is a cap on how many requests an identity can send in a window of time. The identity is usually an IP address, but it can also be an account, an API key or a session. The limit can come from the site you are collecting from, from an API behind it, or from your proxy provider. Staying under it takes three things: knowing which layer is pushing back, reading the signals it sends, and pacing your workload so you rarely hit the cap. Mobile carrier IPs, which we provide at Masklabs, give you a trusted starting point. Pacing and retry discipline do the rest.
This guide is for developers, data scientists, marketers and researchers who run scrapers, price monitors or geo-targeted checks. It explains the mechanics once, then shows how to build a collection job that stays inside the limits.
TL;DR
- Rate limits come from three places: the destination site or API, the proxy service, and your own code. The error response usually tells you which one.
- HTTP 429 means "too many requests." If a
Retry-Afterheader is present, wait that long before the next attempt. - Retrying only helps when the limit is a time window. If a quota or spend limit is used up, waiting changes nothing.
- Use exponential backoff with jitter, cap the attempts, and read rate-limit headers so you slow down before the next 429.
- Size concurrency and request pacing separately, and spread geo-targeted work across sessions and cities that match the data you need.
Table of Contents
- Where proxy rate limits come from
- What it means when your IP is rate limited
- Reading rate-limit headers before you hit the wall
- Fixed windows, sliding windows and token buckets
- What a reasonable request rate looks like
- Handling a 429 correctly
- Reduce the load before you tune the limiter
- Setting limits on your own side
- How the IP itself affects the limits you hit
- Monitor so you see limits early
- Where Masklabs fits
- Our perspective
- FAQ
- Sources
Where proxy rate limits come from
The phrase "proxy rate limits" covers three separate controls, and mixing them up is the most common reason debugging goes in circles.
The destination. A website or API protects itself by limiting how often one client can ask for things. According to MDN's reference on HTTP 429, limits may apply server-wide or per resource and are commonly keyed to the client IP, though authenticated requests or cookies can make the limit specific to a user or application. In practice, a logged-in session can hit a ceiling that the IP alone would not, and a shared IP can be throttled because of what other users did.
The proxy service. Providers cap things too: concurrent connections, bandwidth, requests per key or the data pool in your plan. These limits protect shared infrastructure and are defined by your plan, not by the target site.

Your own code. The healthiest limiter is the one you run yourself. Microsoft recommends a local rate limiter, such as a token bucket or leaky bucket, that queues or discards excess requests before they ever reach the API.
When a request fails, ask first which of the three produced the failure. A 429 with a body from the target site is the destination. A rejection at the proxy gateway before the request leaves is the provider. A failure that never leaves your process is your own limiter doing its job.
What it means when your IP is rate limited
When an IP is rate limited, the server has counted more requests from it than the allowance for the window, and it temporarily refuses more. Paddle's API documentation describes a 429 as the API rate limiting requests from your IP after the allowed rate was exceeded. It also notes that an authorized third-party integration can add to the request volume, which matters if several tools share one egress IP.
Three signals tell you what happened:
- The status code. HTTP 429 means the client sent too many requests in a given time and the server is asking it to slow down, as MDN defines it.
- The error body. Many APIs explain which limit was hit. Log it, do not discard it.
- The
Retry-Afterheader. It states how long to wait. Paddle's guidance is to check Retry-After and wait the specified number of seconds.
Status codes are not fully uniform. Microsoft's documentation warns that GitHub may use 403 or 429, while OpenAI, Anthropic and Microsoft Graph use 429, each with service-specific headers and meanings. Build your handler around the provider's documentation, not an assumption that every site speaks the same dialect.
Reading rate-limit headers before you hit the wall
Some APIs tell you how much room is left on every response, not just after a failure. Microsoft Learn explains that RateLimit-Remaining communicates the requests left in a window and RateLimit-Reset communicates when it resets, and that header names and formats differ by API.
Treat those headers as feedback for pacing, not just something to inspect after an error. A simple pattern:
- Read the remaining count on each response.
- When it drops below a threshold, such as 10% of the window, add delay between requests.
- When it reaches zero, sleep until the reset time instead of sending requests you know will fail.
Some providers also tell you which limit you triggered. APIVoid's documentation says its X-RateLimit-Scope header identifies the triggered limit, such as requests or concurrency. That distinction changes the fix: too many requests per second calls for slower pacing, while too many in flight at once calls for fewer workers.
Fixed windows, sliding windows and token buckets
Knowing how the other side counts helps you predict when a limit will bite. Three methods cover most of what you will meet.
| Method | How it counts | What it means for your scraper |
|---|---|---|
| Fixed window | Counts requests in fixed intervals and resets at the boundary | Bursts right around the boundary can slip through, and a burst right before the reset can still cause trouble |
| Sliding window | Weights recent activity across adjoining intervals | Smoother enforcement, so a burst after a quiet period is judged against what came just before it |
| Token bucket | Refills tokens over time and spends one per request | Short bursts are allowed up to the bucket size, then you are held to the refill rate |
These descriptions follow Upstash's rate-limit algorithm documentation. Symfony's docs explain why fixed-window counters can allow bursts at interval boundaries and why sliding windows were designed to smooth that edge effect.
The practical lesson is to avoid designing around the edge of a window. A job that sends a large batch the instant a window resets works against a fixed window and gets caught by a sliding one. A steady rate with small, jittered gaps behaves well against all three.
What a reasonable request rate looks like
There is no universal number. Microsoft's guidance is that limits depend on the API provider or service plan, so the documentation and the response headers are your source of truth.
Published figures show how different the scales can be. PayPal's US help page says its sandbox temporarily blocks an IP for five minutes after more than 50 cumulative requests across endpoints within one minute, and advises staying below 50 per minute. That is a single provider's rule for a single environment, but it illustrates the typical shape: a count, a window and a penalty period.
For sites that publish nothing, start conservatively and measure:
- Begin at a low rate per IP, for example one request every few seconds.
- Raise it in small steps while watching the share of 429 and 403 responses.
- Hold the rate one step below where errors begin to climb.
- Re-check periodically, because limits change with load and with the target's own updates.
Because success rate can hide problems, it is worth tracking more than a single percentage. Our piece on why 95% to 99% proxy success rates can mislead covers what to measure beyond the headline number.
Handling a 429 correctly
The goal is to respond to the server's request to slow down, and to avoid making things worse by retrying at full speed. Microsoft's guidance gives a clear order of operations.
Step 1: decide whether the 429 is retryable. Microsoft advises checking whether the response reflects exhausted quota, credits or spend limits, because retrying will not fix those. If the body says the quota is gone, stop the job and fix the plan or budget.
Step 2: honor Retry-After or the reset header. If the server says to wait 30 seconds, wait 30 seconds. This is the most accurate information you will get.
Step 3: otherwise back off exponentially with jitter. Double the wait after each failure and add a random offset so that many workers do not retry in lockstep. Stop after a few attempts instead of looping forever.
Step 4: slow down proactively. Use the remaining-allowance headers to reduce speed before the next 429 arrives.
A minimal Python pattern for the retry decision:
import random, time
def wait_seconds(resp, attempt):
ra = resp.headers.get("Retry-After")
if ra and ra.isdigit():
return int(ra)
return min(60, 2 ** attempt) + random.uniform(0, 1)
Call it after each 429, sleep for the result, and give up once attempt passes your cap. If you want to go deeper on jitter strategies and a global deadline per job, we wrote up three jitter options and one global deadline. For jobs that touch many targets, four failover strategies that stop wasted retries show how to route around a throttled path instead of hammering it.
Reduce the load before you tune the limiter
The cheapest request is the one you do not send. Microsoft recommends requesting only the data you need and using batch requests where supported. Paddle suggests reducing calls with include parameters or short-term caching.
For scraping and price monitoring, that translates to:
- Cache what does not change. Category pages, product metadata and images rarely need a fresh fetch every cycle.
- Skip unneeded assets. Do not load images, fonts or trackers if you only need the HTML or JSON.
- Deduplicate URLs before queuing them.
- Schedule by volatility. Check fast-moving prices more often than static listings.
These steps also cut data use. Because our billing is metered per byte, trimming payloads lowers your bill as well as your request count; our guide to cutting proxy bandwidth by 50 to 90% maps each fix to what you are billed for.
Setting limits on your own side
If you run an internal collection platform, set limits on it too, so one runaway job cannot burn through the budget or trigger blocks for everyone. ScrapeNest's documentation suggests organization-wide concurrency and credit limits, plus per-key requests per minute, daily quota and concurrent-job limits. It also describes an observe mode that meters and logs limits before you enforce them, and enforced limits return 429.
The observe-first approach is worth copying: run new limits in log-only mode for a week, check which jobs would have been blocked, then turn enforcement on.
Concurrency and pacing are different dials. APIVoid distinguishes concurrent requests from requests per second, and says its X-Concurrency headers describe active requests, while X-RateLimit headers describe requests started per window. Size worker count and request pacing separately. Ten workers each sleeping between requests can stay well inside a per-second limit while still breaching a concurrency cap if responses are slow.
How the IP itself affects the limits you hit
Because many limits are keyed to the client IP, the type of IP matters. Traffic from a datacenter range is easy for a site to recognize and often gets tighter treatment. Our approach at Masklabs is to use real US mobile carrier IPs rather than datacenter proxies, so that data requests look like ordinary mobile traffic and are less likely to be blocked or throttled.
That does not replace pacing. It gives your well-paced requests a better starting position. Two session modes then let you match the IP behavior to the task:
- Sticky sessions keep one IP for a stretch of time. Use them for logins, carts, paginated flows and anything where the site expects a consistent client.
- Rotating sessions change the IP between requests. Use them for independent lookups, such as checking many product pages, so no single IP carries the whole count.
For geo-targeted work, city-level choice matters just as much. A price check or local search test should come from the city whose result you need. We cover 46 US cities, so a regional price comparison can spread requests across several locations rather than concentrating them on one.
For a longer look at how limits shape large jobs, see how rate limiting affects large-scale web automation. If proxy errors that look like limits are actually authentication failures, our guide to fixing 407 proxy authentication errors separates the two.
Monitor so you see limits early
Limits rarely appear without warning. Watch these signals per target:
- Share of 429 responses, and of 403 responses where the site uses those for throttling.
- Median and 95th percentile latency, since slowdowns often precede hard blocks.
- Requests completed per minute against the rate you intended.
- Data used per job, to catch a job that suddenly fetches more than expected.
Sample more than once when you test an endpoint, because a single check can look healthy by luck. Our note on running three to five samples for proxy uptime monitoring explains the approach.
Where Masklabs fits
If your scraper, price monitor or ad-verification job runs on US mobile traffic, you can start without a sales call or a long-term commitment. Plans are tiered by monthly data pool: Starter is $30.00 per month, Basic is $100.00 per month, Advanced is $500.00 per month and Scale is $1,000.00 per month. Prepaid data add-ons are billed per GB, and a free trial with limited data lets you measure your own request rates first. You get API access, HTTP, HTTPS and SOCKS5 support, team credential management and a usage dashboard, so you can see consumption per job. See the plans and pricing or the US mobile proxy overview.

Our perspective
Most rate-limit trouble we see described in engineering discussions is a pacing problem wearing a proxy costume. Teams fix it by buying more IPs, when the faster win is a steady request rate, honest backoff and a clear view of what each job consumes.
We built Masklabs around that idea. Real US mobile carrier IPs give requests an authentic profile, city-level targeting lets you collect the data a region actually shows, and sticky or rotating sessions let you choose the right identity for each task. Metered per-byte billing and a usage dashboard keep the cost of a well-paced job visible.
Our advice is to design the limiter first. Set a rate per target, cap concurrency, honor Retry-After, and let the IP layer support a job that already behaves well.
Jon
FAQ
What is a reasonable API rate limit?
It depends on the provider and your plan, so the documentation and response headers are the reference. Published examples vary widely: one payments sandbox, for instance, blocks an IP for five minutes above 50 requests per minute. Start low, raise the rate in steps and stay one step below where errors begin.
What does it mean if my IP address has been rate limited?
The server counted more requests from your IP than the allowance for that window and is temporarily refusing more. You will usually see HTTP 429, often with a Retry-After header. Wait the stated time, then resume at a lower rate.
Should I retry every 429?
No. First check whether the response says a quota, credit balance or spend limit is exhausted, because waiting will not fix that. If it is a time window, honor Retry-After, or use exponential backoff with jitter and a small attempt cap.
Do rotating IPs remove the need to pace requests?
No. Rotation spreads requests across addresses, but limits can also be keyed to accounts, cookies or API keys, so one identity can still be capped. Combine rotation or sticky sessions with a per-target rate and concurrency limit.
Sources
- 429 Too Many Requests, MDN
- Too many requests, Paddle API errors
- How to Handle Rate Limiting, Microsoft Learn
- 429 Too Many Requests: what it means and how to handle it, Microsoft Learn
- Rate Limits & Concurrency, APIVoid Docs
- Why did I get HTTP error code 429? PayPal US
- Ratelimit TypeScript SDK Algorithms, Upstash Documentation
- Rate Limiter, Symfony 7.3 Docs
- Rate Limits & Quotas, ScrapeNest Docs