Scraper Retry Logic: Three Jitter Options, One Global Deadline

Geometric retry attempts within a deadline boundary

The best approach to scraper retry logic is capped exponential backoff with full jitter, applied only to transient, idempotent failures, and bounded by a global deadline. Cap attempts at a sensible maximum, watch retry success rates in real time, and never let a retry loop run longer than the operation can tolerate. Everything else in this guide builds on that one rule.

TL;DR:

Retry network timeouts, resets, 429s, and selected 5xx responses; fail fast on malformed requests, authentication failures, 404s, and parser validation errors.

Start with three total attempts, set explicit connection and read timeouts, and clamp server suggested waits to the remaining operation deadline.

For hard blocks such as 403s or CAPTCHA pages, rotate the proxy before retrying; throttling such as 429s calls for backoff on the same IP.

Track retry success by endpoint and proxy pool, separate hard block alerts from throttling, and test policy changes under simulated failures before deployment.

Distinguishing transient vs permanent failures
Backoff strategies: exponential backoff, caps, and jitter variants
Timeouts and the global deadline for an operation
Idempotency as a precondition for retries
Which errors to retry and which to fail fast
Implementation patterns: Scrapy, Requests, and SDKs
Retry budgets and circuit breakers for fleet-wide safety
Recommended defaults and a quick cheat sheet
Handling proxy-specific failures like IP bans and throttling
Retry logic and rotating vs sticky proxy sessions
Retry logic for geolocation-specific scraping
Monitoring retry patterns across proxy environments
What running scraper fleets taught me about retries
A different lever: fewer blocks means fewer retries
FAQ
Sources

Distinguishing transient vs permanent failures

Not every failed request deserves a second attempt. Transient failures include network timeouts, connection resets, intermittent 5xx responses, and throttling signals like a 429 status code: these may resolve on their own. Permanent failures include most 4xx responses, like malformed requests or authentication errors, where retrying wastes a request cycle without changing the outcome.

Transient signals: connection reset, read timeout, 502/503/504, or a 429 with a Retry-After header.
Permanent signals: 400 (bad request), 401/403 (auth or access denied), 404 (not found), or any validation error from your own parsing layer.
Test your classification by logging every retry outcome and checking whether retried permanent errors ever succeed; if they never do, exclude them from your retry list.

Getting this classification wrong can waste compute in scraper fleets. A scraper that repeatedly retries a 403 before giving up is not being cautious, it is burning request budget on errors that may not change.

Backoff strategies: exponential backoff, caps, and jitter variants

Exponential backoff doubles the wait time after each failed attempt, but without a cap, a handful of retries can push wait times into the minutes. A cap keeps worst-case delay bounded and predictable, which matters when you are running many concurrent scraper tasks.

Three jitter variants are common: Full Jitter, Equal Jitter, and Decorrelated Jitter. AWS's architecture guidance recommends Full Jitter for large fleets because it spreads retry attempts across the widest possible window, which does the most to prevent a thundering herd of retries hitting a recovering server at the same moment. The formula is straightforward: random(0, min(cap, base_delay * 2^retry)).

Jitter type	Formula	Best for
Full Jitter	`random(0, min(cap, base * 2^n))`	Large fleets, maximum spread
Equal Jitter	`half + random(0, half)`	Moderate spread, more predictable floor
Decorrelated Jitter	`random(base, prev_delay * 3)`	Avoiding clustering across retries

A working pseudocode pattern for Full Jitter:

delay = random(0, min(cap, base_delay * (2 ** retry_count)))
sleep(delay)

Choose a base_delay and a cap based on your error types and workload. Beyond a suitable waiting period, additional waiting rarely improves success odds and just extends the total job runtime.

Timeouts and the global deadline for an operation

Per-request timeouts and a total operation deadline solve two different problems. A connection timeout (commonly around 10 seconds) and a read timeout (commonly around 30 seconds) stop a single hung request from blocking a worker, as Requests' advanced usage docs describe. A global deadline stops the retry loop itself from running indefinitely across multiple attempts.

Set explicit per-request timeouts so no single call can hang the worker thread.
Track elapsed time from the first attempt and stop retrying once a global deadline (say 60 seconds for a single page fetch) is reached, regardless of attempts remaining.
Clamp long Retry-After values to whatever time remains in the global deadline rather than sleeping past it.

Without a cumulative cap, a single request can effectively hang forever, consuming a worker slot while the rest of the queue backs up.

Idempotency as a precondition for retries

Retrying only makes sense when repeating the operation produces the same result as running it once. For most scraping GET requests this holds naturally, but any workflow that writes data, triggers a webhook, or submits a form during the crawl needs a second look before you wire in automatic retries.

Stick to safe HTTP methods (GET, HEAD) wherever possible during the fetch phase of a scrape.
Use idempotency keys on any POST or PUT call that your scraper issues, so a duplicate retry is recognized and ignored server-side.
Deduplicate at ingest rather than relying on the scraper layer alone, since a crash between request and log write can still create duplicates.

Pro Tip: Put your dedupe check at the ingest pipeline, not just the client, since that is the one place that sees every attempt regardless of where in the retry chain it failed.

AWS's Well-Architected guidance specifically calls out verifying idempotency as a precondition before enabling any retry policy.

Which errors to retry and which to fail fast

A clear classification table saves hours of debugging later. Treat transient responses as potentially retryable, with backoff applied each time, and classify them according to the server and client behavior. Treat other 4xx codes as likely permanent failures, since retrying a malformed request or an auth rejection may not change the outcome.

Retry: connection timeouts, connection resets, and status codes classified as transient for your application.
Fail fast: errors classified as permanent, such as parsing or validation exceptions raised by your own code.
Honor Retry-After when a server sends it, but clamp the value to your backoff cap so one oversized header can't stall the whole job.

This split keeps your fleet from wasting cycles on requests that will never succeed while still giving genuinely flaky connections a fair second chance.

Implementation patterns: Scrapy, Requests, and SDKs

Scrapy's built-in RetryMiddleware handles most of this out of the box. Key settings are RETRY_TIMES (max retry attempts), RETRY_HTTP_CODES (which statuses trigger a retry), and the dont_retry meta flag for requests you want excluded. The get_retry_request helper reschedules a failed request, tracking attempt count in meta['retry_times'] and optionally adjusting request priority with priority_adjust so retries don't jump the queue ahead of fresh work.

For raw Requests or urllib3, configure a Retry object with settings such as total (max retries), backoff_factor (scales the delay between attempts), and status_forcelist (which status codes count as retryable).

total caps the number of retry attempts across the whole request lifecycle.
backoff_factor multiplies into the exponential delay formula between attempts.
status_forcelist lists the HTTP codes that should trigger a retry instead of raising immediately.

If you're using a cloud SDK rather than raw HTTP calls, check whether it ships retry quota or token-bucket behavior and an adaptive retry mode before building your own layer. Well-Architected guidance recommends favoring a proven built-in SDK retry implementation over custom logic whenever one is available, since it has already been tuned against real failure patterns.

Retry budgets and circuit breakers for fleet-wide safety

Individual retry logic protects a single request. Retry budgets and circuit breakers protect the whole fleet from amplifying an outage. A token-bucket retry budget tracks how many retries are allowed across a time window and blocks new retries once that budget is spent, which stops a struggling upstream service from being hit harder the moment it starts failing.

A circuit breaker adds a state machine on top: closed (normal traffic), open (stop sending requests after a failure threshold), and half-open (send a trial request to check recovery before fully reopening).

Open the breaker after a defined run of consecutive failures or a failure rate threshold over a rolling window.
Half-open periodically to test recovery without flooding the recovering endpoint.
Close again once trial requests succeed consistently.

Useful metrics: retry count per endpoint, retry success rate, per-endpoint latency, and how often Retry-After headers appear.

Retries only help when they're bounded and watched. A single request should never be allowed to hang indefinitely without a cumulative time cap, which is exactly the failure mode a retry budget and a breaker are designed to prevent.

Retry attempts stop at a shared time limit

Recommended defaults and a quick cheat sheet

Most scraper workloads do well starting from the same baseline settings and adjusting from there based on observed retry success rates.

Max attempts: 3 (1 initial try plus 2 retries) as a starting point for most endpoints.
Base delay: choose a delay appropriate to the error type and service behavior.
Cap and jitter: use a bounded cap and apply Full Jitter on every attempt.
Timeouts: connection timeout around 10s, read timeout around 30s.
Global deadline: set a reasonable ceiling for each page fetch.

Setting	Starter value
Max attempts	Choose based on endpoint and observed outcomes
Base delay (transient)	Choose based on error type and service behavior
Base delay (throttling)	Choose based on error type and service behavior
Backoff cap	Set an appropriate bounded cap
Connection timeout	10s
Read timeout	Set an explicit timeout
Global deadline	Set a reasonable operation ceiling

Tune from there by measuring retry success rate per endpoint, watching for repeated server-side error patterns that suggest a permanent block rather than a transient hiccup, and raising the cap only when you have evidence that longer waits actually improve success. Test the whole policy against simulated failures before trusting it in production, since a retry policy that looks correct on paper can still behave badly under real timing and concurrency.

Handling proxy-specific failures like IP bans and throttling

Proxy-related failures behave differently from generic server errors, and your retry logic should treat them differently too. An IP ban or a sustained 403 from a specific proxy exit node is not something a backoff delay will fix: the IP itself is the problem, not the timing of the request.

For these cases, retry logic needs to trigger a proxy rotation, not just a longer wait. A sensible pattern separates the two failure types: throttling signals (429, slow responses, intermittent 503s) get standard backoff with jitter on the same IP, while hard blocking signals (403, CAPTCHA pages, abrupt connection drops from the same exit node repeatedly) should trigger an immediate IP swap before any retry delay is applied.

Tag each failure with whether it looks like throttling (retry same IP with backoff) or blocking (rotate IP, then retry).
Track a per-IP failure count so a proxy that fails repeatedly gets pulled from rotation instead of retried indefinitely.
Avoid retrying through the same banned IP even after a backoff delay, since the ban rarely clears on its own within a scraper's operational timeframe.

This is also where retry logic and proxy infrastructure quality intersect directly. A fleet built on residential or mobile carrier IPs may produce fewer hard blocks, which can mean fewer rotation triggers and a simpler retry policy overall. Mapping your retry classification to the right action (wait vs rotate) keeps the logic from either giving up too early on a recoverable throttle or wasting cycles retrying a dead IP.

Retry logic and rotating vs sticky proxy sessions

How you configure proxy sessions changes what a sensible retry policy looks like. A rotating session assigns a new IP on each request or at intervals, which can work well for page fetching where each request is independent. A sticky session pins one IP for a longer window, which can suit multi-step flows like paginated crawls or login-gated content where consistency matters more than volume.

Retry behavior should match the session type. On a rotating setup, a retry after a block often lands on a fresh IP automatically, so your retry logic can be simpler: retry with backoff, and the rotation layer handles the identity change. On a sticky setup, a retry that reuses the same pinned IP after a block is retrying into the same wall, so the retry logic needs an explicit instruction to request a new sticky session rather than just waiting and trying again on the same one.

Rotating sessions: retry with standard backoff; a new IP usually arrives with the next attempt by default.
Sticky sessions: on a hard block, explicitly request a new session rather than retrying the pinned IP.
Match session length to task length: short sticky windows for quick multi-step flows, rotation for bulk single-page fetches.

Getting this mapping wrong is a common cause of retry storms: a scraper stuck in a sticky session that keeps retrying a banned IP for the full session duration burns far more requests than one that swaps sessions on the first hard block.

Retry logic for geolocation-specific scraping

Scraping tasks that depend on location, like local search results, regional pricing, or geo-gated content, add a constraint that generic retry logic doesn't account for: a retry on the wrong location is not useful even if it succeeds. If a request needs to appear to originate from a specific city, a retry that happens to land on an IP from a different region returns data that looks successful but is actually wrong for the task.

For these workflows, retry logic needs a location check as part of its success criteria, not just an HTTP status check. A 200 response from the wrong city should be treated the same as a failure and trigger a retry targeted at the correct location, not a generic retry that might land anywhere.

Validate the response location, not just the status code, before marking a geo-targeted request as successful.
Retry with the same city-level target rather than falling back to a generic IP pool when a geo-specific request fails.
Set a tighter global deadline for geo-targeted tasks if the available IP pool for that location is smaller, since waiting for a specific city's IP availability takes longer than a generic retry.

This matters more as the geographic target gets narrower. A country-level target may have a broader IP pool to draw from, while a city-level target can have fewer available IPs, so the retry logic should account for location availability.

Monitoring retry patterns across proxy environments

Generic retry metrics tell you whether retries are happening, but proxy-aware monitoring tells you why, which matters more for catching problems before they become fleet-wide outages. The baseline metrics from earlier (retry count, retry success rate, per-endpoint latency) need a proxy dimension added: which IPs, which sessions, and which locations are driving the retries.

Break retry success rate down by proxy pool or session type rather than looking at a single fleet-wide number. A fleet-wide success rate can hide a city pool or sticky session batch that is failing at a much higher rate, and that detail can help determine whether the fix is a retry policy change or a proxy rotation change.

Track retry success rate per proxy pool, not just per endpoint, to catch a bad IP batch before it drags down the whole metric.
Alert on a rising hard-block rate (403s, CAPTCHA triggers) separately from a rising throttle rate (429s), since they call for different fixes.
Watch session churn, meaning how often sticky sessions get forcibly rotated due to blocks, as a leading indicator of proxy pool quality degrading.

Setting alert thresholds on these proxy-specific signals, rather than only on generic HTTP error rates, can help a team catch a failing IP pool sooner.

What running scraper fleets taught me about retries

Retry storms rarely come from bad backoff math. They come from skipping the staging test where you simulate real failures and watch what the fleet actually does under load, not what the code looks like it should do. Run that drill before every meaningful retry policy change.

The other lesson is quieter: a lot of retry volume is really a signal about block rate, not request reliability. Cut the source of the blocking and the retry logic gets simpler almost by default.

— Jon

A different lever: fewer blocks means fewer retries

Retry logic handles failures well, but the best retry policy is the one that gets invoked less often. If your fleet is still hitting a persistent 429 or 403 rate even after tuning backoff, caps, and classification, the bottleneck is usually the IP pool, not the retry code.

Masklabs

We built MaskLabs around mobile carrier IPs, with city-level targeting, sticky and rotating sessions, and HTTP, HTTPS, and SOCKS5 support so you can plug it into existing retry logic. When tuning stops moving the needle on block rates, check our pricing page or see the full proxy lineup to evaluate whether more authentic IPs solve what better retry code can't.

FAQ

What is the best retry strategy for web scrapers?

Capped exponential backoff with full jitter is the strategy AWS's architecture guidance recommends for large-scale systems, since it spreads retries widely enough to avoid thundering-herd spikes on recovering servers. Pair it with a global deadline and idempotency checks so retries stay bounded and safe.

How many times should a scraper retry a failed request?

Choose a reasonable starting attempt limit for transient errors, then adjust it based on observed retry success rates. Permanent errors like 401 or 404 should not be retried at all regardless of attempt count.

Should I retry a 429 rate limit response?

Yes, a 429 is typically transient and should be retried with backoff, ideally honoring any Retry-After header the server sends. Clamp that header value to your own backoff cap so a long server-suggested wait doesn't blow past your global deadline.

What is a global deadline in retry logic?

A global deadline is a total time limit across all retry attempts for a single operation, separate from any individual request timeout. AWS's retry behavior documentation notes that without one, a request can effectively hang indefinitely while consuming resources.

Does better proxy infrastructure reduce how many retries I need?

Fewer IP bans and throttling events generally mean fewer retries triggered in the first place, since much of the retry volume in scraping comes from blocking rather than generic server flakiness. Switching to real carrier IPs with city-level targeting, like what we offer through MaskLabs, is one way teams address that root cause directly.