Try 500 MB of US mobile proxy data free for 30 days.Start free trial
September 9, 202614 min read

Copy Paste Wiring for Developers: Scrapy Proxy Rotation and MaskLabs

Isometric illustration of proxy routing paths

For most projects, the fastest reliable path is a gateway-managed rotating proxy or the scrapy-rotating-proxies middleware, paired with ban detection and per-proxy throttling. Use per-request rotation for stateless, high-volume crawls; switch to sticky sessions for anything that requires multi-step flows like logins or carts. To get moving right now: add the rotating middleware to DOWNLOADER_MIDDLEWARES, set ROTATING_PROXY_LIST or your gateway credentials, and turn on ban detection with backoff before you scrape a single page.

TL;DR:

  • Proxy rotation is essential when experiencing repeated blocking signals such as 429 or 503 responses, especially for high-volume, geofile-specific, or sensitive multi-step flows.
  • Configuring rotation requires installing middleware, setting proxy lists or credentials, and implementing ban detection policies that analyze response content, not just status codes.
  • The most suitable rotation mode depends on the job: per-request for stateless tasks, timed for short bursts, and sticky sessions for multi-step flows like logins, with proxy type affecting speed and stealth.
  • Ensuring proxies actually change IPs involves checking middleware priorities and response content, while cost-effective scaling involves adjusting pool size, concurrency, and health monitoring as request volume grows.
  • Mobile carrier proxies from Masklabs provide a high-anonymity alternative when datacenter or residential proxies are flagging, especially for geolocation-targeted, multi-step, or hard-to-evade targets.

Table of Contents

Do You Actually Need Scrapy Proxy Rotation?

Rotation adds moving parts, and not every spider needs it. If you're pulling a few hundred pages a day from a permissive target, a single well-behaved proxy or even your own IP might be enough.

You'll want it once you see specific signals: repeated 429 or 503 responses, a target that fingerprints and blocks after a burst of requests, or a job that needs data from multiple US cities or regions. Before wiring anything, confirm you have the basics in place.

  • Python 3.9+ and Scrapy installed via pip install scrapy, plus a working spider that already crawls your target without proxies.
  • A proxy source: either gateway credentials from a provider or a vetted list of IP:port entries, and clarity on whether those are datacenter, residential, or mobile IPs.
  • A rough sense of scale: thousands of requests per day, frequent blocking, or geolocation requirements are the clearest triggers for adding rotation now rather than later.

If none of those apply yet, skip rotation and revisit once your ban rate climbs or your request volume outgrows a single IP.

Which Rotation Method Fits Your Scraping Job?

Three rotation modes cover almost every use case: per-request, timed, and session-based (sticky). Per-request rotation swaps the exit IP on every outgoing request, which is ideal for stateless jobs like scraping product listings or search results pages where each request stands alone.

Timed rotation holds an IP for a fixed window (say, three minutes) before cycling to the next one. It's a middle ground: useful when a site tolerates a short burst from one IP but flags sustained traffic. Sticky sessions pin one IP for the life of a task, which matters when a target tracks session state across a login, a checkout flow, or a paginated form.

Proxy type matters as much as rotation mode:

  • Datacenter proxies are fast and cheap, but easy for anti-bot systems to fingerprint by ASN.
  • Residential proxies route through real home IPs, which makes them harder to flag but pricier and sometimes slower.
  • Mobile proxies run on real carrier networks, giving the highest anonymity because mobile IPs are harder to detect and better suited to aggressive anti-bot targets, at a higher cost per gigabyte.

A stateless price-monitoring crawl pairs well with per-request rotation over datacenter IPs. A multi-step account flow or a heavily protected retail site is a better match for sticky sessions over residential or mobile IPs.

How Do You Wire Rotation Into Scrapy Settings?

Scrapy handles proxy rotation through downloader middleware, the same layer that governs how requests and responses move through the framework. Getting the wiring right up front saves you a debugging session later.

  1. Install and register the middleware. Add rotating_proxies.middlewares.RotatingProxyMiddleware and rotating_proxies.middlewares.BanDetectionMiddleware to DOWNLOADER_MIDDLEWARES in settings.py. Priorities matter: set RotatingProxyMiddleware around 610 and BanDetectionMiddleware around 620, so proxies get assigned before the request goes out and bans get evaluated after the response comes back.

  2. Point it at your proxy source. For a static list, set ROTATING_PROXY_LIST = 'proxy1:port', 'proxy2:port', ...] or point to a file with ROTATING_PROXY_LIST_PATH. For gateway-based providers, you typically skip the list entirely and instead set a single proxy URL with embedded credentials, since [the provider handles IP diversity and health checks on their end.

  3. Add sticky session support if you need it. scrapy-rotating-proxies rotates per request by default, so multi-step flows need a small custom middleware:

class StickySessionMiddleware:
    def process_request(self, request, spider):
        session_id = request.meta.get('session_id')
        if session_id:
            request.meta['proxy'] = f"http://user-session-{session_id}:[email protected]:8000"

That pattern generates a session token per spider task and reuses it across requests, keeping one exit IP for the whole flow instead of rotating mid-session.

Catching Bans Before They Wreck Your Crawl

A 200 status code doesn't mean the page you wanted came back. Reliable ban detection checks response content, not just HTTP status: empty bodies, redirect loops to a challenge page, unusual HTML structure, or visible CAPTCHA markers all signal a soft block that a status-code check alone will miss.

scrapy-rotating-proxies ships a default BanDetectionPolicy, but you'll usually want to override it per target:

  • Check for known challenge page strings ("Access Denied", "unusual traffic") in the response body.
  • Flag unexpectedly small response sizes for pages that normally return several kilobytes of HTML.
  • Return None when you're unsure, since that tells the middleware to treat the response as inconclusive rather than a hard ban.

Pro Tip: Write a five-line policy that inspects the first 500 characters of the response body for your target's specific block page text. Generic policies miss soft blocks that custom checks catch immediately.

Once a proxy is flagged dead, ROTATING_PROXY_BACKOFF_BASE and ROTATING_PROXY_BACKOFF_CAP control how long it sits out before a recheck, using randomized exponential backoff so you don't hammer a banned IP on a fixed schedule. ROTATING_PROXY_PAGE_RETRY_TIMES sets how many different proxies a single page gets tried on before Scrapy gives up and logs a failure.

Tuning Concurrency and Cost Without Guesswork

Once rotation is live, your old concurrency settings mean something different. Enabling RotatingProxyMiddleware makes Scrapy's default concurrency options apply per-proxy rather than globally, so a CONCURRENT_REQUESTS value tuned for a single IP will behave very differently across a pool of fifty.

Start conservative and ramp up from real signal, not guesswork:

  • Set CONCURRENT_REQUESTS_PER_DOMAIN low (2 to 4) and DOWNLOAD_DELAY around 1 to 2 seconds per proxy for your first run.
  • Turn on AUTOTHROTTLE_ENABLED so Scrapy adjusts delay based on server response times instead of a fixed guess, then watch your ban rate over a few hundred requests before increasing concurrency.
  • Track how download delays and per-proxy concurrency interact under load to spot where you're leaving throughput on the table.

Cost control follows the same logic as request tuning. A tiered escalation strategy tries cheaper datacenter proxies first and only moves to residential or mobile IPs once a target shows repeated ban signals on the cheaper tier. That keeps spend proportional to how hard a site actually fights back.

Why Are Requests Still Getting Blocked?

Rotation configured correctly still fails sometimes, and the fix is usually one of a short list of causes.

  1. Verify the exit IP is actually changing. Hit an IP echo endpoint through your proxy with curl and compare it against what Scrapy logs in request.meta['proxy']. A static IP across "rotated" requests usually means the middleware isn't registered at the priority you think it is.
  2. Inspect the response body, not just the status. A 200 with a CAPTCHA page inside it will slip past status-only checks; grew the body for your target's known block strings.
  3. Check for session breakage. If a login or multi-step flow suddenly fails partway through, you're likely rotating mid-session when you should be sticky. Over-rotating is one of the more common causes of broken logins or abandoned carts.
  4. Confirm the pool isn't exhausted. A run of failures across every proxy usually means your list shrank below a workable size. Dedicated proxy checker tools can confirm health and speed before you assume the target changed its defenses.

Which Proxy Libraries Work Best With Scrapy?

scrapy-rotating-proxies remains the most widely used middleware for list-based rotation, and it's the right starting point if you're managing your own proxy pool rather than a gateway. It handles dead-proxy detection, backoff, and per-proxy concurrency out of the box, which covers most of what a custom build would otherwise require.

Beyond that project, a few integration patterns show up repeatedly in production Scrapy setups. Gateway-based providers skip proxy-list middleware entirely: you authenticate against a single endpoint, and the provider rotates the underlying IP on their side. This shifts health monitoring and dead-proxy detection off your codebase, which is worth the trade if you don't want to own that infrastructure.

For teams running Scrapy alongside browser automation (Playwright or Selenium for JavaScript-heavy targets), proxy configuration typically lives at the browser context level rather than in Scrapy's middleware stack, so you'll maintain two separate proxy configurations if your project mixes both approaches. Keep credentials centralized in environment variables or a settings module either way, so a proxy swap doesn't mean editing code in three different places.

Custom middleware still has a place when you need logic scrapy-rotating-proxies doesn't offer out of the box: weighting proxy selection by response time, routing specific domains to specific proxy pools, or logging per-proxy success rates to a database for later analysis. Writing that layer on top of Scrapy's existing downloader middleware hooks is straightforward once the basic rotation is working, since you're extending a pattern rather than inventing one.

Which Proxy Libraries Work Best With Scrapy? — overview diagram

How Should You Choose Proxies by Geolocation?

Geolocation requirements usually come from one of two places: the target website enforces region-specific content, or you specifically need data as it appears to users in a given city or country.

For simple country-level targeting, most gateway providers let you pass a country code as a parameter and get IPs from that region without further configuration. That's sufficient for jobs like checking whether a site geofences content or pricing by country.

City-level targeting is a different problem. Search results, local business listings, and geotargeted ad verification often shift meaningfully between neighboring cities in the same state, not just between countries. That level of precision generally requires a provider offering city-specific IP pools rather than a broad regional average, since a generic "US" IP can land anywhere from Seattle to Miami depending on the provider's infrastructure that day.

Match your rotation mode to the geolocation need as well. A single-city price check across many product pages works fine with per-request rotation, as long as every request draws from the same city pool. A multipage flow that depends on the site remembering your detected location (a shopping cart that locks in local pricing, for instance) needs a sticky session anchored to that city, or you risk the site resetting your location mid-crawl and returning inconsistent data.

Security and Privacy Considerations You Shouldn't Skip

Rotating proxies route your traffic through third-party infrastructure, which means credential handling deserves the same care you'd give any external API key. Store proxy credentials in environment variables or a secrets manager, never hardcoded in settings.py or committed to version control, since a leaked gateway credential can let anyone route traffic (and cost) through your account.

Traffic through a proxy is visible to that proxy operator, so avoid sending sensitive authentication tokens or personal data through the same connection you use for scraping, unless the provider explicitly supports that use case. This matters more with residential and mobile proxies, where the exit IP belongs to a real device rather than infrastructure the provider fully controls end to end.

Rotating IPs also has a compliance dimension worth planning around: scraping a site behind rotating proxies doesn't override that site's terms of service or applicable data protection law in your jurisdiction. Build your data handling around what you're legally permitted to collect and retain, and treat proxy rotation as a technical tool for reliability, not a way to sidestep a target's access rules.

Finally, watch your logs. Debugging often means printing proxy URLs and response bodies to console, and it's easy to accidentally log credentials alongside scraped data. A quick audit of your logging configuration before a production run prevents that kind of leak from sitting in a log file longer than it should.

Scaling Proxy Rotation Without Losing Reliability

Performance at scale is less about raw throughput and more about keeping your ban rate flat as request volume climbs. A pool that handles 10,000 requests a day cleanly can start failing at 100,000 if you haven't revisited pool size, concurrency, and health check frequency together.

Pool size needs to scale with request volume, not stay fixed. A rough rule: if a single proxy is handling more requests per hour than the target site tolerates from one IP before flagging it, your pool is too small for your current volume, regardless of how well rotation is configured.

Batch your health checks instead of checking proxies reactively only after a failure. Running periodic health checks against a sample of your pool. lets you catch degrading proxies before they cause a wave of failed requests mid-crawl, particularly important on long-running jobs that can't afford to stall.

Separate your retry logic from your rotation logic. A page that fails because of a genuine network timeout doesn't need a new proxy, just a retry; a page that fails because of a ban does. Conflating the two means you either burn through proxies unnecessarily or keep retrying on a dead one. Session management practices that separate these failure types tend to hold up better under sustained load than a single generic retry policy.

Finally, monitor cost per successful request, not just cost per proxy. A cheaper pool that requires three retries per page can end up costing more in bandwidth and time than a pricier pool that succeeds on the first attempt.

Scaling Proxy Rotation Without Losing Reliability — overview diagram

Build vs Buy: The Real Cost of Managing Your Own Proxy Pool

Running your own proxy list means owning dead-proxy detection, ban policy tuning, and backoff logic indefinitely, not just at setup. That operational tax is easy to underestimate until a target changes its blocking behavior and your whole pool goes stale overnight.

The scale signal that should push you toward a managed gateway isn't request volume alone. It's how often you're touching ban detection code. If you're rewriting your BanDetectionPolicy every few weeks to chase a moving target, you're paying an engineer's time to do what a managed rotating gateway already handles. Factor that maintenance cost against the price difference before assuming DIY is cheaper.

— Jon

MaskLabs: Mobile Rotating and Sticky Proxies When You Need Real Carrier IPs

When residential and datacenter proxies keep getting flagged, mobile carrier IPs are the next escalation, and that's the specific gap Masklabs fills. Masklabs routes requests through real mobile carrier networks instead of datacenter infrastructure, which means your requests look like they're coming from an actual phone on an actual carrier, not a server farm.

Masklabs

The platform supports both rotating and sticky sessions, so you can switch modes depending on whether your job is a stateless per-request crawl or a multi-step flow that needs one IP held in place. City-level targeting covers geolocation needs down to metro area, which matters for local search testing, geotargeted price monitoring, or AI dataset collection that needs to reflect what a real user in a specific city sees. API access means you can wire it into the same settings.py and middleware patterns already covered here, gateway credentials in, rotation logic out.

If your current pool is hitting ban walls that datacenter or residential proxies can't clear, start a Masklabs trial and test your target against real carrier IPs before committing to a full data plan.

Key Docs and Guides to Reference

Bookmark these before your next debugging session:

Sources

FAQ

What Is Proxy Rotation?

Proxy rotation is the practice of cycling through multiple IP addresses when making requests, so a target site sees traffic from many sources instead of one, reducing the chance of getting blocked.

Is Scrapy Outdated?

No. Scrapy remains actively maintained and widely used for production scraping, and its downloader middleware architecture is exactly what makes proxy rotation straightforward to implement in the first place.

How Do You Use a Proxy in Scrapy?

Set request.meta['proxy'] to your proxy URL in a downloader middleware, or install scrapy-rotating-proxies and configure ROTATING_PROXY_LIST in settings.py to handle rotation automatically.

How Do You Make Rotating Proxies?

You can build rotation manually with a middleware that cycles through a list of proxy URLs, or use scrapy-rotating-proxies for list-based rotation with built-in ban detection, or use a gateway provider like Masklabs where rotation happens automatically behind a single endpoint.

When Should You Use Sticky Sessions Instead of Rotating Proxies?

Use sticky sessions for any flow where the target site tracks state across requests, like logins, checkout carts, or paginated forms, since rotating mid-session tends to break the flow and trigger errors.

Recommended

Made with BabyLoveGrowth to create SEO content