Try 500 MB of US mobile proxy data free for 30 days.Start free trial
October 17, 202614 min read

Scraping at Scale: 9 Components, Separate Discovery and Retrieval

Modular geometric structure suggesting scalable architecture

Scraping at scale requires decoupled discovery, per-target concurrency limits, polite adaptive throttling, a resilient proxy strategy, durable queues, and monitored data sinks. Robots.txt is governed by RFC 9309, and Scrapy's AutoThrottle is a proven dynamic-throttling model. Mobile proxies such as ours at MaskLabs are one option for the IP layer.

TL;DR:

  • Separate URL discovery from retrieval with a durable queue, then scale each independently and enforce concurrency per host rather than across the whole crawl.
  • Use AutoThrottle and gradual canaries to tune each target; back off when latency rises or 429 responses appear, and honor robots.txt.
  • Use sticky proxy sessions for logins and other stateful flows, rotating IPs for stateless volume, and quarantine failing routes before retrying.
  • Prefer sitemaps, exports, or search endpoints over sequential link walking; split capped inventories by category or price band to avoid missing results.
  • Keep raw responses in object storage and structured records in a warehouse; canonical URLs and content hashes prevent duplicate work and support reprocessing.

Table of Contents

What do you need to assemble for a production-scale scraper?

Think of this as a blueprint, or as an audit list for the stack you already run. Each component has a natural owner, and most failed crawls trace back to one that nobody owned.

  • URL discovery: sitemaps, exports, search endpoints and seed generators that produce work. Scraper developers own this.
  • Durable queue: the buffer between discovery and retrieval. Platform or infra owns it.
  • Per-target limiter: concurrency and delay rules enforced per host, not globally. Scraper developers define the rules, platform enforces them.
  • Proxy pool: IP supply, session policy and health checks. Platform owns the pool, scraper developers choose the session mode per target.
  • Headless or browser workers: reserved for pages that genuinely need JavaScript rendering. Shared ownership, because the compute bill lands on infra.
  • Parsing pipelines: extraction logic that turns responses into records. Scraper developers.
  • Dedupe and transform: canonical URLs, content hashes and normalization. Data engineering.
  • Storage: an object store for raw responses plus a warehouse for structured output. Platform and data engineering.
  • Monitoring and runbooks: metrics, alerts and written recovery steps. Platform, with scraper developers contributing per-target alert thresholds.

Common bottlenecks tend to cluster by layer. Discovery stalls when a single process holds every URL in memory. Retrieval stalls on a fragile target or a saturated proxy pool. Parsing stalls when CPU-bound callbacks block the event loop. Storage stalls when the warehouse load job falls behind and the pipeline backs up into the crawler.

If you can name the owner and the failure signal for every item above, you have an operable system. If any item has neither, that is where your first outage will come from.

How should you decouple discovery from retrieval?

Scaling beyond a simple script means splitting the job in two. Discovery finds URLs. Retrieval fetches them. A durable queue sits between the two, and each side scales on its own. The Scrapy documentation on distributed crawling describes this pattern: decouple discovery from retrieval, manage URLs in a durable queue, and enforce concurrency per target rather than globally.

URLs moving from discovery through a durable queue

Why it matters: a crawler that follows links and fetches in the same process keeps its entire frontier in memory. One deep site can push that process into an out-of-memory crash, and you lose the frontier with it. Separate workers can be restarted, scaled up or replaced without touching discovery.

Practical queue choices include:

  • Redis streams: low latency and simple to operate, good for mid-sized frontiers with consumer groups.
  • SQS: managed, with visibility timeouts that handle worker crashes for you.
  • Kafka: high throughput and replayable history, at the cost of more operational weight.
  • Disk-backed persistence: Scrapy's JOBDIR-style job state and disk-backed dupe filters let a crawl survive restarts instead of starting over.

Per-target concurrency deserves its own rule. A global concurrency limit sounds safe until all requests land on one small domain. Enforce limits per host or service, and use queue priorities so pagination and discovery seeds are fetched ahead of low-value detail pages. If you write async workers in Python, the asyncio synchronization primitives such as semaphores work well for internal limits, but they are not thread-safe, so build the limiter around the resource you are actually protecting.

The trade-offs are real. Memory is fast but fragile, disk survives restarts but adds latency. Batching URL production raises throughput, while on-demand link following keeps the frontier fresh and small. Pick per target, not per project.

Pro Tip: Give every queue message a target key and a priority, then let workers pull by target so a single slow domain never blocks the rest of the crawl.

How do you design a proxy and IP strategy at scale?

Your proxy design decides your success rate, your geographic accuracy and a large share of your bill. Start by matching the session mode to the traffic.

  • Sticky sessions: use them for logins, carts, multi-step flows and anything session-bound. One IP holds for a window, so the site sees a coherent visitor.
  • Rotating IPs: use them for high-volume, stateless requests such as listing pages and search results, where no single IP should carry much traffic.
  • Mixed proxy types: datacenter IPs keep cost down on tolerant targets, while residential or mobile IPs add authenticity on targets that block aggressively. Apify's large-scale scraping guide lists mixing proxy providers among its standard patterns, because relying on one provider turns one outage into a total outage.
  • Health checks and failover: track errors per IP and per target, quarantine an IP after repeated failures, and fail over automatically instead of retrying the same bad route.

Pool sizing starts from your request rate per target. Divide that rate by what a single IP can sustain without errors, then add headroom for quarantined IPs. Revisit the number whenever a target tightens its rules. Session reuse needs a written policy too: how many requests per sticky session, how long it lives, and when it is retired.

API-driven monitoring helps here. If your provider exposes an API, pull session and usage data into the same dashboards as your crawl metrics so a proxy problem is visible next to the error rates it causes.

We built our service around this split. We provide real US mobile carrier IPs with city-level targeting, plus sticky and rotating sessions and API access, so one provider can cover both modes. Our post on rotating proxies for scalable data extraction covers the rotation side in more depth.

Pro Tip: Tag every request with the proxy session ID in your logs. When a target starts failing, you can tell in minutes whether the problem is one IP or the whole route.

How do you measure the bottleneck and throttle politely?

Before you raise concurrency, find out what is actually limiting the crawl. The Scrapy optimization docs describe the usual suspects: the downloader, scheduler queues, parse callbacks, DNS resolution, CPU-bound callbacks and saturated bandwidth. Each leaves a different trace.

  • A downloader that is always at its active-request ceiling means concurrency is the limit.
  • A growing scheduler queue with an idle downloader points at discovery or callbacks.
  • Rising retry counts and spikes of 429 or 503 responses mean the target is pushing back.
  • High CPU in callbacks means parsing, not the network, is the bottleneck.

Scrapy's AutoThrottle adjusts request delays from observed response latency so the crawler approaches a target concurrency without overloading the site. You configure a start delay, a maximum delay and the target concurrency per remote server. It also ignores non-200 responses when lowering delays, so an error page that returns quickly cannot trick it into speeding up.

Politeness also means respecting the Robots Exclusion Protocol. Our robots.txt checklist for developers walks through the checks worth automating. For the wider effect of server-side limits on automation, see our piece on how rate limiting affects large-scale web automation.

A safe tuning loop looks like this:

  1. Run a small canary against one target and record latency, error rate and throughput.
  2. Raise concurrency in small steps, holding each step long enough to see a trend.
  3. Watch latency and error rates after every change.
  4. Back off immediately when latency climbs or 429s appear, then settle one step below that level.

Scrapy's documentation recommends raising concurrency gradually and measuring each limit before changing the next, which is the whole method in one sentence: change one variable at a time and read the metrics.

How do you produce URLs fast enough to keep the downloader busy?

Concurrency settings are useless if the frontier is empty. The aim is to generate large URL sets quickly without exhausting memory.

  • Prefer bulk sources: sitemaps, data exports and search endpoints beat walking pages one after another. The Scrapy docs list sitemaps among their recommended patterns for large crawls.
  • Split by filter when pagination is capped: many sites stop at a fixed number of result pages. Slice the inventory by category, brand or price range so each slice stays under the cap. Apify's guide describes this pagination splitting as a standard pattern.
  • Split recursively: if a price range still returns more results than the cap allows, divide it in half and repeat until every slice fits. This enumerates an inventory with no gaps and no sequential walk.
  • Prioritize pagination requests: list pages unlock many detail pages, so give them higher queue priority than the detail pages themselves.
  • Persist the frontier: for very large inventories, use a disk-backed scheduler or JOBDIR-style job state so a restart resumes instead of rediscovering.

Say a catalog caps results at a few hundred items per query. Splitting by category first, then by price band, turns one capped query into dozens of complete ones. Run discovery as its own process, feed the queue, and the retrieval workers never wait for work.

When should you use headless browsers?

Choose the cheapest tool that returns the data. Plain HTTP is the right default when the data sits in an API endpoint or in server-rendered HTML. Reach for a headless browser when the page needs complex JavaScript rendering or a client-driven flow before the data appears.

  • Check for an API first: open the network panel and look for the JSON call behind the page. Calling it directly is faster and far cheaper than rendering.
  • Pool the workers: keep a fixed set of browser workers behind the queue instead of launching one per request.
  • Reuse contexts where safe: sharing a browser context across pages of one session cuts startup cost, but never share one across unrelated sessions or accounts.
  • Capture rendered HTML: store the post-render DOM and parse it offline. It is smaller than screenshots and can be re-parsed later.
  • Pace realistically: sites check client behavior with JavaScript, so sensible request pacing and session rotation matter more than any trick.

Browser traffic usually runs through the same proxy layer as everything else. Our guide to the Proxy API for browser automation projects covers wiring that up.

Hybrid designs work best: a cheap HTTP pass for the bulk of pages, and a browser pass only for the subset that fails to parse.

Pro Tip: Log the share of pages that needed a browser. If it creeps upward, a site has moved more logic to the client, and it is time to look for the underlying API again.

How should you store and deduplicate scraped data?

Treat raw responses and structured records as two different assets, and store both.

  • Raw layer: write raw HTML or JSON to object storage. Snapshots let you replay and re-parse when a selector breaks, without hitting the target again.
  • Structured layer: load parsed records into a warehouse for querying and downstream jobs.
  • Content hashing: hash normalized content to detect unchanged pages and skip redundant downstream work.
  • Canonical URL mapping: strip tracking parameters, normalize case and trailing slashes, and map variants to one canonical key before deduplicating.
  • Incremental merge: run dedup as a stage that merges new records into existing ones, so reruns update rather than duplicate.
  • Backpressure policy: watch queue depth and pipeline lag. Decide ahead of time whether each stream buffers, scales its consumers or drops low-value items when downstream falls behind.

Writing the drop-versus-buffer decision down before an incident saves an argument during one.

What should you monitor, and what is the runbook?

A large crawl needs a short list of metrics that you actually watch: requests per second, per-domain error counts, queue depth, pipeline lag and worker health. Alert on trends, such as a rising error rate or queue depth that keeps growing, not on single spikes. Our post on developer checks that catch silent failures is a good source for success-rate checks that go beyond HTTP status codes.

When something breaks, work down this list:

  1. Rising 429s: pause the affected seeds, cut that target's concurrency, then resume in small steps.
  2. Memory growth: check for an in-memory frontier or a leaking callback, move the frontier to disk and restart workers.
  3. Pipeline backlog: scale consumers or apply your drop-versus-buffer policy, and hold new discovery until lag clears.
  4. Proxy pool failures: quarantine failing IPs, rotate to fresh ones, and shift the target to a different proxy type if errors persist.

For testing, run small canaries before every large run, roll out changes gradually, and keep synthetic probes on critical endpoints so you learn about a block before the full crawl does.

Pro Tip: Write each runbook step as a command or a dashboard link, not a sentence. At 3 AM nobody wants to interpret prose.

Where does the money go, and how do you cut cost per record?

Large-scale scraping costs accumulate in a few places:

  • Proxy data transfer: metered per byte on many services, so heavy pages and retries cost real money.
  • Headless compute: browser workers are far more expensive than HTTP workers.
  • Bandwidth and storage: raw snapshots add up quickly.
  • Engineering time: often the largest line, and the easiest to forget.

Levers that reduce cost without hurting reliability:

  • Targeted crawling windows: crawl when targets are quiet and limits are looser.
  • Mixed proxy pools: send tolerant targets through cheaper IPs and keep premium IPs for the ones that need them.
  • Cache rendered pages: do not render the same page twice.
  • Efficient parsing: skip unneeded assets and block images where the data does not need them.
  • Cost per successful item: the one metric that counts retries, failures and compute together.

Instrument that metric early, then iterate with small pilots rather than guessing.

What are the operational lessons that matter most?

A few habits separate crawls that last from crawls that collapse: decouple discovery from retrieval, measure the bottleneck before touching a setting, set concurrency per target, and let adaptive throttling do the tuning. Most scraping outages I have seen came from skipping one of those, not from anything exotic. Teams reach for more workers when the real limit was a polite delay on one fragile host. Fix the measurement first, and the scaling decisions get boring in the best way. For deeper operational posts, our engineering writing lives on the MaskLabs blog.

— Jon

How can MaskLabs support a scalable scraping project?

Masklabs

If you are building at scale, the proxy layer should not be the part you fight with. We provide real US mobile carrier IPs, not datacenter ranges, with city-level targeting, sticky and rotating sessions and API access, so your requests look like ordinary mobile traffic. Our pricing page lists the Starter plan at $30.00 per month, Basic at $100.00 per month, Advanced at $500.00 per month and Scale at $1000.00 per month. A sensible path is to run a pilot on one target, measure success rate and error profile, then size your data usage and pool from real numbers. Start your pilot on our US mobile proxy platform.

FAQ

What is the process of web scraping?

Web scraping discovers URLs, fetches each page, extracts the data you need and stores it. At scale, discovery and fetching are separated by a durable queue so each can grow independently, as described in the Scrapy optimization docs.

Can ChatGPT do web scraping?

A chat assistant can help you write scraper code, debug selectors and design a pipeline. It does not replace the infrastructure for fetching pages reliably, such as queues, throttling and proxies, which is what scraping at scale depends on.

What are some free tools for scraping websites?

Scrapy is a free, open-source framework with built-in AutoThrottle and job persistence, and Python's asyncio handles lightweight concurrent fetching. Free tools cover extraction well, but large crawls usually also need a proxy and monitoring layer.

Is web scraping hard to learn?

The basics of fetching and parsing a page are approachable if you know HTTP and some Python. The difficulty is operational: throttling, retries, proxies and monitoring, which you pick up gradually by running small canaries before scaling.

Sources

Recommended