Try 500 MB of US mobile proxy data free for 30 days.Start free trial
October 13, 202612 min read

Five Inputs Turn Scraping Bandwidth From Guesswork Into a Budget

Geometric inputs converging into a bandwidth budget

The planning formula is simple: monthly bytes equal items per month multiplied by requests per item, multiplied by average bytes per request, multiplied by a retry multiplier, multiplied by a growth factor. Run that calculation with five inputs, items per month, requests per item, average bytes per request (upload, download, and headers combined), a retry multiplier, and a growth margin, and you get a defensible bandwidth budget. The catch is that every one of those numbers needs to come from a real sample, not a guess.

TL;DR:

  • Keep HTTP and browser request counts separate: browsers load supporting assets and background calls, so mixing their counts can distort estimates by an order of magnitude.
  • Sample traffic across cold and warm caches, regions, and network conditions; browser transferSize may show zero for cached resources or cross origin requests without permission.
  • Count failed attempts and redirects as bandwidth: two redirects can triple requests per item, while POST workflows also require measuring upload bytes.
  • Add a 10% to 30% buffer to the worst case projection for measurement error and unplanned retries; the example budget reaches 182 to 215 GB.
  • Pilot a few hundred to a few thousand representative items across several hours, then remeasure monthly for stable targets and weekly for frequently changing sites.

Table of Contents

Define the Unit of Work and Requests per Item

Before you calculate anything, decide what counts as "one item." A unit of work might be a single HTML page, one API record, or a multi-page sequence like a product page plus its reviews tab. Pick whichever matches how your project is scoped, because that choice changes every number downstream.

Once you have a unit, count every request that belongs to it. Mixing a lightweight HTTP client count with a full browser session count is a common error that throws off estimates by an order of magnitude, since a browser loads dozens of supporting assets a bare HTTP request never touches.

Requests to count per item typically include:

  • The initial HTML page load or primary API call
  • XHR and background API calls triggered after load
  • Images, scripts, fonts, and stylesheets, when using a browser
  • Redirects the target site issues before landing on final content
  • Authentication or session-setup steps, such as login or token refresh

Measure a Representative Sample for Average Bytes per Request

Averages only mean something when they come from a sample that resembles production traffic. For simple HTTP targets, cURL's --write-out option reports size_download and size_upload directly, which gives you a fast way to log byte counts across a batch of real requests. For browser-driven targets, Playwright's Request API exposes requestBodySize, responseBodySize, and responseHeadersSize on every network event, which you can log across a crawl.

When reading browser metrics, note the difference between transferSize and decodedBodySize. The MDN documentation on transferSize explains that a cached or cross-origin resource without a Timing-Allow-Origin header can report transferSize as zero, which understates your real usage if you are not careful.

Sample across conditions, not just pages:

  • Pages with both cached and uncached assets
  • Different times of day and network conditions
  • Multiple target locales or regions, since page weight varies by market

Pro Tip: Run your sample multiple times under cold and warm cache conditions to observe how caching affects your bandwidth use.

The Planning Formula and a Worked Example

The formula has five variables, and each one maps to something you measure or decide:

  1. Items/month: the volume of units of work you plan to process.
  2. Requests/item: the count you built from the unit-of-work breakdown above.
  3. Avg bytes/request: upload plus download plus headers, from your sample.
  4. Retry multiplier: a factor above 1.0 that accounts for failed attempts and redirects.
  5. Growth factor: a margin for expected volume increases over the billing period.

Say you plan to scrape many product pages a month, each requiring multiple requests on average (HTML plus a handful of XHR calls), with an average of 150 KB per request, a retry multiplier of 1.15, and a growth factor of 1.2 to cover a planned expansion.

That gives a calculation multiplying those values, resulting in a substantial monthly bandwidth usage in the range of hundreds of gigabytes.

On top of that raw number, add a 10 to 30% buffer for measurement error and unplanned retries, pushing a realistic budget to somewhere between 182 and 215 GB before you commit to a provider plan.

Model Retries, Redirects, and Failed Requests

The retry multiplier is where estimates usually go wrong, because developers forget that a failed request still consumes bytes.

Redirects count as separate requests with their own headers and sometimes a small body, so a target that issues two redirects before the final page effectively triples your per-item request count. For POST-heavy workflows, remember to include upload bytes in your sample, since form submissions and authentication payloads add weight that a GET-only estimate misses entirely.

  • Log retry attempts and redirect hops during your pilot run
  • Separate "redirect overhead" from "retry overhead" in your notes
  • Recalculate the multiplier any time the target site changes its flow

Browser Automation: Assets and Why Page Loads Are Heavier

A full browser load pulls in far more than the page you asked for. Images, fonts, third-party scripts, analytics beacons, and background XHR calls all add to the total transferred, often multiplying the byte count of a single page by five or more compared to a plain HTTP fetch of the same URL.

HTTP/2 multiplexing, browser caching, and Service Workers can all shrink what Chrome DevTools reports as transferred on repeat visits, which is useful for cost control but can also make a single sample look lighter than a cold-start production run.

Common heavy additions in a browser session:

  • Hero images and product photography
  • Web fonts and icon sets loaded on every page
  • Analytics, tag managers, and chat widgets
  • Background XHRs that refresh content after initial load

When your target renders content dynamically or needs a logged-in session, a headless browser sample is the only way to capture realistic numbers. For static HTML or well-documented APIs, an HTTP client is lighter and gives you a cleaner baseline.

Reduce Waste Before Scaling: Practical Optimizations That Cut Billed Bytes

Before committing to a monthly plan, trim what you do not need. Selective asset blocking, skipping images, fonts, and third-party scripts in a headless browser, can cut page weight substantially, and switching to an API endpoint instead of a rendered page avoids the asset problem entirely when the target offers one.

Caching responses and deduplicating requests across repeated runs stops you from paying for the same bytes twice. Tuning concurrency and backoff settings reduces the 429 responses and retries that quietly inflate your multiplier, and instrumenting your scraper to log the highest-bandwidth requests shows you exactly where to focus first.

  • Block unnecessary asset types in browser automation runs
  • Cache and deduplicate requests across scheduled jobs
  • Tune concurrency and backoff to cut retry-driven waste
  • Log per-request bytes to find your heaviest offenders

Pro Tip: Sort your logged requests by size once a week; the top five often account for a disproportionate share of your bill. For a deeper walkthrough of these fixes, see our guide on cutting proxy bandwidth through developer-side changes.

Forecast, Pilot, and Calibrate: Turning Estimates into a Budget Range

A short pilot beats a spreadsheet guess every time. Run a few hundred to a few thousand representative items, long enough to cover a few hours of real variation in network conditions and target behavior, before locking in a number.

  1. Design the pilot with targets that match your production mix, not just the easiest pages to scrape.
  2. Compute best, expected, and worst-case monthly projections from the pilot's observed bytes per item.
  3. Apply your buffer to the worst case, then set a re-measurement cadence, monthly for stable targets, weekly for ones that change often.

Treat the first pilot's numbers as a starting point rather than a final budget. Target sites redesign pages, add tracking scripts, or change pagination, and any of those shifts can move your average bytes per request enough to matter.

Impact of Proxy Type on Bandwidth Usage and Estimation Accuracy

The kind of proxy you route traffic through does not change how many bytes a page weighs, but it changes how reliably your estimate holds up once you are running at volume. Datacenter proxies are often flagged or throttled by anti-bot systems, and a blocked or challenged request still consumes bandwidth without returning usable data, which quietly inflates your retry multiplier beyond what a clean pilot predicted.

Mobile proxies route through real carrier IP addresses, which tend to draw less scrutiny from anti-bot systems than datacenter ranges. Fewer blocks and challenges mean your observed retry rate stays closer to what you measured during the pilot, which keeps your bandwidth estimate accurate once you scale from a few hundred requests to tens of thousands.

This matters most for the retry multiplier in your planning formula. If a provider's IPs get flagged at a higher rate, every blocked attempt either burns bytes on a failed response or forces a retry that burns bytes twice, and that effect compounds across a month of volume. A proxy type with a lower block rate keeps your multiplier closer to 1.0, which is the assumption your worked example depends on.

When you are comparing proxy types for a scraping project, factor in not just the advertised success rate but how that rate holds up on the specific sites you target. A pilot run through your actual candidate proxy type, rather than a generic benchmark, is the only way to know whether your retry assumptions will survive contact with production traffic.

Impact of Proxy Type on Bandwidth Usage and Estimation Accuracy — overview diagram

Proxy Session Type: Sticky vs Rotating and Bandwidth Consumption

Session type changes how many connection setups your traffic incurs, and connection setup carries its own overhead in headers and handshakes. A sticky session pins one IP for a set window, often a few minutes, which lets you complete a multi-step flow, like a login followed by several authenticated requests, without repeating authentication overhead on every call.

Rotating sessions assign a new IP more frequently, sometimes on every request. That pattern is useful for high-volume, independent requests where each one needs a fresh identity, but it can add overhead if your target requires session continuity, since a dropped session forces you to restart multi-step sequences and repeat requests you had already completed.

For bandwidth estimation, the practical takeaway is to match session type to your request pattern before you sample. If your unit of work is a multi-request sequence, a sticky session sample will give you a more accurate per-item byte count, because a rotating session might force extra authentication retries that inflate your multiplier artificially. If your unit of work is a single independent lookup, rotating sessions avoid unnecessary connection persistence and your sample will reflect that lighter pattern.

Measure both ways if your project could reasonably use either, since the gap between sticky and rotating byte counts on the same target can be large enough to shift your monthly projection by a meaningful margin.

Proxy Session Type: Sticky vs Rotating and Bandwidth Consumption — overview diagram

Accounting for Geolocation-Specific Latency and Overhead in Bandwidth Calculation

Where your proxy IP appears to be located affects more than compliance with a target's geo-restrictions. Routing through a region farther from the target server adds latency, and higher latency increases the odds of a timeout, which triggers a retry that counts as a fresh set of bytes against your multiplier.

Page content itself can also vary by locale. A localized version of a page might load different images, a different currency widget, or additional region-specific scripts, which changes your average bytes per request independent of any network overhead. If your pilot sampled only one geography, your estimate will miss that variation entirely once you expand to other regions.

When your project needs city-level or region-specific targeting, build that into your sample from the start. Measure average bytes per request separately for each geography you plan to target, rather than assuming a single national average applies everywhere, and fold any observed timeout or retry differences into a region-specific multiplier if the gap is meaningful.

Publisher Perspective: Quick Lessons and Pilot Checklist

The mistakes we see most often come from small samples, mixing HTTP client counts with browser counts in the same formula, and forgetting redirects entirely. A pilot run on fifty pages during off-peak hours will not tell you what a production crawl at ten thousand pages during business hours looks like.

Before trusting any estimate, confirm your sample covers representative pages, both cached and uncached loads, and at least one retry scenario. For implementation details on automated browsing, see our guide on reliable scraping setups and our notes on how rate limiting affects large-scale automation.

— Jon

MaskLabs as an Option for Metered Mobile Proxy Bandwidth

Once your pilot gives you a realistic byte-per-item number, the next decision is which provider to budget against. We bill per gigabyte through metered mobile proxy plans built on real carrier IPs with targeting options, so your pilot's measured bytes translate directly into a predictable monthly cost rather than a rough guess.

Masklabs

Our sticky and rotating session options let you match the session type to your unit of work, the same distinction that matters for your retry multiplier.

  • Use the Starter plan at $30.00 per month to run your first pilot without overcommitting
  • Scale into higher tiers once your calibrated estimate is confirmed
  • Review the proxy product details for session and targeting options before you commit

Start your pilot on Starter, measure your real bytes per item, then move to the tier that matches your calibrated monthly number.

FAQ

Is AI Scraping Illegal?

Legality depends on jurisdiction, the target site's terms of service, and what data is collected, so no blanket answer applies everywhere. Check the specific laws and terms that apply to your target and region, and consult legal counsel for a project-specific answer rather than relying on a general rule.

How Do You Calculate Bandwidth?

Multiply items per month by requests per item, by average bytes per request, by a retry multiplier, by a growth factor, using numbers pulled from a representative sample rather than assumptions. This planning formula, sampled with tools like Playwright's request timing API, gives a defensible monthly estimate.

How Much Bandwidth Is Required for 500 Users?

There is no fixed figure for "500 users" because bandwidth depends entirely on what each user's session requests, how many pages or API calls they trigger, and how heavy each page is. Run the same planning formula with your own measured requests per user and average bytes per request to get a number specific to your application.

Can ChatGPT Do Web Scraping?

A language model like ChatGPT can help write or debug scraping code, but it does not itself make HTTP requests to external sites as part of a conversation. Actual data collection still requires a script or browser automation tool, such as Playwright, paired with proxy infrastructure to handle requests at scale.

Sources

For deeper technical detail, these sources cover the measurement mechanics referenced throughout this guide:

Recommended