Building AI training and RAG datasets with mobile proxies
By The Masklabs team
A model is only as good as the data you fed it, and a growing share of the public web now serves a different page to crawlers than it serves to people. If you're building a training corpus, a fine-tuning set, or a live retrieval index for RAG, that gap is a data quality problem before it's ever a scraping problem. Here's what's actually happening and how to crawl around it.
The page a bot sees isn't the page a person sees
Sites increasingly fingerprint the request, not just the content they return. A request arriving from a known datacenter IP range gets treated differently: a CAPTCHA wall, a stripped-down HTML shell missing the content that loads via client-side JavaScript, a cached "lite" version, or in some cases content that's deliberately altered to poison scrapers. None of that is visible to you as a binary blocked/not-blocked signal. You get a 200 response and a page that looks plausible, and it goes straight into your dataset.
That's the dangerous failure mode. A hard block is annoying but obvious; you retry, you notice the empty output, you fix it. A cloaked page that returns different text, different prices, different product descriptions, or a thinner version of an article is silent. It sits in your training set or your vector store looking exactly like real data, and every downstream model that learns from it inherits the distortion.
A request from a real US mobile carrier IP doesn't get sorted into a "probably a bot" bucket the same way. It's the same address space a phone in someone's pocket uses to load the same site. The page you get back is the page a human reader on that carrier would get back, which is the whole point if your goal is a dataset that reflects the actual live web.
Rotating IPs for breadth across many targets
Most corpus-building work is the opposite of a single deep session: you're pulling from thousands of distinct pages, domains, or sources, and no individual request depends on the state of the one before it. That's exactly what rotating sessions are for. Masklabs rotates by default, no suffix needed, so every request from your crawler credential can land on a different exit IP:
mlabs_a1f9c3:<PASSWORD>@proxy.masklabs.io:8080
Spreading requests across many exits means no single target sees a wall of
traffic from one address, which matters more the wider your source list gets.
If a handful of pages need session continuity, say a paginated result set
that's keyed to the requesting IP, append _sticky to hold one IP for about
three minutes on just that credential. See
the sessions and locations guide for the full
syntax, including city targeting if you need geo-specific results (a store
locator, region-priced listings) rather than a generic national view.
Byte-for-byte metering keeps a dataset build on budget
Corpus builds are usually sized in advance: you know roughly how many gigabytes of HTML, PDFs, or JSON responses you need before you start. Masklabs meters by the byte, no rounding up to some minimum per request, so a crawl that pulls 200 GB of source data costs exactly 200 GB against your plan's included pool or your prepaid balance. That predictability matters when you're budgeting a dataset build the same way you'd budget compute: you can estimate cost from target page sizes and request counts before you run the job, and the invoice matches the estimate.
If a run is going to exceed your monthly pool, buy a data refill at your
plan's per-GB rate rather than getting stuck mid-crawl. If a credential does
run out, it pauses instead of overbilling, so a runaway crawler script can't
turn into a surprise bill. Up to 50 credentials share one balance, which
works well if you're running separate crawlers per source or per project,
say a crawler credential for general corpus pulls and a price-monitor
one for a narrower recurring job, all drawing from the same pool with
separate usage lines.
Training corpora, fine-tuning sets, and live RAG
The same clean-page problem shows up in three slightly different jobs:
- Pretraining or large corpus builds. Breadth matters most here: many domains, many pages, rotating IPs so no target notices the volume, and metered cost so a multi-hundred-gigabyte pull stays predictable.
- Fine-tuning sets. Usually a narrower, curated source list where accuracy per page matters more than volume. A cloaked or CAPTCHA'd page in a fine-tuning set is worse proportionally, since there are fewer examples to dilute the bad ones.
- Live RAG retrieval. Freshness is the constraint, not breadth. A retrieval pipeline that's re-pulling the same set of pages on a schedule needs to see the current live page each time, not a stale cached version served to known crawler ranges. Rotating requests through mobile exits on each refresh keeps the retrieval index looking at what a person would actually find right now.
A short note on responsible use
None of this is a workaround for ignoring a site's rules. Check robots.txt
before you crawl, respect a site's terms of service, and don't pull content
you don't have the right to use in a dataset, especially anything
paywalled, personal, or explicitly restricted. Mobile IPs get you past bad
signals from IP reputation, not past the actual rules a site or the law puts
on its content.
The bottom line
If your dataset is only as good as the pages you scraped it from, the exit IP you scrape with is a data quality decision, not just an access decision. Rotating mobile IPs get you the page a real reader sees instead of a cloaked or blocked substitute, byte-for-byte metering keeps a planned crawl on budget, and the same setup covers pretraining pulls, fine-tuning sets, and live RAG refreshes without changing your approach. Start with the quickstart and check pricing for plan sizes if you're estimating a build.