Proxy Usage Logging: What to Capture and How

Geometric illustration of proxy logging layers

Proxy usage logging means recording every request that passes through a proxy: who sent it, when, where it went, what came back and how many bytes it cost. Do it on your side of the connection, with a fixed set of fields, UTC timestamps and masked credentials. That gives you something you can debug, audit and bill against. This guide covers which fields matter, what HTTPS hides from a proxy, how to log from your own scraper, and how to read usage on a metered mobile proxy account.

TL;DR

Log at minimum a UTC timestamp, the client IP, HTTP method, URL, status code, user agent and bytes transferred. These are the core fields Splunk lists for web proxy data.
Log the internal client or job that sent the request, not just the proxy IP. Otherwise you cannot trace anything back.
HTTPS tunneled through CONNECT hides paths, headers and bodies from the proxy, so detailed logs for scraping traffic usually have to come from your own client.
Mask credentials and personal data before a line is written, and ship logs to a central, protected store.
Logs only help if something reads them: set alerts on error spikes and unusual volume, and reconcile bytes against your provider's usage dashboard.

What a proxy usage log is for
The fields every proxy log should carry
What the proxy can and cannot see
Logging from your own client
Logging rotating and sticky sessions
Where logs should live
Turning logs into alerts
Our perspective: log both sides, reconcile weekly
Reading usage on a metered mobile proxy plan
A starter checklist
FAQ
Sources

What a proxy usage log is for

A proxy sits between a client and a server. In the words of the Pydoll documentation, it parses the method, URL and headers, decides whether to check credentials, match an access list or serve a cached copy, then opens its own connection to the destination. Every one of those decisions is a chance to write a log line.

Teams keep proxy logs for four practical reasons:

Debugging. When a scraper starts returning empty pages, the log shows whether the problem is a 403, a timeout or a redirect loop.
Security and audit. Logs show which system reached which destination, and whether traffic patterns look wrong.
Cost control. On metered plans, bytes per job tell you which project is spending your data pool.
Compliance evidence. Some environments need a record of outbound access. The Canadian government's event logging guidance lists proxies among the network devices that should be logged.

There are two kinds of proxy, and the logging story differs. A forward proxy acts for the client and handles outbound traffic for filtering, logging, IP masking or scraping. A reverse proxy acts for the server. The webscraping.cc lifecycle explainer draws that line clearly. This article is about forward proxies, the kind you use for scraping, price monitoring and ad verification.

The fields every proxy log should carry

There is no single mandatory schema, but the same fields show up in nearly every guide. The Splunk Lantern page on web proxy data names source IP, URL, HTTP method, status and bytes as key fields. Add a user agent and a UTC timestamp and you have a log that answers most questions.

Field	Why it matters	Example use
Timestamp (UTC)	Lets you correlate across servers and time zones	Line up a spike with a deploy
Client identifier (source IP, job ID or credential name)	Attribution	Find which scraper caused the errors
HTTP method	Separates reads from writes	Spot unexpected POST traffic
Destination URL or host	Shows where traffic goes	Per-domain success rates
Status code	Outcome of each request	Detect 4xx and 5xx spikes
User agent	Identifies the software	Catch a misconfigured client
Bytes transferred	Cost and volume	Per-job data spend
Elapsed time	Latency	Find slow routes
Session or exit label	Ties requests to a sticky or rotating session	Debug session drops

The last two rows go beyond the minimum, but they pay off quickly. The Squid cache project documents a native format that includes request duration, and its customizable log formats page shows how to define your own fields if the default does not fit.

Log the client, not the proxy

One common mistake is logging the proxy's address as the source. Guidance on logging as part of cyber resilience says to record the client-side IP so you can attribute requests. If every line shows the same proxy address, you cannot tell which internal system started the request.

For scraping fleets, an internal IP is often not enough, because dozens of workers may share a host. Add a job ID, worker name or credential label to each line.

Mask what should never be stored

The same guidance recommends masking personal data and credentials in log content. In practice that means:

Strip Authorization and Proxy-Authorization headers.
Redact passwords and tokens in query strings.
Hash or truncate identifiers if the logs leave your security boundary.

A proxy username and password pasted into a log line is a credential leak waiting to happen. Treat logs as sensitive data from the first write.

What the proxy can and cannot see

This is where developers get surprised. According to the webscraping.cc explanation, HTTPS traffic sent through an HTTP CONNECT tunnel hides request URLs, headers and bodies from the proxy operator, because the proxy acts as a blind TCP relay. What the proxy sees is the destination host and port, the time, and the bytes moved.

Two consequences follow.

If you run the proxy, your server-side logs for HTTPS scraping traffic will be thin. You will have host, port, timing and byte counts, but not paths or status codes. To get full detail you would need TLS inspection with a custom CA certificate, which exposes everything in the clear. That is a heavy step with real privacy implications, and it is only appropriate on systems you control, for traffic you are entitled to inspect.

If you use a proxy provider, they cannot read your HTTPS requests either, and you should not want them to. Your detailed request logs belong in your own client. Our guide to HTTP vs HTTPS proxy decision rules covers how the two connection styles differ in what they expose.

The practical rule: use the proxy side for volume, host and timing, and the client side for URL, status and content-level outcomes.

Comparison of client and proxy log details

Logging from your own client

For scraping and automation, client-side logging is the most useful layer. You know the target URL, the response status, the retry count and which session was used. Wrap your HTTP calls in one function that writes a structured record every time.

A minimal pattern in any language looks like this:

Generate a request ID and note the start time in UTC.
Send the request through the proxy with your session or exit label.
On response (or failure), record method, host, path, status, response bytes, elapsed milliseconds and retry number.
Redact secrets, then write one JSON object per line.

JSON lines work well because every log platform can parse them and you can add fields later without breaking old queries. Keep the schema stable and version it.

Example record

A single line might contain: timestamp in ISO 8601 UTC, job name price-monitor-eu-catalog, session label sticky-0412, method GET, host, path, status 200, bytes 48213, elapsed 640 ms, retry 0. Nothing in it needs the proxy password.

Browser automation

With headless browsers, one page load triggers dozens of sub-requests. Log at two levels: one record per page navigation for outcome, and optionally per sub-request for bytes, since images, scripts and trackers can dominate your data spend. Blocking resources you do not need often cuts both bytes and log volume.

Logging rotating and sticky sessions

Mobile proxies add one wrinkle: the exit IP can change. With rotating sessions, consecutive requests may leave from different IPs. With sticky sessions, you keep the same IP for a set time. If your log does not record which mode and which session was in use, you cannot explain why one request failed and the next worked.

Record these in each line:

Session mode (rotating or sticky) and a session ID.
The exit IP, if your provider exposes it through an echo endpoint or response header.
The target city or region you requested, for geo-targeted work.

When you find a block, you can then ask: was it the same IP hit too many times, or a fresh IP that was flagged on arrival? Our notes on session management for stable proxy workflows go deeper on choosing session length, and the scraper success rate checks show how to catch silent failures such as a 200 status with an empty or challenge page.

Where logs should live

Writing to a local file is fine for a prototype. For anything shared or audited, centralize. The OWASP Proactive Controls entry on security logging recommends protecting log integrity, collecting logs from distributed systems in one place, and keeping them on a trusted system.

A workable setup for a small data team:

Applications write JSON lines to stdout.
A shipper forwards them to a central store with restricted write and read roles.
Retention is set deliberately: long enough for audits and billing disputes, short enough that you are not holding old data without reason.
Clocks are synced everywhere, so UTC timestamps actually line up.

Google Cloud's documentation on Secure Web Proxy transaction logs is a good example of what a managed service records for every transaction it mediates, and a useful reference when you design your own schema.

Turning logs into alerts

Logs nobody reads are storage costs. The OWASP Developer Guide stresses that monitoring means live review with automation, not only recording. OWASP's proactive controls also point to detecting attack patterns that rotate IP addresses to get around standard lockouts.

For a scraping or monitoring team, start with a short list of alerts:

Signal	Likely meaning	First response
403 or 429 rate climbs for one host	Target is throttling or blocking	Slow down, lengthen sticky sessions, check TLS and headers
5xx spike across all hosts	Network or provider issue	Compare against provider status
Bytes per job jumps 3x	Page weight change or a retry loop	Pause the job, inspect samples
Many requests from one credential at odd hours	Leaked or misused credentials	Rotate the credential
Response sizes suddenly tiny	Block or challenge pages returned as 200	Add content checks

On the defensive side, OWASP notes that a robust system should respond in real time, for example by invalidating sessions or flagging an account. If you run a proxy for your organization, wiring suspicious-pattern alerts to an automated response matters. This write-up on the VoidProxy phishing alert shows why proxy-style infrastructure is also a threat vector worth watching.

For access control rather than traffic analysis, the same discipline applies to your people. Reviewing who has proxy credentials is easier when auditing does not interrupt daily work, as this guide on auditing user access without disrupting work explains.

If you want to verify rotation behavior and speed as part of the same monitoring, see our roundup of proxy checker tools and the piece on proxy uptime monitoring with 3 to 5 samples.

Our perspective: log both sides, reconcile weekly

We run mobile proxies on real US carrier IPs, billed per byte, so usage logging is not an abstract topic for us. It is how our customers decide whether a project is affordable and healthy.

Here is what we recommend to teams building scrapers, price monitors and ad verification jobs:

Treat your client log as the source of truth for outcomes. Because HTTPS is tunneled, we cannot and should not read your requests. Your own log knows the status, the page and the retry.
Use the provider dashboard for volume. Bytes and requests per credential on the dashboard give you an independent total. If your client-side byte count and the dashboard total drift apart by a large margin, something is off: a retry loop, uncounted browser sub-requests or a credential in use somewhere you did not expect.
Give each project its own credential. With team and organizational credential management, one credential per job or per teammate makes attribution simple, and a leaked credential can be rotated without stopping everything else.
Tag every line with session mode. Sticky versus rotating explains most mysterious differences in success rate between two runs.
Alert on trends, not single failures. A lone 403 is noise. A rising 403 rate on one host is a signal.

The point of logging is to make a decision faster: slow down, switch session type, change the city, or stop the job. If a field in your log never informs a decision, drop it.

— Jon

Reading usage on a metered mobile proxy plan

If you are comparing logs with a bill, per-byte metering changes what you track. Bytes, not requests, drive cost, so log response size and, for browser automation, total page weight.

Masklabs plans are tiered by monthly data pool: Starter at $30.00 per month, Basic at $100.00 per month, Advanced at $500.00 per month and Scale at $1,000.00 per month. Prepaid data add-ons are billed per GB, and there is a free trial with limited data, so you can run a realistic job, compare your client-side log with the usage dashboard, and size a plan before you commit. Credentials are self-serve, with HTTP, HTTPS and SOCKS5 support and API access, so you can set up per-project logging without a sales call. See US mobile proxies on real carrier IPs for the full picture, and our post on proxy API features worth comparing for what to look for in usage reporting.

A starter checklist

Decide the fields: timestamp (UTC), client or job ID, method, host, path, status, bytes, elapsed, session mode.
Log in your client with one JSON object per line.
Redact credentials and personal data before writing.
Use one proxy credential per project or teammate.
Send logs to a central, access-controlled store with set retention.
Build three alerts first: block rate per host, bytes per job and error spikes.
Reconcile your byte totals against the provider dashboard weekly.
Review the schema every quarter and remove fields nobody uses.

FAQ

What should a proxy log contain?

At a minimum: a UTC timestamp, client IP or job identifier, HTTP method, requested URL, status code, user agent and bytes transferred, matching the fields Splunk identifies. Add elapsed time and session identifiers if you run scrapers. Mask credentials before anything is stored.

Can a proxy see my HTTPS requests?

Not normally. With a CONNECT tunnel, the proxy relays encrypted bytes and sees the destination host, timing and volume, but not URLs, headers or bodies, as webscraping.cc explains. Only a proxy that performs TLS inspection with a custom CA certificate can read them.

Where should I log: on the proxy or in my client?

Both, for different jobs. Proxy-side logs give you volume, host and timing. Client-side logs give you status codes, URLs and outcomes, which is what you need for debugging scrapers. Reconcile the two on bytes.

How long should I keep proxy logs?

There is no universal number. Keep them long enough for your audit, security and billing-dispute needs, and no longer. If you operate under a compliance regime, ask your security or compliance lead what retention it requires.

How do I log rotating proxy sessions?

Record the session mode, a session ID and, where available, the exit IP on every request. That lets you tell whether failures follow a specific IP or happen on fresh ones.