Incremental Web Scraping: ETags, 304s, and Fresh Data
Learn how to use ETags, conditional GETs and context-aware baselines to collect fresh web data with fewer wasted requests—without confusing a 304 with proof that every market is unchanged.

A web collection pipeline can be technically successful and still deliver stale or misleading data. Re-fetching every URL wastes transfer and parser time; treating every response as a change inflates downstream work. Incremental web scraping asks a narrower question: for this permitted URL, in this market and request context, has the selected representation changed since the last verified observation? HTTP validators are a useful first signal, but the collection contract and stored baseline determine what the answer means.
TL;DR — Store a baseline per URL and request context. Revalidate with ETag or Last-Modified when the source provides them, retain the baseline on a valid 304, and inspect a 200 response before declaring a meaningful change. Keep location, language and session variants separate; confirm apparent removals and respect source limits. Arcvenza Residential Proxies can provide an appropriate location-aware route, while Crawler Console can help organize crawl operations; neither replaces the validation logic described here.
1. Define freshness before optimizing requests
Start with the dataset contract, not the proxy pool. List the permitted source paths, entity key, fields, market, language, observation interval and acceptable delay. A price monitoring job may need an observed price, currency, seller, availability and timestamp; a documentation monitor may care only about the main article text. The same byte change is not equally important to both jobs. Record an initial complete snapshot and the timestamp at which it was actually observed. A stale or incomplete baseline makes every later ‘unchanged’ result suspect.
Define three separate signals: transport freshness (when the source was checked), representation change (whether the returned bytes or validators changed), and business change (whether the extracted fields that matter changed). A 304 can establish that the selected HTTP representation remains valid for a conditional request. It does not prove that a product is still in stock in every market, or that a dynamic client-side API has not changed. Where the HTML shell and data API differ, track the representation that actually carries the required fields.
A useful record contains source URL, normalized entity ID, request context, collection time, response status, validator values, content digest, parser version and accepted fields. Keep the previous accepted record until a new observation passes validation. This makes a failed request visible as an unknown state rather than a fabricated ‘no change’.
2. Use HTTP validators as a conditional check
Under RFC 9110 HTTP Semantics, a server may send an ETag or Last-Modified validator with a 200 response. On the next GET, send If-None-Match for a stored ETag, or If-Modified-Since for a stored modification date. If the selected representation has not changed, a 304 Not Modified response tells the client to reuse its stored representation; it has no new response body to parse. If it has changed, the server can return a new 200 response and representation. When both conditional headers are sent, If-None-Match takes precedence, so the example below deliberately chooses one.
An ETag can be weak or strong. It is an origin-provided validator, not a semantic diff of the fields you care about. Last-Modified usually has coarser time precision and may miss rapid changes; a source may omit either header or implement them inconsistently. Test behavior against a small sample before assuming it saves bandwidth. A CDN or intermediary can also participate in revalidation, so capture response headers and the observed route when diagnosing unexpected 304s.
HTTP caching rules, including Vary, explain why a stored response is reusable only for a matching request variant. Do not transfer an ETag observed in one locale or session to another and interpret its 304 as a global no-change result. Content negotiation and personalized responses can make the same URL represent different content.
| Response | Safe interpretation | State action | Do not assume |
|---|---|---|---|
| 304 Not Modified | The selected representation validates against this stored baseline | Keep baseline; update check time and relevant metadata | All markets or hidden API data are unchanged |
| 200 OK | A representation was returned | Validate fields; compare digest and business fields; then replace baseline | Any byte difference is a business change |
| 404 / 410 | Resource unavailable or reported gone in this context | Flag for same-context confirmation and source review | One response proves permanent deletion |
| 429 Too Many Requests | Source has applied a rate limit | Pause and honor Retry-After where provided | Changing IPs authorizes continued requests |
3. Partition baselines by market and session context
The cache key for a collection job is often wider than the URL. Include the target path and query, market or exit region, language, relevant request headers, access scope and any session state that changes the selected representation. Respect the source’s Vary header, but do not rely on it to document every application-level difference. A currency switch, delivery postcode, logged-in view or consent state may alter a page even when the path is identical. Treat these as distinct sampling contexts and describe them in the output.
For permitted regional observations, Arcvenza Residential Proxies provide location and session controls that can help keep the network route aligned with a market-specific sample. Choose and verify the observed exit location, then hold the route and application context consistent for a comparable series. A proxy changes the route; it does not guarantee that the target will return the intended locale, and it does not supply conditional-request logic by itself.
Store validators under the full context key. If a session expires, an exit market changes, or a site begins returning a different variant, establish a new baseline instead of reusing the old hash. Check the actual response language, currency and entity ID before accepting a 304 or 200 into a market series. This is especially important where a site redirects by geography or sets market cookies on the first visit.
4. Implement a small conditional-fetch loop
The following Python Requests example shows the decision point for one permitted URL. It intentionally uses an in-memory state dictionary to make the state transition visible; production code needs durable, atomic storage, a scheduling layer, structured logs, and a source-specific parser. Set PROXY_URL only when a proxy is part of the authorized collection design. The example disables ambient environment proxy settings so the route is explicit, sets connect/read timeouts, and refuses redirects until the destination can be reviewed.
import hashlib
import os
import requests
URL = "https://example.com/" # Replace with a permitted source URL
CONTEXT = ("US", "en-US", "public") # Market, language, access scope
state = {} # Teaching demo; use durable storage in production
key = (URL, *CONTEXT)
proxy_url = os.getenv("PROXY_URL") # Optional; keep secrets out of logs
proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else {}
def fetch_once(session):
old = state.get(key, {})
headers = {"Accept-Language": CONTEXT[1]}
if old.get("etag"):
headers["If-None-Match"] = old["etag"]
elif old.get("last_modified"):
headers["If-Modified-Since"] = old["last_modified"]
response = session.get(
URL, headers=headers, proxies=proxies,
timeout=(5, 20), allow_redirects=False,
)
if response.status_code == 304:
if "body" not in old:
raise RuntimeError("304 without a stored representation")
print("unchanged; retained bytes:", len(old["body"]))
elif response.status_code == 200:
body = response.content
digest = hashlib.sha256(body).hexdigest()
print("changed:", digest != old.get("sha256"))
# Validate parsed fields before saving a production baseline.
state[key] = {
"body": body, "sha256": digest,
"etag": response.headers.get("ETag"),
"last_modified": response.headers.get("Last-Modified"),
}
elif response.status_code == 429:
raise RuntimeError("Rate limited: honor Retry-After and pause")
elif response.status_code in (404, 410):
raise RuntimeError("Possible removal: confirm in the same context")
else:
response.raise_for_status()
raise RuntimeError(f"Unexpected status: {response.status_code}")
with requests.Session() as session:
session.trust_env = False
fetch_once(session) # First observation
fetch_once(session) # Teaching demo; schedule future checks responsibly
The Requests advanced guide documents Session behavior and proxy configuration. This example prefers ETag and falls back to Last-Modified; it does not send both. It also refuses to interpret a 304 without a stored baseline. A 200 is hashed only after a complete response, and the baseline should be persisted only after parsing and quality checks succeed. In production, handle request exceptions and controlled retries, refresh metadata carried by 304 responses, and put secrets in a secure store rather than source code or logs.
Do not use the raw-byte digest as the only business signal. An advertising slot, timestamp or A/B marker can alter HTML while the measured price remains the same. Compare normalized, documented fields after parsing, preserve both raw and canonical digests if useful, and keep the parser version with the output. If a source supplies a documented change feed or API, use it in preference to guessing from page HTML.
5. Reconcile changes, gaps and apparent deletions
A changed 200 response should pass a schema check before it replaces accepted data: expected entity ID, required fields, currency or units, plausible field types and source timestamp. Store the raw observation for audit, then compute a field-level diff. A missing field is not automatically a true removal; it might be a consent page, incomplete render, redirect, geo mismatch or parser regression. Flag anomalies for inspection and keep the last valid observation labeled with its age.
Treat 404 and 410 as evidence about a particular request, not an instant deletion command. Recheck with the same market, language and access context at a bounded later time; examine redirects and any official listing or API status. If the item truly disappears, record a tombstone with the last-seen time and confirmation evidence. For a catalog, reconcile the item set as well: an unchanged category page may still conceal changes on individual detail pages, while a reordered list can create duplicate observations.
For the broader measurement design—how to separate real page updates from market noise—see Proxies for Website Change Monitoring. For multi-page collection, separately verify pagination completeness and continuation signals. Incremental fetching improves an already valid crawl; it cannot repair an incomplete page inventory.
6. Operate within source limits and measure the result
Before scheduling, check the publisher’s terms, authorization and Robots Exclusion Protocol rules for the relevant paths. Robots instructions are a crawl-control signal, not permission to access data. Use an identified client where appropriate, bounded concurrency, backoff, and a stop rule for 403 or 429 responses. Honor Retry-After when supplied. Changing network routes must not be used to ignore a source’s refusal or rate limit.
Track coverage (accepted entities versus expected inventory), age of last valid observation, 304 share, bytes transferred per accepted update, parser rejection rate, 429 rate, and market-context mismatch. Segment those metrics by source and context; a high 304 share can mean efficient revalidation or a broken, overly broad cache key. Run periodic full checks of a sampled subset to test whether validators and field extraction still agree with reality. Record the sampling method and exceptions rather than advertising an unsupported success rate.
Arcvenza Crawler Console is positioned for configuring, launching and monitoring crawls. It can support operational visibility while the collector still owns baseline storage, conditional requests, parser tests and the meaning of a change. Start with a small authorized source sample, prove the baseline and context keys, then expand only when freshness and quality metrics remain stable.
In practice, a reliable incremental pipeline is a disciplined state machine: observe a permitted representation, validate it, compare it in the right context, and publish a changed record only when evidence supports it. HTTP 304 reduces avoidable transfer; it is not a substitute for complete coverage, correct parsing or responsible source access.
Official Standards & Technical Documentation
IETF — HTTP Semantics (RFC 9110)
IETF — HTTP Caching (RFC 9111)

