How to Validate Scraped Data: Schema, Coverage, and Freshness
A practical acceptance framework for web data: validate record structure and meaning, reconcile source coverage, measure freshness, and preserve evidence before publishing.

A successful HTTP response is an observation, not a trustworthy dataset. A crawler can receive 200 OK from every requested URL and still collect a consent screen, the wrong market, an incomplete last page, duplicate records, or prices parsed in the wrong currency. Data quality must be decided after access and extraction, at the point where observations become accepted records.
TL;DR: Define an acceptance contract for each source and market. Check every record’s schema and business meaning, reconcile the accepted IDs against a defensible inventory, measure freshness from real observation times, and retain enough provenance to reproduce a disputed value. Quarantine failures instead of publishing them as zeros or deletions. Proxies and crawler tools help obtain and track observations; your pipeline owns the decision to release them.
1. Define the unit of truth and the acceptance contract
Write down the intended answer before designing a validator. For a public product catalog, the unit might be an offer identified by product, seller, variant, delivery market and currency. For a job board, it might be a posting and its employer, location and status. A URL alone is often not the entity key: one item can have multiple URLs, and one URL can show different offers by region or session.
An acceptance contract names the permitted source paths, stable entity key, required fields, field types, valid ranges, target markets and languages, expected observation window, and treatment of missing values. It also states what “complete” means for that run. If the source supplies a documented API or export, prefer its field definitions and pagination contract. For an HTML source, record the selectors or parser version and verify a small manual sample against the rendered source.
Separate observation status from business value. A blocked request, empty parser result, and true zero price are different states. Store them differently. An unavailable item is not necessarily a deleted item; a translated price is not necessarily comparable to the original currency. Make these distinctions part of the contract so downstream analysts do not have to infer them from nulls.
2. Validate record shape, then validate meaning
JSON Schema 2020-12 gives teams a versioned way to express required properties, types, ranges and patterns. Its job is narrower than truth verification. A record can satisfy a schema while still referring to the wrong product or market. Follow structural checks with domain checks: currency agrees with the storefront, an identifier belongs to the observed entity, a stock label has a documented interpretation, and a price is plausible for the source’s unit conventions.
The table is a practical triage guide. The numbers in a real release gate must come from your source contract and error budget, not from a universal “good data” percentage.
| Gate | Evidence to retain | Typical failure | Action |
|---|---|---|---|
| Transport and context | URL, status, final URL, market, language, time | 200 is a challenge or wrong-market page | Quarantine; inspect route and page identity |
| Record schema | Schema version, required fields, types, ranges | Missing ID, negative price, malformed timestamp | Reject record with field-level reason |
| Business semantics | Source label, units, currency, entity match | Price from a different variant or shipping zone | Review parser or context; do not coerce silently |
| Dataset coverage | Expected inventory basis, accepted IDs, duplicates | Last page missing or IDs repeated | Mark run incomplete; investigate pagination |
| Freshness and lineage | Observed time, source update time, response hash, parser version | Old cached result or untraceable value | Hold release or label as stale |
This compact Python example checks a small set of already extracted records. Install jsonschema[format] if you use the date-time format check; the library’s validation documentation notes that format validation is optional and some formats need extra dependencies. The example’s inventory and records are hypothetical, not a measured Arcvenza result.
from jsonschema import Draft202012Validator, FormatChecker
SCHEMA = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["id", "url", "market", "price_minor", "currency", "observed_at"],
"properties": {
"id": {"type": "string", "minLength": 1},
"url": {"type": "string", "format": "uri"},
"market": {"const": "US"},
"price_minor": {"type": "integer", "minimum": 0},
"currency": {"const": "USD"},
"observed_at": {"type": "string", "format": "date-time"},
},
"additionalProperties": False,
}
records = [
{"id": "A-1", "url": "https://example.com/a-1", "market": "US",
"price_minor": 12999, "currency": "USD", "observed_at": "2026-11-16T10:00:00Z"},
{"id": "A-2", "url": "https://example.com/a-2", "market": "US",
"price_minor": -1, "currency": "USD", "observed_at": "2026-11-16T10:00:00Z"},
]
expected_ids = {"A-1", "A-2", "A-3"} # Same-source, same-market inventory
Draft202012Validator.check_schema(SCHEMA)
validator = Draft202012Validator(SCHEMA, format_checker=FormatChecker())
accepted, rejected, seen = [], [], set()
for record in records:
errors = sorted(validator.iter_errors(record), key=lambda e: str(e.path))
if errors:
rejected.append((record.get("id"), [e.message for e in errors]))
continue
if record["id"] in seen:
rejected.append((record["id"], ["duplicate ID in this run"]))
continue
seen.add(record["id"])
accepted.append(record)
accepted_ids = {r["id"] for r in accepted}
coverage = len(accepted_ids & expected_ids) / len(expected_ids)
missing_ids = sorted(expected_ids - accepted_ids)
print({"accepted": len(accepted), "rejected": rejected,
"missing_ids": missing_ids, "coverage": coverage})
The example rejects a negative price, identifies a missing expected ID, and reports coverage; it does not decide whether the remaining record is factually correct or whether the run should be published. For monetary calculations, keep explicit currency and minor units or an appropriate decimal representation. Use RFC 3339 timestamps with offsets so “recent” can be calculated consistently across regions.
3. Reconcile coverage without inventing completeness
Coverage is a property of the accepted dataset, not the number of HTTP requests. Let (E) be a defensible set of expected entity IDs for the same source, query, market and observation window; let (A) be accepted unique IDs. A simple coverage measure is (|A ∩ E| / |E|). It is meaningful only if the expected set is documented and nonempty. A current sitemap, official API count, licensed export or separately verified sample can help form that denominator, but each has its own limitations.
For a live catalog, a displayed total may change while crawling. Do not call 98 of 100 pages “98% complete” without checking whether pages carry equal numbers of records, whether items moved between pages, and whether the total was captured under the same filters. Track page or cursor continuation, unique IDs, duplicate rate and the explicit terminal signal. A 200 response to the last requested page is not proof that the final page was reached. A parser returning zero rows may signal a changed layout rather than an empty catalog.
Segment coverage by market, category, source and run. Global averages can conceal a missing region or a broken category. Keep a missing-ID list and a reason for each exclusion; if the denominator is unknown, report “coverage not established” alongside observed counts instead of assigning a false percentage. The related e-commerce collection guide explains why market and currency context need to travel with each observation.
4. Measure freshness as a property of accepted evidence
Record at least three times when available: when your system requested the page, when it accepted a record, and when the publisher says the underlying item changed. These are different clocks. A newly fetched page can contain an old price; a source-provided updated_at can lag the visible page; a CDN can return a representation that is valid by its caching rules but too old for your business requirement. State the freshness target in the contract, such as an observation age for an agreed source and market, and compute it from the last valid accepted observation, not the last attempted request.
Use timezone-aware timestamps and retain the original offset. RFC 3339 defines an Internet timestamp representation; normalizing to UTC simplifies comparisons, but preserve the source string for audit. Watch for clock skew, source time zones, and timestamp fields that actually mean publication rather than last modification. If an item disappears, do not set its update time to the failed request time. Mark it as last seen and pending confirmation.
Freshness needs context. A weekly government release and a rapidly changing retail inventory require different schedules and alerts. Sample a subset with independent full fetches or manual checks to test whether “unchanged” and “fresh” labels still match source reality. Treat a failed check as a quality incident. It should age the last trusted record, not reset its age.
5. Preserve provenance so anomalies can be reproduced
For every accepted value, keep a lineage from published field back to observation: canonical source URL, final URL after permitted redirects, UTC observation time, market and language, status, request context, raw response or durable pointer, response digest, parser and schema versions, and validation outcome. Store only data you are permitted to retain, with access controls appropriate to the source. A hash helps prove that a saved artifact has not changed; it does not prove that the source was truthful or that extraction was correct.
Run IDs and reason codes connect records to their collection attempt. For example, wrong_market, missing_required_field, duplicate_id, challenge_page, and stale_baseline tell operators what to investigate without hiding failure behind a generic “scrape failed.” Keep a small reproducible sample of both accepted and rejected pages for parser regression tests. When a parser version changes, compare its output on the same saved artifacts before promoting it.
Provenance also prevents false causal claims. A residential exit may help observe a market-specific public page, but it does not establish that a price is representative of all buyers in that market. Record postal code, currency, selected variant, session and delivery assumptions where they affect the page. Re-run the same context when investigating a surprising change.
6. Put release gates around the whole dataset
Build the release decision from multiple gates. At minimum, reject records that fail structural or semantic validation; hold a run if required source-market slices or continuation checks are incomplete; label or withhold stale records according to the contract; and keep unresolved 403, 429 or challenge responses out of the accepted set. Report each gate’s numerator, denominator, threshold and exceptions. Thresholds must be chosen for the business decision and tested against actual source behavior, not copied from a blog.
Finally, obey the publisher’s terms, authorization and applicable Robots Exclusion Protocol rules; robots controls are not a grant of access. Use bounded concurrency and stop or back off on refusal or rate limiting. An honest dataset includes its unknowns. Publishing fewer records with clear evidence is better than manufacturing apparent completeness from successful requests.
Official Standards & Technical Documentation
JSON Schema — Specification, Draft 2020-12
Python `jsonschema` — Schema Validation and Format Checking
IETF — Date and Time on the Internet: Timestamps (RFC 3339)
IETF — Robots Exclusion Protocol (RFC 9309)

