The most dangerous error in a scraped dataset is the one that does not crash anything. A job finishes, the file has the usual number of rows, and half the prices are empty because a page changed its layout overnight. The nine checks below catch that kind of silent failure. Each one is cheap to automate, and together they turn a raw extraction into data an analyst can trust. Run them after every collection run, before the data reaches a report, a model or a client.
1. Validate every record against a written schema
Start by writing down what a valid record looks like: each field's name, type, whether it is required and what format it uses. A product record might require a source URL, a title as text, a price as a decimal number, a currency as a three-letter code and a fetch time as a timestamp.
Then check every record against that schema and count the failures by field. Schema validation libraries exist for most languages, but even a short function that tests each field is enough to start. What it catches: a price that arrives as the text "Call for price", a date in the wrong format, a missing required field. What it misses: values that have the right type but the wrong content, which is why the next checks exist.
2. Track the null rate of each field, run over run
A field that is empty in 2 percent of records one day and 60 percent the next is almost always a broken extractor, not a change in the world. Compute the share of empty values for each field on every run and compare it to recent runs.
Hypothetically, suppose a field for product ratings is usually empty for about 10 percent of records, because new products have no reviews. Set an alert when it moves more than a few points away from that baseline in either direction. Note that a drop in empty values can also be a bug, for example when the extractor starts picking up a neighboring element and filling every row with the same number.
3. Remove duplicates using a stable key
Scraped data duplicates easily: the same item appears on two listing pages, a retry fetches a page twice, or a URL is collected with and without tracking parameters. Decide on a stable key for each record type, such as a product identifier from the page, a canonical URL or a combination of source and item ID.
Normalize URLs before using them as keys: lowercase the host, remove fragments and drop parameters that only track the visit. Then count duplicates per run. A sudden rise usually points to a pagination problem, such as the same page being fetched repeatedly because the next-page link stopped changing.
When two records share a key but differ in content, do not keep one at random. Keep the one with the later fetch time, or keep both with a flag if the difference matters, such as two prices for the same item. Fetching the same page twice also puts unnecessary load on the source, so a rising duplicate count is a reason to fix the crawl, not just the dataset.
4. Normalize units, currencies, dates and time zones
Values that look comparable often are not. One site writes 1.299,00 for a price, another writes 1,299.00, and a third shows prices including tax. Dates come as 03/04/2026, which means March 4 in one convention and April 3 in another. Engagement counts come as 1.2K or 3M.
Convert everything to one representation at extraction time and keep the original text in a separate field. Store timestamps in UTC with the original time zone noted. A concrete test: pick ten records from each source and confirm by hand that the normalized value matches what the page shows. This single check prevents most comparison errors in price and trend analysis.
Write the conversion rules down per source, for example which decimal separator it uses, whether its prices include tax and which time zone its dates refer to. When a source changes one of these, the written rule is what tells you the conversion needs updating, and the original-text field lets you reprocess past records instead of collecting them again.
5. Check ranges and flag outliers
Every numeric field has a plausible range. A hypothetical bookstore dataset should not contain a paperback priced at 0.00 or at 45,000. A rating field on a five-star scale should not contain 47. Define lower and upper bounds per field, and flag values outside them for review rather than silently dropping them.
Relative checks are useful too. If an item's price changes by more than a set percentage between two runs, flag it. Some of those flags will be real sales; others will be the extractor reading a monthly installment price or a bundle price instead of the item price. Reviewing a short list of flags is much cheaper than explaining a wrong chart later.
Keep the flagged records in a separate review table with a status such as confirmed, corrected or rejected. Over a few weeks, that table shows which sources and fields produce the most false values, which is where extractor work pays off first. It also lets you tighten or loosen bounds based on evidence instead of guesses: if every flag on a field turns out to be a genuine value, the bound is too narrow.
6. Record freshness and reject stale records
Every record should carry the time it was fetched, not just the time the job finished. Fetch time lets you answer two questions: how old is this value, and did this run actually refresh it?
Set a freshness rule that matches the use. Price monitoring might treat anything older than a day as stale; a catalog of company addresses might accept a month. When a source fails for a run, carry the old value forward only if it is marked as such, never as if it were new. Mixing fresh and stale values without a marker is one of the most common reasons two reports built from the same dataset disagree.
7. Keep provenance for every value
Provenance is the record of where a value came from: the source URL, the fetch time, the extractor version and, ideally, a stored copy or hash of the raw page. With it, any odd value can be traced back and explained. Without it, the only answer to "where did this number come from?" is a guess.
Provenance also supports the legal and ethical side of a project. It shows which sources were used and when, which makes it possible to honor a site's request to stop, or to delete data from a source whose terms changed. Keep provenance for records that contain personal data with the same care as the data itself, and apply the same retention limits. The overview of ethical and legal web data extraction in the privacy era explains why that traceability matters.
8. Spot-check a sample against the live source
Automated checks only test what you thought to test. A manual spot check catches the rest. After each significant run, pick a small random sample, for example 20 records spread across sources, open the source pages and compare field by field.
Record the result as a simple accuracy rate per field. If titles match 20 out of 20 but prices match 16 out of 20, investigate the four misses before using the run. Typical findings include a sale price captured instead of the regular price, a variant page showing a different size than the default, or text truncated by the page layout.
9. Monitor drift in counts and distributions
The last check looks at the dataset as a whole. Compare each run with the previous ones: number of records per source, number of distinct values in category fields, average and median of key numbers. Sudden shifts point to structural problems that record-level checks miss.
Suppose, hypothetically, that in one run a source returns 2,000 records instead of the usual 8,000. Every record may pass the schema, but the source has likely changed its pagination or started serving a shorter page. A gentle response is the right one: a site that returns less content than usual may be limiting traffic, so slow down and review rather than retry harder. Drift alerts are also where outsourced collection needs scrutiny. If a managed service such as ScrapingIsNotACrime runs part of your collection, apply the same drift checks to what it delivers, so quality is measured the same way across every source.
