The hardest error to catch in scraped data is a value that is still valid but no longer means what it used to. A price field that starts including fees, a date that switches from listing date to last update, a rating that moves to a different scale: each passes every schema, null and range check, because the number is still a number in a sensible range. This walkthrough follows a hypothetical team through one such change, from the first odd result to the fixes, to show which checks find this kind of drift and which ones cannot.
The hypothetical project: a monthly rent index
Let us follow a hypothetical housing research group that publishes a monthly median asking rent for one-bedroom apartments in a mid-sized city. It collects public listing pages from 15 property management websites, after checking each site's robots.txt and terms, at a slow pace once a week. Each record holds the listing identifier, the asking rent, the number of bedrooms, the floor area, the neighborhood, the date the listing became available, the source and the fetch time.
The pipeline already runs the checks most teams start with: required fields present, rent between a plausible minimum and maximum, bedrooms between zero and five, null rates per field within their usual levels, and duplicate listings merged. Every run in the first six months passes.
Month seven: an increase nobody can explain
In month seven, the median asking rent rises by about 6 percent against the previous month, after months of changes below 1 percent. Record counts are normal, no field has more nulls than usual, and no value falls outside its range. The dataset-level drift check, which compares the overall distribution of rents with the previous month, flags the jump, but it cannot say whether the market moved or the data did.
The team's first decision is the right one: hold the publication until the cause is known. A housing index that reports a 6 percent jump will be quoted, and a correction a month later reaches fewer people than the original number.
Step 1: break the change down by source
A city-wide median mixes 15 sources, so the first question is whether all of them moved or only some. The team computes the median rent per source for the last three months and finds that 14 sources changed by between minus 1 and plus 2 percent. One source, which supplies roughly a fifth of the listings, shows a rise of about 18 percent in a single month.
A market shift rarely hits one landlord's listings so hard while leaving the rest untouched. A change of that size in one source is far more likely to come from the source's pages than from its prices.
What this check catches: any change concentrated in one source. What it misses: a change that affects many sources at once, such as a new legal requirement on how rents are advertised.
Step 2: compare the raw pages
The team keeps raw responses for a sample of pages each run, so it can open a listing page saved in month six and one from month seven for the same source. The structure looks almost identical. The large figure at the top of the listing is still in the same element, with the same attributes, so the extractor reads it without trouble.
The difference is in the small label next to it. In month six it said "Monthly rent". In month seven it says "Total monthly cost", and a new section further down the page breaks that total into base rent, a mandatory amenities fee and a utilities package. The extractor has been collecting a different quantity under the same field name for a month.
While checking this source, the team also notices that its "available from" date now changes every time a listing is refreshed, so long-running listings look new. That second change does not affect the rent index, but it would distort any measure of how long apartments stay on the market.
Step 3: decide what the index should measure
Fixing the extractor requires a decision first: does the index measure base rent or total monthly cost? Total cost is arguably more useful to renters, but most of the other 14 sources show base rent, and some do not publish fees at all. Mixing the two definitions would make the index depend on which sources happen to show which number.
The team keeps base rent as the index definition, updates the extractor for the changed source to read the base rent line from the breakdown, and adds the total cost as a separate field where sources provide it. The definition is written into the project's documentation, along with the date each source's extraction changed.
Step 4: repair the history
One month of records from the changed source holds total cost instead of base rent. Because raw pages were saved only for a sample, the team cannot re-extract every affected listing. It has three options:
- re-collect the listings that are still online, at the usual pace, and re-extract them;
- exclude that source from month seven and publish the index with a note;
- estimate base rent from total cost, which would introduce a modeled number into an index of observed values.
It combines the first two: listings still online are re-collected and corrected, and those that disappeared are left out of month seven, with the number of excluded listings stated in the methodology note. The corrected median rises by less than 1 percent, in line with previous months.
Step 5: add checks that would catch it next time
The incident shows that value checks alone cannot detect a change of meaning. The team adds four checks aimed at the meaning of fields:
- Label capture. For each key field, the extractor stores the visible label next to the value, and a run fails for a source when the label text changes from its expected wording. This is the check that would have caught month seven on the first day.
- Per-source distribution baselines. Each source's median and spread are compared with its own recent history, with an alert on a step change above a set threshold, rather than relying on the city-wide figure.
- Reference listings. A small set of long-running listings per source is followed every run. When their values jump while the listings themselves have not changed, the extraction is the first suspect.
- Field stability checks. For fields that should not change for a given listing, such as the available-from date or the floor area, the team counts how often they change between runs and alerts when that rate rises.
These checks have limits. Label capture misses a change of meaning that keeps the old label, and baselines can be fooled by a gradual change over several months. They narrow the gap without closing it, which is why the methodology note also invites readers to report anything that looks wrong. The broader set of data quality checks to run on every scraped dataset covers the structural side.
What to take from the rent index case
Three lessons carry over to most scraped datasets. First, monitor each source against its own history, because a change in one source disappears inside an aggregate until it is large enough to move the total. Second, capture the context of a value, its label and unit, not just the value, since the context is where a change of meaning shows. Third, keep raw pages for a sample on every run and write down each field's definition, because both are needed to decide how to repair the history once drift is found.
The same expectations apply to data that comes from outside. If a provider such as ScrapingIsNotACrime delivers some of the sources, agree in advance on how changes in a site's labels or definitions are reported, and run the per-source baselines on delivered data too.