The error that slips into scraped datasets most quietly is overwriting. Each run replaces yesterday's price, status or description with today's, the table always looks current and clean, and the history that analysts later ask for, such as when a price dropped or how long an item stayed listed, is gone. By the end of this tutorial you will have three layers that prevent this: raw snapshots of every run, a current-state table, and a change log in which every version of a record carries the dates it was valid. You will also have two rules that keep the history honest: one for items that disappear and one for changes caused by your own extractor rather than by the source.
Step 1: Decide which fields need history
Not every field deserves a history. Go through the schema and classify each field into one of three groups:
- Tracked: changes matter to someone's question, such as price, availability, status or rank.
- Overwritten: only the latest value matters, such as a corrected spelling in a title, or an image address.
- Ignored for change detection: values that change on every visit without meaning anything, such as a "viewed 3 minutes ago" counter or a session parameter in a link.
Check: for each tracked field, write the question it answers, such as "how often does this product go on sale?". A tracked field with no question attached is a candidate for the overwritten group.
Common mistake: tracking everything. If volatile fields are tracked, nearly every record "changes" on every run, and the change log becomes too large and noisy to use.
Step 2: Store each run as an append-only snapshot
Before any comparison, write the extracted records of each run to a snapshot store that is never updated in place. Every record gets the run identifier, the fetch time and the source address. If storage allows, keep the raw page bodies for a limited period as well, so a doubtful value can be re-extracted later.
Snapshots are the safety net for every later step: if the change logic has a bug, the history can be rebuilt from them. Set a retention period that matches the project's needs and any obligations that apply, particularly if records contain personal data, and delete older snapshots on schedule once the change log is verified.
Partition the snapshots by run date or by source, so that reading one source's last few runs does not mean scanning everything ever collected, and so that expired snapshots can be removed as whole partitions rather than record by record.
Check: pick one record and confirm you can list every value it had across the last five runs from the snapshots alone.
Step 3: Compare each run with the current state by stable key
Keep a current-state table with one row per item, keyed by a stable identifier: the source's own item identifier where it exists, otherwise a normalized address. After each run, match the new records to the current rows by that key.
To detect changes efficiently, compute a fingerprint of the tracked fields only, after normalization: trimmed text, prices as decimals, a consistent currency code and a consistent date format. If the fingerprint of the new record differs from the stored one, a tracked field changed. Overwritten fields are simply updated without creating history.
Check: run the comparison twice on the same snapshot. The second pass should report zero changes. Any change it reports comes from unstable normalization, such as inconsistent whitespace or number formatting, and needs fixing before the history is trusted.
Stable keys and normalization are also what make deduplication work, a point covered in the guide to deduplicating scraped records.
Step 4: Write versions with valid-from and valid-to dates
For each detected change, close the current version and open a new one. Each version row holds the item key, the tracked values, a valid-from time and a valid-to time. The open version has an empty valid-to. When a change arrives, set the open version's valid-to to the fetch time of the run that saw the change, and insert the new values with that same time as valid-from.
Use the fetch time, not the time your pipeline processed the run, since processing can lag by hours. And be precise about what the dates mean: a change was observed between the last run that saw the old value and the first run that saw the new one. The true change time lies somewhere in that window, and analysts should know how wide it is.
With the versions in place, the current-state question and the history question use the same table: the open versions give today's values, and filtering by a date gives the values as they stood on that date. Analysts asking "what was the price on the first of the month?" no longer need a separate export.
Check: for any item, the versions should form an unbroken chain, each valid-to equal to the next valid-from, with exactly one open version. Gaps or overlaps mean a run was processed out of order or twice.
Step 5: Treat disappearance carefully
An item missing from one run has not necessarily been removed. The page may have failed to load, the listing may have returned partial results, or pagination may have stopped early. Marking items as removed on a single absence fills the history with false removals and false reappearances.
Set a rule: an item is marked removed only after it has been absent from a set number of consecutive successful runs, for example two or three, and a run counts as successful only if it passed the run-level checks, such as a normal record count for that source. When the rule triggers, close the open version and record a removal, dated to the first run in which the item was missing.
Check: count reappearances, items marked removed that later came back. A few are normal, since sources do relist items. A high count means the absence threshold is too low or the run checks are too lenient.
Step 6: Separate extractor changes from real changes
When you fix or rebuild an extractor, values can change for every item at once: a price now includes cents that were dropped before, or a status is read from a different element. Those are changes in your measurement, not in the source, and if they enter the history as ordinary changes they look like a sudden event in the data.
Record the extractor version on every snapshot and version row. When the extractor version changes, run the comparison in a special mode: changes found on the first run after the new version are written as corrections, not as new versions with new dates, or are flagged so analysts can filter them out.
Check: monitor the share of items that change per run. A sudden spike on the day of a deploy points to an extractor effect; a spike with no deploy points to a real event at the source, or to a source layout change worth investigating. The data quality checks to run on every scraped dataset discuss similar run-level signals.
Step 7: Walk one item through the whole process
Before relying on the history, trace a hypothetical item through every rule. Suppose a product is collected daily with a two-run absence threshold:
| Day | Observed | Result in the change log |
|---|---|---|
| 1 | Price 20.00, in stock | Version 1 opens, valid from day 1 |
| 2 | Price 18.00, in stock | Version 1 closes on day 2; version 2 opens |
| 3 | Price 18.00, in stock | No change |
| 4 | Missing | No change yet; first absence |
| 5 | Missing | Version 2 closes; removal recorded, dated day 4 |
| 6 | Price 18.00, in stock | Reappearance; version 3 opens, valid from day 6 |
Run this sequence through the pipeline using saved snapshots and compare the result with the table. Then repeat it with a twist: make day 4 a failed run with a low record count. The item should not register as absent that day, and day 5 alone should not trigger a removal.
Whether the collection is run in-house or delivered by a provider such as ScrapingIsNotACrime, the same walk-through works as an acceptance test for history handling, provided the delivery includes fetch times and stable keys.
