By the end of this tutorial you will have a folder of saved sample pages, a hand-written expected result for each one, and a test run that tells you within seconds whether a change to your extractor broke a field. The test set runs entirely offline, so you can change selectors as often as you like without sending a single extra request to the site you collect from. The idea is simple: separate fetching from parsing, freeze a few real pages, and compare what the parser extracts against what a person reading those pages would write down.
Step 1: Choose sample pages that cover the variation
A test set is only as good as the cases it contains. Ten copies of the same ordinary product page will pass every time and catch nothing. Pick pages that differ in the ways that break extractors: a normal page, one with a missing field, one with a sale price next to a regular price, one marked out of stock, one with very long text, one with accented or non-Latin characters, and one built on a different template if the site uses more than one.
For a typical project, 10 to 20 pages per source is enough to start. Write a one-line reason next to each, such as "sale price shown with the old price crossed out" or "no reviews section at all".
Check: read your list of reasons. If two pages have the same reason, drop one. If you know of a variation the list does not cover, find a page for it.
If a source serves different layouts by region or language, include at least one page per layout you actually collect. A test set drawn only from the English version of a site will not warn you when the German product pages move their price into a different element.
Common mistake: choosing only pages where the scraper already works. Include the page that broke last month; that is the one most likely to break again.
Step 2: Save each page exactly as the scraper receives it
Save the raw response body that your fetcher gets, not the page as a browser shows it. A browser runs scripts and rebuilds the document, so a copy saved from the browser can contain elements that never appear in the response your scraper parses. Fetch each sample once, at your usual polite pace, with the same descriptive user agent the scraper uses, and store the body as a file.
Next to each file, keep a small record of the URL, the fetch date, the status code and the content type with its character set. The character set matters: a page saved as one encoding and read back as another turns accented letters into garbage and makes text tests fail for the wrong reason.
Avoid pages that show personal information, such as reviewer names or profile details. If a page you need contains some, replace those values in the saved copy with obvious placeholders. Test files get copied around, shared and kept for years, so they should hold nothing you would not want in a code repository.
Check: open one saved file in a plain text editor and search for a value you can see on the live page, such as the price. If you cannot find it, the value is added later by a script, and your extractor needs a different source for it.
Step 3: Write the expected output by hand
For each saved page, write down what every field should contain, reading the page as a person would: the title, the price as a number, the currency, the availability, and so on. Use the same field names and types your scraper produces, and write an explicit empty value where a field is genuinely absent.
Do not generate the expected output by running the current extractor and copying the result. That turns today's bugs into tomorrow's correct answers, and the test will defend the bug.
Check: have a second person, or yourself a day later, compare three expected files against the saved pages. Any disagreement about what a field should contain is a specification question to settle now, before it becomes a silent inconsistency in the data.
Step 4: Run the extractor against the files, with no network
Make the parsing step accept a page body as input, instead of fetching the URL itself. The fetcher hands the body to the parser in production; the test hands it a saved file. This separation is the single change that makes offline testing possible, and it also makes the parser easier to debug in general.
Then write a test run that loops over the saved pages, passes each one to the parser and collects the results. Keep the run fast: a set of a few dozen pages should finish in seconds, which is what makes it practical to run after every small selector change instead of once a week.
Check: switch off the network connection, or configure the fetcher to fail on any call, and run the tests. They should run to completion. If anything tries to reach the internet, some part of the parser is still fetching on its own, often a follow-up request for prices or images.
Step 5: Compare field by field, with rules per field type
Compare each extracted field with its expected value separately, and report every mismatch as page, field, expected value and extracted value. A report that says only "record differs" sends you back to diffing by eye.
Set the comparison rule by field type:
- Identifiers and URLs: exact match after your usual normalization.
- Free text: match after collapsing repeated spaces and trimming the ends, so a harmless whitespace change does not fail the test.
- Prices and other numbers: compare as decimal numbers, not as text, so "19.90" and "19.9" count as equal.
- Empty fields: an expected empty value must come back empty, not as a zero or an empty string standing in for a missing price.
Check: deliberately break one selector and run the tests. The report should name the right field on every page that uses it. If it reports nothing, the comparison is too loose.
Step 6: Add a test case every time something breaks
When the live scraper fails because the site changed, start the fix from the test set. Save the new version of the page, write its expected output, and run the tests with the old extractor: the new case should fail. Fix the extractor until the new case passes and all the old ones still pass.
Keep the old page in the set as long as the site may still serve the old template, for example on older listings. Once it is gone for good, move it to an archive folder rather than deleting it, with a note on when it stopped appearing.
The same saved pages and expected outputs work as an acceptance check for data delivered by someone else. If a managed provider such as ScrapingIsNotACrime collects part of your sources, comparing a delivered sample against pages you checked by hand is a concrete way to confirm the specification was understood. The data quality checks to run on every scraped dataset apply to delivered data as well.
Common mistake: fixing the live extractor first and adding the test "later". Later rarely comes, and the next change breaks the same field again.
Step 7: Refresh the saved pages against the live site on a schedule
Saved pages go stale. Your tests can pass for months while the live site has moved to a new layout, because the tests only see the old one. Once a month, or after any failed run, fetch the same sample URLs again at your normal pace and run the extractor on the fresh copies without comparing to expected values yet. Look only at whether each field comes back filled.
If a field is empty on the fresh copy but filled on the saved one, the site has changed: save the fresh copy as a new case and go back to Step 3. If a sample URL now returns a not-found or a redirect, replace it with a current page that shows the same variation. Pages that the site no longer allows you to fetch, because of a new robots.txt rule or a change in its terms, should be removed from the sample and from the collection.
As the number of sources grows, this refresh becomes part of the run schedule rather than a manual chore.
Check: record the date of the last refresh for each source in the same place as the saved pages. Any source whose refresh date is older than your chosen interval is a source whose tests no longer prove much.
