A scraped dataset can pass every validation rule and still be missing a third of what it should contain. Records that were never collected do not show up as errors, nulls or duplicates; they simply are not there. Measuring coverage means comparing what you collected with an independent estimate of what exists, and doing it per segment, so that a gap in one region or category is not hidden by a full count elsewhere. The steps below build that measurement from a definition of the target population to a coverage statement that users of the data can read. Each step says what its check catches and what it misses.
Step 1: Define the population the dataset is meant to cover
Coverage is always coverage of something. Write one sentence that names the population: for example, "all job postings published on the career pages of the 120 employers in our source list, live at any point during the month". The sentence fixes the sources, the type of record and the time window.
Without it, any coverage figure is ambiguous. A dataset of 9,000 postings might be complete for the source list and very incomplete for the regional job market, and both readings can be true. The check at this stage is simple: show the sentence to the people who will use the data and ask whether it matches what they assume. Disagreement here is the cheapest kind to fix.
Write down what falls outside the population as well. Postings removed before the first run of the month, duplicate postings for the same role in several cities, and internal-only postings are typical edge cases. Deciding them now prevents the coverage figure from shifting later because someone changed what counts.
Step 2: Find a reference count for each source
A reference count is a number you did not produce with your own scraper. Useful places to find one include:
- the result count a site displays on its own listing or search page, such as "1,240 results";
- the number of matching URLs in the site's XML sitemap;
- the totals in an official API, feed or bulk download for the same records;
- an official register or published statistic, when the population is defined by one.
Record each reference with its date and how it was obtained. A displayed result count is easy to read, but it may be rounded, capped or include items that are not shown, so treat it as an estimate. This step catches large, obvious gaps. It misses gaps that the reference shares with you: if the sitemap omits a section of the site, so will both counts.
The mistake that usually breaks this step is comparing counts taken at different moments. A listing page read on Monday and a scrape that finished on Wednesday describe two different populations on a site where items come and go daily. Read the reference as close as possible to the end of the run, or read it twice, before and after, and use the range.
Step 3: Compare by segment, not just in total
Divide your record count by the reference count for each source, and then for each segment that matters to users, such as category, region or month. Totals hide offsetting errors. Suppose a hypothetical dataset of property listings reaches 97 percent of the reference overall, but the breakdown shows one city at 62 percent while duplicates inflate another to 108 percent. The total looks healthy; neither city is.
A ratio above 100 percent is as informative as one below. It usually means duplicates, records outside the time window, or a reference that undercounts. Investigate both directions. This step catches uneven gaps between segments. It misses gaps inside a segment, where the records you have and the ones you lack are spread evenly.
When a segment comes out low, start with the simplest explanations. Check whether the scraper visited every listing page for that segment, whether its records failed validation and were dropped further down the pipeline, and whether the segment labels in your data match the ones the site uses. A city spelled two ways splits one segment into two, and each half looks incomplete.
Step 4: Check a list of known items
Build a short list of records you know should be in the dataset, collected by hand from the live sources: perhaps five per source, chosen to include unusual cases such as items on the last page of a listing, items in rarely used categories and items with long or unusual titles. After each run, look each one up.
A known-item check is small, but it finds the failures that counts cannot, such as a pagination loop that stops one page early or a category that the scraper never visits. Keep the list current; an item that has since been removed from the site is not a miss. The check misses failures that happen to spare your chosen items, which is why the list should be refreshed with new items regularly. Note the date each item was confirmed live, and replace any item older than a few runs, so the list keeps testing the site as it is now rather than as it was when the list was made.
Step 5: Use a second, independent source to estimate what is missing
When no reference count exists, two independent collections of the same population can estimate its size. The idea comes from ecology, where it is called capture and recapture. If source A finds a set of records, source B finds another, and you can match records between them, the overlap tells you how much each one misses.
A hypothetical example: your scraper finds 800 events in a city for a given month. A second, independent list, such as an official listings feed, contains 300 events for the same month, and 240 of those 300 are also in your data. Your scraper found 80 percent of the second list, which suggests it finds roughly 80 percent of all events. The standard estimate of the total is 800 multiplied by 300, divided by 240, which gives 1,000, so your coverage is around 80 percent.
The estimate depends on assumptions that rarely hold perfectly. The two sources must be independent, so that being in one does not make a record more or less likely to be in the other, and matching must be accurate. If both sources favour large, well-publicized events, the estimate will be too high in coverage and too low in total. Treat the result as an order of magnitude and as a way to compare runs, not as an exact figure.
Step 6: Track coverage over time and set thresholds
Store each run's coverage figures by source and segment, next to the run date. A coverage series is far more useful than a single number, because a sudden drop points to a change on a site or in the scraper, while a steady level tells users what they can rely on.
Set a threshold for each source based on its normal range, and decide in advance what happens when a run falls below it: hold the data, deliver it with a warning, or rerun after a fix. Suppose a hypothetical source normally sits between 92 and 96 percent; a run at 81 percent should stop the delivery until someone has looked at it. The check catches sudden breaks. It misses slow erosion, so review the trend each quarter as well as the per-run figure.
Step 7: Write a coverage statement for the people who use the data
Publish the results where users will see them, as a short statement with each delivery or in the dataset documentation. A good statement gives the population sentence from Step 1, the coverage estimate by source and main segment, the method used for each estimate, the known blind spots and the date of the last known-item check.
A statement such as "estimated 85 to 90 percent of postings in the source list; lower for employers whose career pages load listings in batches; independent comparison last run in March" lets an analyst decide whether a small regional trend is real or an artefact of missing data. When data comes from a provider, such as ScrapingIsNotACrime or another service, ask for the same figures and methods with each delivery, so the statement can be written without guessing. The data quality checks to run on every scraped dataset cover the accuracy checks that sit alongside coverage.