Duplicate records are among the errors that slip into a scraped dataset unnoticed, because each copy looks perfectly valid on its own. The same product reached through two category pages, the same listing with and without a tracking parameter, the same article republished under a new URL: each adds a row, and every count, average and trend built on the data drifts upward. Deduplicating scraped records well takes seven steps: define what a duplicate means for your data, pick a key, normalize URLs, normalize the compared fields, remove exact duplicates with a clear rule for which record survives, catch near duplicates with a reviewed similarity check, and monitor the duplicate rate on every run. This tutorial walks through each step with a hypothetical dataset.
Step 1: Define what counts as a duplicate
Before writing any rule, decide what one record represents. There are two common answers, and mixing them up causes most deduplication mistakes.
- Entity records describe one thing: one product, one company, one job listing. Two rows for the same thing are duplicates.
- Observation records describe one thing at one time: the price of a product on a given day. Two rows for the same product on different days are not duplicates; they are history. Two rows for the same product on the same day probably are.
Write the definition down in one sentence, for example: "one row per product per source per collection date." Every later step tests against that sentence.
What this catches: the error of deleting legitimate history, which happens when an observation dataset is deduplicated as if it held entities. What it misses: nothing on its own; it only makes the following steps correct.
Step 2: Choose the key that identifies a record
A key is the field or combination of fields that should be unique under your definition. The best key comes from the source itself: a product ID, a listing number, a SKU or a manufacturer part number shown on the page or in its structured data. Combine it with the source name, because two sites can use the same ID for different things, and with the collection date for observation records.
When the source shows no ID, the page URL is the usual fallback, after the normalization in the next step. A combination of descriptive fields, such as name plus brand plus size, is the last resort, because those fields change more often than IDs and are harder to compare.
Check: count the records in one run and the number of distinct key values. If they are equal, the key has no exact duplicates in that run. If they differ, you have duplicates to examine, or a key that is not as unique as you thought.
Common mistake: using an ID that identifies the page rather than the item. On some sites, one product page holds several variants, each with its own variant ID, while the page carries a single product ID. If your records are per variant, a key built on the page-level ID will merge different sizes or colors into one row. Look at a few product pages with variants and confirm which ID changes when the variant changes before settling on the key. The same check applies to listings that are relisted: some sites issue a new ID each time an item is posted again, which makes a key based on that ID miss the repeat entirely, a case Step 6 addresses.
Step 3: Normalize URLs before using them as keys
The same page can appear under many URLs, and a URL-based key only works after they are reduced to one canonical form. Apply these rules in order:
- lowercase the scheme and host name, and use one scheme, usually https;
- remove the fragment, the part after a hash sign;
- remove tracking and session parameters, such as those beginning with utm_, while keeping parameters that change the content, such as a product variant or page number;
- sort the remaining query parameters so their order does not matter;
- apply one rule for trailing slashes.
Where the page declares a canonical URL in a link element with rel set to canonical, record it and prefer it as the key when it points to the same site. It is the site's own statement of which address is the main one.
Common mistake: removing all query parameters. On many sites, a parameter selects the color or size of a product, and dropping it merges different variants into one record. Build the list of parameters to remove per source, after looking at real URLs.
Step 4: Normalize the fields you compare
Exact comparisons fail on differences a person would never notice. Before comparing any text field, create a normalized version of it: trim leading and trailing whitespace, collapse repeated spaces, replace non-breaking spaces, normalize Unicode so that accented characters have one representation, and convert to lowercase. For numbers, parse them into a numeric type using the right decimal separator for the source. For dates, convert to one time zone and format.
Keep the original values alongside the normalized ones. Normalization is for comparison, and the original text may be needed for display or audit.
Check: after normalization, count distinct values for one text field, such as brand. A drop from, say, 410 distinct brands to 385 in a hypothetical catalog shows how many variants existed only because of spacing and case.
Step 5: Remove exact duplicates with a survivor rule
With a reliable key and normalized fields, exact duplicates are records that share a key. Group them by key and keep one. The important decision is which one: write a survivor rule rather than keeping whichever row happens to come first. Common rules are:
- keep the most recently collected record;
- keep the most complete record, the one with the fewest empty fields;
- keep the record from the most authoritative page, for example the product page rather than a listing page that shows a summary.
When duplicates disagree on a field, such as two different prices for the same product on the same day, do not silently pick one. Flag the group for review, because the disagreement usually means the scraper captured something else, like a sale price on one page and a regular price on another.
Suppose a hypothetical run collects 10,000 listings and finds 600 keys that appear more than once, covering 1,350 rows. Removing the extra copies leaves 9,250 records. If 40 of those 600 groups disagree on price, those 40 are worth a person's time before the data is published.
Step 6: Catch near duplicates with a similarity check
Some duplicates do not share a key: the same job posted twice with different listing IDs, the same product listed by the same seller under two titles. Catching them requires comparing content, which is slow if every record is compared with every other one.
Use blocking first: only compare records that share something cheap to check, such as the same source, the same brand and a price within a small range, or the same company and city for job listings. Within each block, compute a similarity score on the normalized title or description, using a string similarity measure or the overlap of word sets. Pick a threshold, then review a sample of pairs just above and just below it to see whether it separates true duplicates from similar but different items.
What this catches: reposted content and duplicate listings under new IDs. What it misses: duplicates that fall into different blocks, and genuinely different items with nearly identical text, such as two sizes of the same product. That is why near-duplicate matches should be flagged or linked rather than deleted automatically, at least until the threshold has proved itself on reviewed samples.
Step 7: Log every merge and watch the duplicate rate
Deduplication changes data, so it needs a record. For each removed or merged row, log the key, the rule that removed it and the identifier of the surviving record. This makes it possible to undo a bad rule and to explain to an analyst why a count changed.
Then track the duplicate rate per source on every run: the share of collected rows removed as exact duplicates and the share flagged as near duplicates. A stable rate is normal. A sudden jump usually means the source changed something, such as adding a tracking parameter to every link or listing products under new paths, and that is worth investigating before the data is used. The related data quality checks to run on every scraped dataset fit alongside this monitoring. Whether records come from an in-house collector or are delivered by a service such as ScrapingIsNotACrime, run the same duplicate checks on arrival, since duplicates can enter at the source, during collection or when files are combined.
