The price error that slips into a scraped dataset unnoticed is not an empty field, which a null check catches. It is a value that looks valid and is wrong: 1.299 read as one point three instead of one thousand two hundred ninety-nine, a price in Canadian dollars stored as US dollars, or a monthly instalment recorded as the full price. To compare prices from different websites, each value has to go through the same steps: keep the raw text, identify the currency, parse the number according to the source's conventions, store it exactly, label what kind of price it is and on what tax basis, and only then convert or compare. This tutorial covers those steps in order, with a check for each and a note on what the check does not catch.
Step 1: Keep the raw price text and its context
Store the price exactly as it appeared, as text, before any parsing, together with the source, the page address, the collection time and, where available, the page's declared language or locale. Every later step can then be redone if a rule turns out to be wrong, without collecting the page again.
Keep the nearby wording too, when the extractor can capture it: labels such as "from", "per month", "per kg", "was" or "incl. VAT" change what the number means. Check: every normalized price in the dataset can be traced back to a raw string. What it misses: it does not tell you whether the raw string is the right price on the page; that is the extractor's job and needs its own spot checks.
Step 2: Identify the currency explicitly
Record each price's currency as a three-letter ISO 4217 code, such as USD, EUR, GBP or JPY, never as a symbol. Symbols are ambiguous: the dollar sign is used for several currencies, including the US, Canadian, Australian and Mexican ones, and "kr" is used in Sweden, Norway, Denmark and Iceland.
Use the most reliable evidence available, in this order: a currency code in the page's structured data or in the price markup; a code printed next to the price; the site's country or regional version; and only then the symbol, combined with the site's country. Record which evidence was used. Check: count prices per source by currency; a source that should be single-currency but shows two has a detection problem. What it misses: a site that shows prices converted to the visitor's location will look consistent while being in a currency you did not expect, so note the currency the site displayed to your collector during the source review.
Step 3: Parse separators according to the source's conventions
Decimal and thousands separators differ by locale. The same amount can appear as 1,299.00, 1.299,00, 1 299,00 with a space or narrow no-break space, or 1'299.00 in Switzerland. Indian grouping writes larger amounts as 1,29,999. A parser that guesses from the string alone will get some of these wrong.
Set the expected convention per source, or per regional version of a source, and parse with that rule. Remove spaces of every kind, including no-break spaces, before parsing. When a string contradicts its source's rule, such as a number with two decimal separators, reject it to a review list instead of guessing.
A hypothetical example shows the risk: a European site lists a jacket at 1.299 with no decimals. Read with a US convention, the parser stores 1.299, and the jacket appears as the cheapest in the dataset by a factor of a thousand. Check: for each source, compare the median parsed price with a few prices read by eye from the live site. What it misses: isolated errors in a source whose median looks right; the validation in Step 8 catches those.
Step 4: Store exact decimals with the right precision
Store parsed prices as exact decimal values, or as integers in the currency's minor unit (cents, pence), never as binary floating-point numbers, which cannot represent many decimal amounts exactly and accumulate rounding errors in sums and comparisons. Keep the currency code in its own column, next to the amount.
Not every currency has two decimal places. The Japanese yen is usually shown without decimals, and some currencies, such as the Kuwaiti dinar, use three. Take the precision from the ISO 4217 table rather than assuming two. Check: no amount has more decimal places than its currency allows. What it misses: amounts correctly stored but wrongly labeled, which is why Step 2 comes first.
Step 5: Label the type of price
A product page can show several prices, and only some of them are comparable. Give each price a type from a short fixed list, for example:
- current: the price a buyer pays now;
- regular: the price before a discount, often shown struck through;
- from: the lowest of several variants or dates;
- unit: price per kilogram, litre or item;
- instalment or subscription: a recurring amount, with its period and number of payments.
Keep a range as two values, low and high, rather than picking one. Store unit prices with their unit, and convert units only within the same dimension. Check: for each source, the share of each price type is stable from run to run; a sudden rise in "current" prices that are much lower than usual often means the extractor started reading an instalment. What it misses: a site that changes the meaning of a label without changing its layout.
Step 6: Record the tax and shipping basis
Prices are displayed on different bases. In many countries, consumer prices are shown including value-added tax, while in the United States they are usually shown before sales tax, which is added at checkout. Business-to-business sites often show prices excluding tax. Shipping may be included, extra or free above a threshold.
Add columns for the tax basis (included, excluded, unknown) and, where the page states it, shipping (included, extra, unknown). Do not try to add or remove tax yourself unless the rate is published on the page and the project really needs it; rates vary by country, region and product type. Check: no source has an unknown tax basis for most of its prices without a note explaining why. What it misses: taxes and fees that only appear at checkout, which public product pages do not show.
Step 7: Convert currencies with dated rates, and keep the original
If the analysis needs a single currency, convert in a separate column and keep the original amount and code untouched. Use exchange rates from one documented source, fixed to the collection date of each price, and store the rate and its date with the converted value. Converting with today's rate a price collected six months ago mixes price changes with currency movements.
Check: every converted value carries a rate, a rate date and a rate source. What it misses: the difference between a reference rate and the rate a buyer would actually pay, which matters for some analyses.
Step 8: Validate the normalized prices
A general data quality checklist will list "normalize currencies and units" as one item; the step-by-step version needs its own final validation. Run these checks after every load:
- Range by category: flag prices outside a plausible range for the product category and currency.
- Change from the previous run: flag items whose price moved by more than a set factor, for example ten times up or down, which usually means a separator or currency error rather than a real change.
- Cross-source ratio: for matched products, flag sources whose prices sit far from the others in the same currency.
- Current against regular: flag discounted prices that are higher than the regular price.
Send flagged records to review instead of deleting them, and record the decision, so that the rules improve over time. The data quality checks to run on every scraped dataset cover more checks of this kind, and the hypothetical competitor price monitoring project shows where comparable price data is typically put to use. If prices are delivered by a provider such as ScrapingIsNotACrime, ask for the raw price text, the currency evidence and the price type alongside the parsed value, so these checks can run on your side as well.