A competitor price monitoring project succeeds or fails on decisions made before any data is collected: which products to track, which sources are allowed, how products are matched across sites and who maintains the collection. To show how those decisions play out, this walkthrough follows a hypothetical case: a small online cookware retailer, which we will call Ridgeline Kitchen, that wants to know when competitors change prices on the products it also sells. The company, its numbers and its competitors are invented for illustration; the decisions are the ones real projects face.
The starting point: a vague goal and a tight budget
Ridgeline Kitchen sells about 1,200 products. The owner's first request is broad: "track competitor prices." Taken literally, that means every product on every competitor's site, every day. That would be expensive to build, heavy on the competitors' servers and mostly useless, since Ridgeline only reprices a few dozen products a week.
The first decision narrows the goal to a question the business can act on: for the 150 products that bring in most of the revenue, what are three named competitors charging, and when does that change? That single sentence fixes the scope at about 450 product pages, sets a daily frequency as sufficient, and defines the output: a table of product, competitor, price, availability and date.
The consequence is visible right away. A project of 450 pages a day is small enough to run politely and cheaply, and small enough to check by hand when something looks wrong.
Checking what each source allows
Before thinking about tools, the team checks each competitor. For each site, it records three things in a short document with the date:
- whether the site offers a product feed, an affiliate program with a product data feed, or an API that already provides prices;
- what the terms of service say about automated access and reuse of content;
- what robots.txt says about the product page paths the project needs.
In this hypothetical, the results differ by competitor. Competitor A has an affiliate program whose data feed includes prices, so Ridgeline uses that feed and collects nothing from A's pages. Competitor B's terms do not address automated access, and its robots.txt allows the product paths. Competitor C's terms prohibit automated collection, so C is dropped from the automated project; a staff member checks C's prices manually for the top 20 products once a week.
The data itself is low-risk: prices and availability for products, with no personal data. That keeps the privacy questions simple, although the team still avoids collecting customer reviews, which carry reviewer names. Ridgeline's owner asks the company's lawyer to look over the source document once, because rules on databases and site terms vary between jurisdictions and the business sells in more than one country.
Matching products across sites
Price data is only useful if the same product is compared with the same product. This is where Ridgeline's project would have failed if it had started with collection.
The team builds a matching table. Where competitors show a manufacturer part number or a barcode number on the page or in the feed, matching is exact. Where they do not, the team matches by brand, model name and size, and a person confirms each match once. Variants cause most of the trouble: a 10-inch and a 12-inch skillet from the same line can share a page, with the price changing when a size is selected. The decision is to record the price of a specific variant only, and flag pages where the variant cannot be identified.
After two days of work, 141 of the 150 products have confirmed matches for at least one competitor. The other nine are store-brand items with no equivalent, and they are dropped from the comparison rather than matched loosely.
Scheduling collection politely
For competitor B, the plan is one request at a time, a pause of several seconds between requests, and a run in the early morning in B's time zone, when its traffic is likely lower. At that pace, about 150 pages take well under half an hour. The scraper uses a descriptive user agent with a contact address, fetches robots.txt at the start of each run, and stops for the day if it receives repeated 429 or 503 responses.
The team also uses the product sitemap that B lists in robots.txt to detect when a product page is removed, instead of crawling category pages to find changes. This cuts the number of requests and avoids loading pages the project does not need.
The build-or-buy decision
With scope, sources and matching settled, the remaining question is who runs the collection. Ridgeline has one developer who also maintains the online store.
The team estimates both paths with hypothetical figures. Building in-house: about a week to write and test the collector and the quality checks for one site, then an estimated two to four hours a month for maintenance, more when the site redesigns its pages. Buying: a managed service such as ScrapingIsNotACrime, or a similar provider, takes on collection and maintenance for an agreed scope, and the developer's time goes to the matching table and the reports instead. The cost comparison depends on the quotes the company receives, so it treats any price as something to confirm directly.
The deciding factor is not the tool but the developer's time. Every hour spent fixing a broken extractor is an hour not spent on the store. Ridgeline chooses to run the small collector for competitor B in-house for three months as a trial, with a written rule: if maintenance exceeds a set number of hours in any month, or if more competitors are added, it will request proposals from managed providers. Whichever route it takes, the questions for a provider are already listed: how it checks terms and robots.txt, how it paces requests, how it reports failures and what happens to the data if the contract ends.
What happened in the first three months
In this hypothetical, the first month runs smoothly. In the second month, competitor B redesigns its product pages and the price field comes back empty on every page for two days. The null-rate check the team built catches it on the first run: the share of empty prices jumps from near zero to nearly all records, the job stops publishing, and the developer updates the extractor using the structured product data embedded in the new pages, which proves more stable than the old selectors.
In the third month, a pricing error appears: one product shows a price 70 percent lower than usual. The range check flags it, a person opens the page and finds that the scraper captured a monthly installment amount displayed next to the price. The extractor is corrected, and a rule is added to reject values next to installment wording. These are the kinds of checks described in the guide to data quality checks to run on every scraped dataset.
The business result is modest and real: Ridgeline adjusts prices on a few products each week based on observed changes, instead of on occasional manual checks.
What to take from this case
- Narrow the question first. A goal stated as a specific decision, such as weekly repricing of the top 150 products, set the scope, the frequency and the output in one step.
- Check each source separately. One competitor had a feed, one allowed collection and one did not. A single policy for all three would have been wrong for at least two of them.
- Use feeds and sitemaps before pages. They reduce load on the source and are usually more stable than page layouts.
- Invest in matching. Comparing the wrong variant produces confident, wrong numbers.
- Decide build or buy on maintenance time, with a written threshold for switching, rather than on the cost of the first version.
- Build the quality checks before the dashboard. Both incidents in this case were caught by simple checks, not by someone noticing a strange chart weeks later.
