When scraper maintenance starts eating a team's week, the cause is rarely one bad site. It is usually a mix of five things: more sources than the project's value justifies, extractors tied to fragile parts of the page, breakage discovered late, fields collected that nobody uses, and no clear owner. The fix is to cut the scope to what the business uses, make extraction less fragile, detect failures on the run where they happen, and decide deliberately which sources to keep in-house and which to hand off. The trade-off to weigh is time against control: every hour spent repairing a collector is an hour not spent on the analysis the data was for.
The symptom: a small project that became a standing cost
The pattern is familiar. A collector built in two weeks for a handful of sites grows to dozens. A developer who used to spend an afternoon a month on fixes now spends a day or two every week. Requests for new fields queue up behind repairs, and the business starts asking why the data is late.
Before changing anything, measure where the time goes for one month. For every maintenance task, log the source, the cause, the hours spent and whether anyone used that source's data in the same month. Suppose, as a hypothetical, a team running 40 sources logs 45 hours of maintenance in a month. That number alone does not say what to do; the breakdown by source and cause does.
Group the log three ways: hours by source, hours by cause, and hours by whether the source's data was used that month. Typical cause labels are layout change, empty or wrong values found late, new field request, access problem such as rate limiting or an error page, and infrastructure. A single month can be unusual, so if the numbers look surprising, keep logging for a second month before acting. Each of the causes below shows up as a distinct pattern in this breakdown.
Cause 1: more sources than the decisions need
Sources tend to be added and never removed. In the hypothetical log, it is common to find that a small number of sources feed most of the decisions while the long tail is collected out of habit. If 12 of the 40 sources account for most of the data that reaches a report, and the remaining 28 account for most of the repair hours, the arithmetic is clear.
Fix: ask the people who use the data which sources they actually look at. Retire sources nobody uses, and reduce the frequency for sources used only occasionally. Collecting a low-value source monthly instead of daily cuts both the maintenance exposure and the load on that site.
Cause 2: extractors tied to fragile markup
Selectors built on generated class names or deep positional paths break with every front-end deploy. When the maintenance log shows the same few sources breaking every few weeks, fragile extraction is usually the reason.
Fix: rebuild those extractors to read the most stable layer the site offers: an official API or feed, then structured data embedded in the page, then descriptive attributes and semantic elements, and class names only as a last resort. The rebuild costs time once and usually pays back within a few months on a source that breaks often.
Cause 3: breakage found days or weeks later
A broken extractor that runs without errors keeps producing empty or wrong values until someone notices a strange chart. By then the fix includes backfilling, explaining the gap and rechecking reports, which can take longer than the repair itself.
Fix: add checks that run with every job: row counts compared with the previous run, the share of empty values per field, and simple range checks on numbers. A job that fails these checks should stop publishing and alert someone. Finding a break on the first bad run turns a multi-day cleanup into a short repair.
Cause 4: fields that nobody uses
Every extra field is another selector that can break. Projects accumulate fields because someone once asked for them, and nobody asks again when the need passes.
Fix: list every field per source and mark which ones appear in a report or feed a decision. Stop collecting the rest. Fewer fields also mean less data to store and, where any of it touched personal information, less to protect.
Cause 5: no clear owner
When scraper upkeep belongs to whoever has time, it belongs to nobody. Repairs wait, knowledge sits with one person, and the same problems get diagnosed twice.
Fix: name one owner for the collection, give the role a time budget, and keep a short runbook per source covering what it collects, how it was checked against the site's terms of service and robots.txt, how it usually breaks and how it was fixed last time.
Deciding what to keep in-house and what to hand off
Once scope is cut and monitoring is in place, compare the remaining maintenance hours with what that time is worth to the business. A useful rule is to rank sources by repair hours per month against their value to decisions. High-value, stable sources are good candidates to keep in-house. High-value sources that break often, or a long list of sources the team cannot maintain well, are where a managed provider such as ScrapingIsNotACrime or a comparable service can make sense, provided its approach to site terms, request pacing, failure reporting and data quality is checked before signing. The guide to choosing between browser extensions, no-code tools, frameworks or a managed service covers that decision in more detail.
In the hypothetical team, suppose the log shows 20 of the 45 hours going to sources nobody used, 12 to six extractors that break repeatedly, and 8 to cleanups after late detection. Retiring the unused sources, rebuilding the six extractors and adding run-level checks would remove most of those hours, leaving perhaps 10 to 15 a month after the rebuild is done. Whether that remainder stays in-house then becomes a question with numbers attached, not a feeling. Whatever is decided, write down the reason and the date, so the next review can see whether the assumption held.
How to keep maintenance from creeping back
- Require a stated use for every new source and field, including who will look at it and how often.
- Review the source list every quarter against the maintenance log and retire what is no longer used.
- Set a monthly hour threshold per source that triggers a rebuild, a reduced frequency or a hand-off when it is crossed twice.
- Keep the runbook current after every fix, so the next repair starts from what was learned.
- Prefer official APIs, feeds and data licenses whenever a source offers them; they break less often and raise fewer questions with the site owner.
