Ending a scraping project well takes more than switching off the scheduler. The decision is how much to keep and for how long, and the trade-off is between the value of historical data and the cost and risk of holding it: storage, security, privacy obligations and the attention of whoever has to answer for it. A clean shutdown inventories everything the project created, stops collection everywhere it runs, keeps only what has a defined future use, deletes the rest including copies, and leaves a short record behind. The hypothetical case below follows one company through that process, with the choices it faced at each step.
The hypothetical project and why it is ending
A home furnishings retailer has spent two years collecting public product pages from fifteen competitors to track prices and assortment. The collection runs daily from two servers, feeds a pricing dashboard and sends a weekly file to an external consultant. Some sources are marketplaces, so the raw pages include seller names and, on a few listings, contact details that sellers chose to publish.
The retailer has now signed a deal for a licensed market data feed that covers most of the same products, and the pricing team no longer uses the scraped data. Maintenance still costs, in this hypothetical, around six hours of engineering time a month, and the storage bill keeps growing. Management asks for the project to be closed within a month.
Before agreeing, the project lead checks one alternative: keeping a reduced collection for the two competitors the licensed feed does not cover. The comparison is concrete. Those two sources account for a small share of the products the pricing team watches, yet keeping them would preserve most of the fixed cost, since the servers, monitoring and on-call duty stay the same whether the scraper visits fifteen sites or two. The team decides that occasional manual checks of those two sites are enough, and the full shutdown goes ahead. Had the two sources mattered more, handing just those to a provider would have been the next option to price.
Step 1: Inventory everything the project created
The first instinct is to list the scraper code and the main database. The team instead lists every place the project touches, and the list grows quickly:
- the scheduled jobs on two servers and one forgotten test job on a developer's machine;
- the raw page archive, the cleaned database, and the derived tables behind the dashboard;
- nightly backups and a copy of the database in an analytics environment;
- the weekly exports sent to the consultant, who keeps them on their own systems;
- the user agent string, the information page describing the bot, and the contact address on it;
- documentation: the data specification, the review of each source's robots.txt and terms, and the legitimate interests note written for the marketplace data.
The test job is the useful find. Had it been missed, it would have kept sending requests to three sites after the project was officially over. An inventory built from servers, schedules and storage bills, not from memory, is the check that catches this kind of leftover.
Step 2: Stop collection everywhere, then verify
The team disables every scheduled job, removes the credentials and API keys the scraper used, and revokes them with the issuing services so they cannot be reused. It then watches the outbound logs for a week to confirm that no requests leave from any machine with the project's user agent.
The bot information page stays online for a few months with a note saying collection ended on a given date, and the contact address keeps working. Site owners who see the user agent in old logs, or people who want to ask about their data, still have a route to a reply. This costs almost nothing and avoids an awkward silence if a question arrives later.
Step 3: Decide what to keep, and why
The pricing team would like to keep everything "in case". The project lead asks a narrower question for each data set: who will use it, for what, and until when. The answers sort the data into three groups.
- Keep: a monthly price index by category, computed from the scraped data, with no seller or product identifiers. It lets the team compare the new licensed feed with two years of history. Retention is set at three years, then review.
- Keep briefly: product-level prices without seller information for six months, while the team checks that the licensed feed matches the old series on overlapping products.
- Delete: the raw page archive, all marketplace seller data and the analytics copy.
The seller data is the clearest decision. It was collected for a purpose that no longer exists, so the basis for holding it ends with the project. Keeping it would mean carrying privacy obligations with no benefit, which is the kind of trade-off that is easy to state and costly to ignore.
One more check shapes the keep list: the licensed feed's own terms. The team confirms that comparing the feed with its own historical series is allowed, and that nothing in the new contract requires it to stop using data it collected before the contract began. That reading takes an hour and removes a doubt that could otherwise resurface during a renewal.
Step 4: Delete, including the copies
Deletion is planned as a checklist, source by source: the raw archive, the cleaned tables, the analytics copy, then the backups. The backups are the hard part. The team cannot edit old backup files, so it lets them expire on their normal rotation and records the date by which the last copy containing raw data will be gone. Until then, access to those backups is restricted.
The consultant is asked in writing to delete the weekly exports and to confirm when that is done. If collection had run through a managed provider, such as ScrapingIsNotACrime or any other service, the same request would go to the provider, along with a question about any copies held in its own logs or backups. Deletion in practice involves more places than the main database, and each one needs an owner and a date.
Step 5: Keep a short closing record
The last task is a one-page note filed with the project documentation. It states when collection stopped, what was kept, the retention date for each kept data set, what was deleted and when, the date the last backups expire, and who confirmed deletion outside the company. The source reviews and the legitimate interests note are archived alongside it rather than deleted, because they explain why the kept data exists.
This record answers the questions that tend to arrive months later, from an auditor, a site owner or a person in the data, without anyone having to reconstruct the project from memory.
What it cost and what it saved
In this hypothetical, the shutdown takes about four days of engineering time and a few hours from the project lead, spread over the month. Against that, the company stops paying for two servers and the growing archive, frees the six monthly hours of maintenance, and removes a store of personal data it no longer had any reason to hold.
The alternative many teams fall into, leaving the jobs running "for now" or switching them off and forgetting the data, looks cheaper in the first month and costs more every month after.
What to take from this case
- Plan the end at the start. A data specification that names retention periods and owners turns a shutdown into a routine task.
- Build the inventory from evidence: schedules, logs, storage and bills. Memory misses test jobs and copies.
- Keep aggregates, not raw data, unless a named use needs the raw records.
- Treat deletion as a list of places, each with a date, including backups, exports and providers.
- Leave a contact route and a one-page closing record. Both are cheap, and both answer questions later.
Where personal data is involved and the retention choice is not obvious, a short review with a data protection lawyer is worth the time. The principles in ethical data extraction and compliance apply at the end of a project as much as at the start.