When a Website Owner Objects to Your Scraping: A Practical Guide to Responding

Two assumptions are common when a website owner objects to data collection: that the objection can be ignored because the data is public, or that it means the project is over. Both usually break on contact with the facts. An objection is a signal to pause collection from that site, preserve your records, work out exactly what you collected and on what basis, and then choose a response. Many objections are resolved by reducing the rate, narrowing the scope or switching to an official channel. Formal legal letters are a different matter, and they should go to a lawyer before anyone replies. This guide covers the forms an objection takes, the first steps, how to read what is being asked, the realistic responses and how to make the next objection less likely.

The forms an objection takes

Not every objection arrives as a letter. Recognizing the quieter forms early gives you more options.

• An email from a site administrator, often sent to the contact address in your user agent, asking you to slow down or stop.
• Technical signals, such as a new robots.txt rule covering the paths you collect, a run of access-denied or too-many-requests responses, or your requests being blocked.
• A change in the terms of service that adds or tightens a clause on automated collection.
• A notice from a platform that your API key or developer access has been limited or revoked.
• A formal letter, often called a cease-and-desist letter, usually from a lawyer, stating legal claims and demanding that you stop and sometimes that you delete data.

A technical block is an objection, not an obstacle. Working around it, by rotating addresses or disguising the scraper, turns a disagreement into a deliberate act against the site's wishes, which weakens any position you might have and falls outside responsible collection. The response to a block is the same as the response to an email: pause and review.

The first steps, before any reply

Whatever the form, the first moves are the same.

1. Pause collection from that site. Stop the scheduled jobs for the affected source. This costs little and shows good faith whatever happens next.
2. Preserve records; do not delete anything yet. Keep the data, the logs and the code as they are. If the objection turns into a dispute, records of what you did and did not do matter, and in some jurisdictions destroying material relevant to a possible dispute creates its own legal problems. Once a formal letter has arrived, ask a lawyer before deleting anything, even if the letter asks for deletion.
3. Acknowledge receipt, briefly. For an informal email, a short, polite note that you have received the message and paused collection while you review it is usually enough. Do not argue the merits or admit anything in this first reply. For a formal legal letter, let a lawyer decide whether and how to acknowledge it.
4. Name one person to handle it. Replies from several team members, each with a different explanation, make any later conversation harder.

Reconstructing what you collected and how

A useful response depends on knowing exactly what happened. Build a short factual record covering:

• What was requested: which paths or pages, how many requests, over what period, at what rate and concurrency, and with which user agent.
• What the site said at the time: the robots.txt rules, the terms of service and any other signals as they stood when you collected, with dates. If you saved copies during your original review, this is where they pay off.
• What data you kept: which fields, whether any of them are personal data, and where the data is stored, including backups and copies sent to others.
• How the data was used: internal analysis only, published in aggregate, republished in part, or delivered to clients.

Hypothetically, a team might find that its scraper ran within the robots.txt rules but at a rate several times higher than intended because of a configuration error, and that it stored reviewer names it never used. That is a very different conversation from one where everything matched the documented plan, and it is better to discover it yourself than to hear it from the other side.

Reading what the objection asks for

Objections usually ask for one or more of a short list of things: stop collecting, reduce the rate, stop republishing, credit the source, delete what was collected, or obtain a license. Identify which of these the message actually asks for. An administrator worried about server load is asking for something very different from a publisher concerned about its content appearing elsewhere.

Formal letters also state the legal basis they rely on. In general terms, the bases that come up are:

• Contract: a claim that collection breaches the terms of service.
• Copyright and, in some jurisdictions, database rights: a claim about copying or reusing protected content or a substantial part of a database.
• Privacy and data protection: a claim about collecting or processing personal data.
• Computer misuse or unauthorized access laws: a claim that access itself was unauthorized.
• Harm to the service: a claim based on load, disruption or cost to the site.

Which of these has any force depends heavily on the jurisdiction, the facts and how courts there have treated similar cases, and the law in this area has shifted over time. That assessment is exactly what a lawyer with experience in the relevant jurisdiction is for. What you can do without one is map each claim to the facts in your record, so the lawyer starts from an accurate picture rather than a summary from memory.

The realistic responses

Once you know what is being asked and what you did, the options usually fall into five groups.

For informal objections, the second, third and fourth options resolve most cases. An administrator who asked you to slow down is often satisfied by a lower rate, a clear user agent and collection outside peak hours. A publisher who objects to republication may have no problem with internal analysis, or may offer a license.

When a site asks you to stop and you have no strong reason to do otherwise, stopping is usually the cheapest answer, whatever the legal position. Few datasets are worth a dispute with their source.

What deleting actually involves

If the outcome includes deletion, define it precisely. Data from a source can live in raw page snapshots, extracted tables, derived datasets, reports, caches, backups and files sent to colleagues or clients. Agree, in writing where possible, on what will be deleted, what counts as derived data that can be kept (for example aggregated statistics with no content from the source), and how deletion from backups will be handled, since many backups can only expire on their own schedule.

Keep a record of what was deleted and when. If personal data was involved, deletion may also intersect with privacy obligations toward the individuals in the data, which is a separate question from the site owner's request and one to raise with your lawyer.

When the objection comes from a person in the data

Sometimes the objection comes not from the site owner but from an individual whose information appears in your dataset, such as a reviewer, a forum member or a professional listed in a public directory. This is a different situation with different rules. Under privacy laws such as the GDPR in Europe and the CCPA in California, individuals can have rights to know what data is held about them, to have it deleted, or to object to its processing, subject to conditions and exceptions that vary by law and by case.

The practical steps mirror the ones above. Record the request and the date, because privacy laws often set response deadlines. Confirm the identity of the person asking, in proportion to the data involved, so you do not disclose or delete someone else's information. Search all copies of the data, including snapshots and derived tables, for that person. Then decide, with legal advice where the law or the request is unclear, whether to delete, restrict or retain, and reply within the applicable deadline.

A request like this is also a useful prompt to ask whether the project needed that person's data in the first place. If the analysis works without names or profile details, removing those fields for everyone is often simpler than handling requests one by one.

A hypothetical informal objection, start to finish

Suppose a small company collects public product listings from 40 specialty retailers, once a day. One morning, an email arrives from the administrator of one retailer's site: traffic from the company's scraper has been slowing the site during a sale, and they ask it to stop.

Following the steps above, the company pauses that source the same day and sends a two-line acknowledgment. Its logs show the scraper ran at the intended pace, but a retry setting caused it to resend requests repeatedly whenever the site slowed down, multiplying the load exactly when the site was busiest. The robots.txt allowed the paths; the terms said nothing about automated access.

The reply explains the cause in plain terms, describes the fix (a capped number of retries with growing waits, collection moved to early morning, and a lower rate), and asks whether the site would accept collection under those conditions or would prefer a product feed. The administrator offers a daily feed instead. The company switches to the feed, keeps the data already collected, which the administrator never asked it to remove, and ends up with cleaner data and a contact at the retailer.

Writing the substantive reply

For informal objections, a good reply is short and factual: what you collected, why, at what rate, what you have changed, and what you propose. Offer a contact point and, if the site wishes, a way to agree on terms. Avoid legal argument, avoid speculating about the site's motives, and avoid promises you cannot keep, such as deleting data from backups by a date your systems cannot meet.

For formal letters, the reply is the lawyer's call. Your contribution is the factual record and a clear view of how important the source is to the project, so the advice reflects the business stakes.

If a managed provider collects on your behalf, the objection may reach the provider, you, or both. A contract with a provider such as ScrapingIsNotACrime or any other service should say who receives objections, how quickly they are passed on, who pauses collection and who replies, so that nobody answers a site owner without the other party knowing.

Making the next objection less likely

Most objections can be prevented, or at least resolved quickly, with practices that cost little:

• Identify the scraper. A descriptive user agent with a working contact address lets administrators reach you with an email instead of a block.
• Keep rates modest and adaptive. Back off when the site slows down or returns too-many-requests responses, and avoid peak hours where you can.
• Review terms and signals before collecting and on a schedule, and keep dated copies of what you reviewed.
• Collect only what the project uses, especially where personal data is concerned.
• Prefer official channels wherever a site offers an API, a feed or a license.
• Write an objection procedure in advance: who receives messages, the pause-and-review steps above and when to involve a lawyer.

The overview of ethical web scraping and data privacy standards covers the privacy side of these practices.

Handling objections as part of the collection plan

An objection is easiest to handle when the project already expects one. The practical sequence is the same every time: pause the source, preserve records, reconstruct what you did from logs and dated reviews, identify what the owner actually asks for, and choose among stopping, scaling back, switching to an official channel, negotiating or, on legal advice, contesting. Informal requests rarely need more than a prompt pause and a cooperative reply. Formal letters need a lawyer before any substantive answer.

Teams that treat site owners as stakeholders rather than obstacles tend to receive fewer objections and resolve them faster.