Many people assume that if a page is public and robots.txt does not block it, the website's terms of service have nothing to say about collecting it. That assumption breaks often. Terms of service are where a site states whether it permits automated access, what you may do with its content and which country's courts would hear a dispute, and robots.txt answers none of those questions. Reading the terms before a project starts means finding the right documents, locating a handful of specific clauses, judging how clearly they apply to your use, and writing down what you found. This guide covers each of those steps. It describes what is generally true; whether a particular clause is enforceable against a particular project depends on the jurisdiction and the facts, which is a question for a lawyer.
Where a website keeps its terms
The main document is usually linked in the page footer under a name such as Terms of Use, Terms of Service, Terms and Conditions, Conditions of Use or simply Legal. On larger sites it is rarely the only one. Before reading anything closely, list every document that could apply:
- The general terms of use, which cover anyone browsing the site.
- Developer or API terms, which apply if the site offers an API and often set rules for data obtained through it, such as storage limits or display requirements.
- An acceptable use policy, sometimes separate from the terms, which may list prohibited activities including automated access.
- The privacy policy, which does not bind you the way the terms might, but tells you what kinds of personal data appear on the site and how the site describes its users' expectations.
- Content or licensing pages, where a site may state that some material is available under an open license.
A common mistake is to read only the general terms of a site that also has an API. If the site offers an official way to get the data, its terms usually say so, and some general terms point automated users explicitly to the API.
How terms of service become binding
Whether a site's terms bind someone who never clicked "I agree" is one of the questions that depends most on jurisdiction and facts. A few distinctions come up often enough to be worth knowing in general terms.
Clickwrap describes terms you accept by an affirmative action, such as ticking a box or clicking a button when creating an account. Browsewrap describes terms that claim to apply simply because you use the site, usually through a footer link. In the United States, courts have generally been more willing to enforce clickwrap agreements than browsewrap terms, especially where the user had no clear notice of the browsewrap terms. That is a tendency, not a rule, and other countries approach contract formation differently.
The practical consequence for a scraping project is that creating an account changes the picture. Once you have accepted terms that prohibit automated collection, you have a clearer contractual relationship with the site. Content that is only visible after logging in is, for that reason and for privacy reasons, outside the scope of a responsible public-data project. Data behind a login belongs in a conversation with the site owner or in an official API, not in a scraper.
The clauses to look for
Terms of service can run to dozens of pages, but only a few clauses usually matter for data collection. Search the document for words such as automated, robot, spider, crawl, scrape, harvest, extract, data mining and bot, then read the full paragraph around each match. Beyond those, look for the following.
Automated access
This is the clause most readers are looking for. It may prohibit automated access outright, prohibit it without prior written permission, allow it only for search engines, or limit it to what robots.txt permits. Each of those wordings leads to a different decision, as the section on unclear wording below shows.
Ownership and license of content
Terms usually state who owns the site's content and what license visitors receive, often a limited license for personal, non-commercial viewing. This clause matters most for what you do after collection: storing, analyzing, republishing or selling. Collecting product prices for internal analysis and republishing full product descriptions on another site are very different uses, even if the collection step is identical.
Restrictions on commercial use and redistribution
Some terms forbid using the content to build a competing service, or forbid redistribution in any form. If your project feeds a commercial product or shares datasets with clients, read these restrictions as carefully as the automated access clause.
Load and interference
Many terms prohibit anything that places an unreasonable load on the servers or interferes with the site's operation. Even where collection is otherwise permitted, this clause is a reason to keep request rates low and schedule runs outside peak hours.
Governing law and disputes
This clause names the law that governs the terms and where disputes would be heard. It does not tell you whether the terms are enforceable, but it tells a lawyer where to start.
Changes to the terms
Almost every site reserves the right to change its terms. A review done once is a snapshot, which is why the record described later includes a date.
Reading wording that is not clear-cut
Few clauses say "scraping is allowed" or "scraping is forbidden" in those words. Most fall somewhere on a range, and the useful habit is to classify each clause instead of reading it optimistically. Consider three hypothetical wordings for the same kind of site:
- "You may not use any robot, spider or other automated means to access the site for any purpose." This is a broad prohibition. A project that respects the site's terms would not collect from it without written permission.
- "You may not use automated means to access the site without our prior written permission." This is a prohibition with an explicit route to permission. It is an invitation to ask, described later in this guide.
- "Automated access must comply with our robots.txt file and must not degrade the performance of the site." This is conditional permission. Collection that follows robots.txt and stays gentle on the servers fits the wording.
Two other situations are common. Terms that say nothing at all about automated access are silent, not permissive: the remaining clauses on content use, load and redistribution still apply. And terms that grant a license only for "personal, non-commercial use" do not prohibit collection as such, but they make commercial reuse of the content a separate question to resolve before the project starts.
The common mistake at this stage is reading a clause for the most convenient interpretation. If two reasonable people could read a clause differently, record it as unclear and treat it as a question for the site owner or a lawyer.
When the terms and robots.txt disagree
Robots.txt and terms of service are separate signals, and they do not always agree. A site may allow all crawlers in robots.txt while its terms prohibit automated collection, often because robots.txt was written with search engines in mind and never revisited. The opposite also happens: permissive terms and a robots.txt file that disallows the paths you need.
The prudent approach is to follow the more restrictive of the two. Robots.txt is a technical convention that well-behaved crawlers honor; the terms are the site's statement of the conditions of use. A disallow rule in robots.txt is a clear request from the site owner, and a prohibition in the terms is a clear statement of intent. Neither one cancels the other.
Keeping a record of your review
A written record turns a reading of the terms into a decision someone else can check. It also shows, if questions come up later, that the project considered the site's conditions before collecting anything. A simple table per source is enough:
| Field | What to record |
|---|---|
| Source and date | The site, the date of the review and the URL of each document read |
| Automated access | A short quotation of the clause and its classification: prohibited, prohibited without permission, conditional, silent |
| Content use | What the license allows and any restriction on commercial use or redistribution |
| Official alternatives | API, data feed, open data portal or licensing program, if any |
| Decision | Collect, collect with conditions, ask for permission, or do not collect |
| Next review | A date to read the terms again, for example every six months or when the site redesigns |
Saving a copy of the terms as they stood on the review date helps too, since sites update them without notice.
Asking for permission
When the terms prohibit automated access without permission, or when a clause is unclear, asking is often faster than it sounds. Look for a contact address for partnerships, business development, data licensing or legal matters. A useful request is short and specific:
- who you are and what the project is for;
- which pages or data fields you want, and which you will not collect;
- how often you will collect and at what rate;
- how the data will be used, stored and whether it will be shared;
- a contact for any concerns.
A site may answer with permission, a condition, a pointer to an API or feed, a price for a license, or a refusal. Each of those is a usable answer. Keep the reply with the record of your review. If there is no answer, silence is not permission.
What terms of service do not settle
Terms of service are one input among several. Even when a site's terms permit collection, other rules can still apply: privacy law when the data includes personal information, copyright and database rights over the content, and computer misuse laws in some jurisdictions. Conversely, a site's terms cannot by themselves make every use of public facts unlawful in every country. This is where general guidance runs out. When a project depends on a source whose terms are restrictive or unclear, or when it involves personal data, a lawyer familiar with data and technology law in the relevant jurisdictions should review the plan before collection starts.
The wider picture of how these rules interact is discussed in the post on ethical data extraction and compliance, and recent developments for platforms are covered in emerging legal frameworks in social media scraping.
Reading terms of service as part of every scraping project
A terms review is not a formality to complete once and file away. It belongs at the start of every project and every new source, alongside the robots.txt check, whether the collection is built in-house or handled by a managed service such as ScrapingIsNotACrime. In practice, the routine comes down to a short sequence:
- List every document that could apply: general terms, API terms, acceptable use policy and privacy policy.
- Search for the automated access clause and classify it as prohibited, prohibited without permission, conditional or silent.
- Read the clauses on content ownership, commercial use, redistribution and server load against what you plan to do with the data.
- Compare the terms with robots.txt and follow the more restrictive of the two.
- Prefer an official API, feed or license when the site offers one.
- Ask for permission when the terms require it or are unclear, and do not collect while waiting.
- Record the review with a date, a copy of the terms and a next review date.
- Bring in a lawyer when the source is important to the project and the terms are restrictive or unclear, or when personal data is involved.
Projects that follow this sequence rarely discover a problem after the data is already in use, which is the most expensive time to find one. For a view of how privacy obligations fit alongside the terms, see the myths about collecting public personal data under the GDPR and CCPA.
