Is web scraping legal? There is no general rule that makes it either legal or illegal: the answer depends on what data you collect, how you reach it, what you do with it and which jurisdiction applies. The same is true for the question many people search, "is scraping Instagram legal": public visibility does not settle it, because the platform's terms, personal data laws and copyright still apply. The six myths below are the assumptions that most often lead people to the wrong conclusion, in one direction or the other.
What follows describes what generally holds. It is not legal advice, and for a specific project, especially one involving personal data or a platform that has objected, a lawyer who knows your jurisdiction is the right person to ask.
Myth 1: "Web scraping is illegal"
The common assumption is that automated collection is suspect by nature. It breaks down quickly: search engines, price comparison sites, archives and academic researchers all rely on automated collection of public pages, and in most places no law prohibits the technique itself.
What creates legal risk is usually something attached to the scraping, not the scraping as such:
- getting past a login, a paywall or a technical block to reach data that is not public;
- collecting personal data without a lawful basis or using it for something people would not expect;
- copying and republishing protected content;
- sending so many requests that the site's operation suffers;
- breaching terms you have agreed to.
A project that avoids all five is in a very different position from one that does any of them. That is why "is scraping legal?" has to be asked about a specific project rather than about the technique.
Myth 2: "If a page is public, the data on it is free to use"
Public means anyone can view the page. It does not mean the content has no owner or that the people mentioned on it have no rights.
- Copyright: articles, photos, reviews and product descriptions are often protected. Extracting facts, such as a price or a date, is generally treated differently from copying the creative text around them.
- Database rights: in the EU and the UK, a substantial extraction from a database can be restricted even when individual entries are not protected.
- Personal data: under the GDPR, a name or a profile is personal data whether or not it is publicly visible. US state privacy laws such as the CCPA treat publicly available information differently, but the details vary.
A hypothetical example: a team collecting public product listings stores the price, the product name and the date. That is a very different project from one that also stores every review in full, along with the reviewer's username and profile photo, even though both read the same public pages.
The practical question is not "can I see it?" but "what am I keeping, and what will I do with it?"
Myth 3: "Scraping Instagram is legal as long as the profiles are public"
This is the myth behind the search "is scraping Instagram legal", and it combines two separate questions. The first is the platform's contract. Instagram's terms of use restrict collecting information in an automated way without the platform's permission, and anyone with an account has agreed to those terms. Breaching them can lead to account bans and, in some cases, legal claims, even when the pages were visible without logging in. Terms change, so check the current version before you plan anything.
The second question is personal data. Public profiles are full of it: names, photos, locations, opinions. Collecting it at scale brings privacy obligations that do not disappear because the account is public.
For most research and business needs involving social media, the safer starting point is an official API or a research access program, with scraping of public pages considered only after the terms and the privacy questions have been answered, ideally with legal advice.
Myth 4: "robots.txt is only a suggestion, so ignoring it has no consequences"
It is true that robots.txt is a convention, not a law, in most jurisdictions. It is also the most direct way a site owner tells automated visitors what they may and may not fetch.
Ignoring it can matter in several ways: it can be cited as evidence that you knew the owner did not want the collection, it can lead to blocks, and some opt-out signals for text and data mining are expressed through robots rules and related tags. Respecting it costs little. If you are unsure how the rules work, start with how to read a robots.txt file and apply its rules before you scrape a website.
Myth 5: "As long as nothing breaks, the number of requests does not matter"
The assumption is that a site that keeps responding is a site that is fine. A server can keep answering while its real users wait longer, its hosting bill grows or its staff spend a weekend investigating an unexplained spike.
Request volume is one of the few things entirely under your control, and it is often what turns a tolerated scraper into a blocked one or a dispute. A hypothetical example: a job collecting 20,000 product pages can run at one request every two seconds and finish in about eleven hours, or fire hundreds of parallel requests and finish in minutes while degrading the site. The data is the same; the impact is not. The article on rate limiting in web scraping, crawl rates and backoff explains how to set a polite pace.
Myth 6: "Hiring a scraping service moves the legal responsibility to the provider"
Outsourcing the collection changes who runs the scrapers. It does not change who decided what data to collect and why. In privacy law, the organization that sets the purpose of the processing usually carries the main obligations, even when a provider does the technical work.
When you work with any provider, ScrapingIsNotACrime included, you still need to define the sources, the fields and the purpose, check that personal data has a lawful basis, and decide how long the data is kept. A good provider can help you frame those questions; it cannot answer them on your behalf.
A short way to keep all six myths in view is a written checklist reviewed before each project. The ethical web scraping checklist for responsible data scraping covers the points to confirm before and during collection.