Skip to content
Home Posts About us Privacy Policy
Legitimate Interests and Web Data Collection: A Guide to the Three-Part Test

Legitimate Interests and Web Data Collection: A Guide to the Three-Part Test

Share ๐• in f W

A common assumption in web data projects is that personal data on public pages can be collected under "legitimate interests" simply because it is public and the project has a business reason. Legitimate interests is indeed the lawful basis most often considered for collecting public personal data under the GDPR, but it is not automatic. It requires a three-part assessment: whether the interest is legitimate, whether the processing is necessary for it, and whether the interests and rights of the people in the data override it. The answer depends on what data is collected, from whom, for what purpose and with which safeguards, and it has to be written down before collection starts. This guide explains each part of the test, what tends to tip the balance, which data it cannot cover on its own, and how the assessment fits into a project.

What legitimate interests is, and where it applies

The GDPR requires a lawful basis for every processing of personal data, and it lists six: consent, contract, legal obligation, vital interests, public task and legitimate interests. For web data collection, most of these do not fit. There is no contract with the people whose public posts or listings are collected, consent is rarely practical to obtain from them, and the legal obligation and public task bases apply only in specific circumstances. That usually leaves legitimate interests.

The basis allows processing that is necessary for the legitimate interests of the controller or a third party, unless those interests are overridden by the interests or fundamental rights and freedoms of the people concerned, with particular weight given to children. Public authorities cannot use it for processing carried out in performing their tasks. The organization relying on it is responsible for showing that the conditions are met, which is why the assessment has to be documented. Who that organization is in a project with several parties is a question of controller and processor roles, which should be settled first.

The same concepts apply in the UK under the UK GDPR, although UK rules have since diverged in some details. In both cases, data protection authorities publish guidance on legitimate interests, and it is worth reading the current version from the authority that applies to you.

The purpose test: is there a legitimate interest?

The first question is what the processing is for and whether that interest is legitimate. The bar is not high, but the interest has to be real, lawful and specific. A commercial interest can qualify. "Market research" is too vague; "measuring how customers describe delivery problems in public reviews of our products and three competitors, to prioritize logistics changes" is specific enough to assess.

Write the purpose down in a sentence or two, along with who benefits: the organization, its customers, researchers, the public. An interest that is unlawful in itself, such as collecting contact details to send unsolicited marketing where that is prohibited, fails here and the assessment ends.

The interest of a third party can count too. A research team collecting public posts to study how misinformation spreads may point to the interest of the wider public in that knowledge, alongside its own. The more clearly the benefit is described, the easier the balancing test becomes, because there is something concrete to weigh. A purpose that changes later, such as research data that the team then wants to sell, is a new purpose and needs its own assessment.

The necessity test: is this processing actually needed?

The second question is whether processing personal data, in this way and to this extent, is necessary for the purpose, or whether the same goal could reasonably be achieved with less. This is where many web data projects can be improved, because scrapers tend to collect every field on a page.

Go field by field. For the hypothetical delivery-problems project, the review text and date are needed; the star rating is useful; reviewer names, profile links and photos are not, because the analysis is about what is said, not who said it. Ask, too, whether personal data is needed at all: if aggregated figures or content without identifiers serve the purpose, collecting identifiable data is hard to justify as necessary.

Necessity also covers scope and time: how many sources, which period, how often, and how long the data is kept. A project that needs one year of reviews does not need ten.

A common mistake is to treat necessity as a question about convenience: it is easier to collect every field and decide later what to use. That approach usually fails the test, and it creates obligations for data that brings no benefit. A practical way to apply necessity is to write the extraction rules from the assessment rather than the other way round, so the scraper never stores fields that the assessment did not justify.

The balancing test: whose interests weigh more?

The third question weighs the interest against the impact on the people in the data. This is the part that requires judgment, and it is where "the data was public" is one factor among several rather than the answer. The factors usually considered include:

  • The nature of the data. Product reviews differ from posts about health, politics or private life.
  • Reasonable expectations. Would the people concerned expect their data to be collected and used this way, given where and why they published it? A professional listing a business address expects it to be read; someone posting in a small community forum may not expect bulk collection and analysis.
  • The likely impact. Could the processing lead to decisions about individuals, profiling, exposure, embarrassment or loss of control over their information?
  • Who the people are. Children and other vulnerable groups weigh heavily on the individuals' side.
  • Scale. Systematic collection across many sources creates more risk than a small, targeted dataset.
  • Safeguards. Measures that reduce the impact, covered below, can shift the balance.

Hypothetical comparisons make the factors concrete:

ProjectFactorsLikely direction
Aggregate sentiment from public product reviews, names discarded at collectionLow-sensitivity data, expected public use, no decisions about individualsThe interest is more likely to prevail
Monitoring public statements of company spokespeople about an industryProfessional role, statements made publicly for that purposeOften defensible with clear limits
Profiles of individual forum members combining posts across sitesUnexpected, cross-source profiling, possible sensitive inferencesThe individuals' interests are likely to prevail
Collecting emails from public pages to send marketingContrary to expectations, often restricted by marketing rulesFails, often already at the purpose test

Data that legitimate interests cannot cover on its own

Some categories need more than a lawful basis. The GDPR's special categories, such as data revealing health, racial or ethnic origin, political opinions, religious beliefs, trade union membership, sex life or sexual orientation, and genetic or biometric data used to identify someone, can only be processed if one of a separate list of conditions also applies. One of those conditions concerns data that the person has manifestly made public, but how far it reaches is a question of interpretation that depends on the circumstances, and it should not be assumed to cover everything posted publicly. Data about criminal convictions and offences has its own restrictions.

Special category data can also appear by inference: a scraper collecting posts from a patient support group collects health data even if no field is labelled "health". If a project could touch these categories, either design them out or take legal advice before collection.

Safeguards that change the balance

Safeguards are what turn a borderline assessment into a defensible one. The common ones for web data are:

  • Minimization at collection: dropping identifiers in the scraper itself, so they never reach storage.
  • Pseudonymization and aggregation: replacing identifiers with keys held separately, or keeping only counts and summaries for analysis.
  • Short retention: deleting raw data once the derived dataset is built and checked.
  • Access control and security: limiting who can see raw records.
  • No decisions about individuals: a stated rule that the data is used only in aggregate.
  • An easy way to object: a contact route and a process to remove a person's data on request.

Safeguards count only if they are actually in place. If the assessment says names are discarded but the raw page snapshots that contain them are kept for years, the balance on paper does not match the one in practice.

Transparency and the right to object

Relying on legitimate interests brings two obligations that the assessment has to plan for. The duty to inform people whose data was collected from public pages, and its exception for disproportionate effort, is covered in the post on myths about public personal data under the GDPR and CCPA; for the assessment, what matters is that the notice you rely on describes the same purpose, fields and objection route that the assessment records.

The second obligation follows from the basis itself: people have the right to object to processing based on legitimate interests. When someone objects, processing of their data has to stop unless the organization can show compelling legitimate grounds that override their interests, or needs the data for legal claims. For direct marketing, an objection must always be honored. In practice, this means a working process to find and remove one person's data across raw and derived copies.

Writing down the assessment

A legitimate interests assessment does not need to be long, but it needs to exist before collection and to match what the project does. A useful record covers the purpose and who benefits; the data collected, field by field, and why each is needed; the sources and the period; the factors in the balance and the conclusion; the safeguards; how people are informed and can object; the retention period; and the date and the person responsible. If the assessment points to a high risk, a fuller data protection impact assessment may be required.

Review it when the purpose, the sources or the fields change. A project that started with aggregate review analysis and later adds reviewer profiles has a new processing, and the old assessment does not cover it. If collection is handled by a provider such as ScrapingIsNotACrime, share the field list from the assessment so the collection matches it, and record who is responsible for each safeguard. The guide to controllers, processors and service providers covers who answers for each part of the work.

Outside the GDPR

Lawful bases are a GDPR concept. Other privacy laws work differently: the CCPA, for example, does not ask for a lawful basis but relies on notices, rights to know, delete and opt out, and restrictions on sale and sharing, and it treats some publicly available information differently from the GDPR. Other countries have their own rules, and some include concepts similar to legitimate interests. A project collecting data about people in several jurisdictions may need to satisfy several frameworks at once.

This is the point where general guidance stops. If the data includes sensitive categories, involves people in several countries, or the balance is genuinely close, have a lawyer familiar with data protection in the relevant jurisdictions review the assessment before collection starts. The overview of emerging legal frameworks in social media scraping gives more background on how these rules are developing for social data.

Legitimate interests for web data collection

Legitimate interests can support the collection of public personal data, but only for a specific purpose, with data limited to what that purpose needs, and when the balance with the people in the data comes out in the project's favor. Public availability helps in the balance without settling it. In practice, the projects that pass the test are the ones that decide on purpose and fields before building the scraper, drop identifiers they do not need at collection, keep raw data briefly, tell people what is happening and give them a way to object, and keep a dated assessment that matches what the code does. The overview of ethical web scraping and data privacy standards covers the day-to-day practices that keep a project consistent with that assessment.

Share ๐• in f โœˆ