The common assumption is that robots.txt is the only place a website states its preferences about automated access. It is the best known, but not the only one. Sites also use robots directives inside individual pages, the same directives sent as an HTTP header called X-Robots-Tag, link attributes, license links and, increasingly, signals that reserve rights against text and data mining. These signals do different jobs: some address search engine indexing, some address copyright, and none of them replaces the site's terms of service. Knowing which is which keeps a collection project from either ignoring a clear preference or reading a prohibition into a signal that does not express one.
What a machine-readable opt-out is
A machine-readable opt-out is a statement a website makes in a form software can detect, rather than in prose for a person to read. The point is scale: a crawler visiting millions of pages cannot read every legal notice, but it can check a file, a tag or a header on each response.
What generally holds is that these signals communicate the publisher's preferences. What depends on jurisdiction and context is how much legal weight a given signal carries. In some situations a signal is only a convention that well-behaved crawlers follow; in others, such as certain copyright exceptions in the European Union, a machine-readable reservation can change what the law allows. That second question is where a lawyer familiar with the relevant jurisdiction should be consulted for any specific project.
Robots meta tag
The robots meta tag sits in the head section of an HTML page and applies to that page only. Its common directives are:
- noindex: do not show this page in search results.
- nofollow: do not follow the links on this page for crawling or ranking purposes.
- noarchive: do not offer a cached copy of the page.
- nosnippet: do not show a text excerpt of the page in results.
- none: shorthand for noindex and nofollow together.
The tag can address all crawlers or name a specific one. Its vocabulary was designed for search engines, so most directives describe indexing and display rather than collection in general. A noindex page has still been published; the owner simply does not want it surfacing in search. For a data project, that is useful context about how the owner expects the page to be found and reused, even though the directive does not literally say "do not collect".
X-Robots-Tag header
X-Robots-Tag carries the same directives as the meta tag, but in the HTTP response headers instead of the page body. Two differences matter in practice. First, it works for files that have no HTML head, such as PDFs, images and spreadsheets. Second, a server can apply it across a whole section of a site with one configuration rule, so it often reflects a deliberate site-wide decision.
A collector that parses only the HTML body will never see this header. Logging response headers alongside each fetched document is the simplest way to make sure it is not missed.
Link attributes and license links
Individual links can carry rel values such as nofollow, ugc (user-generated content) and sponsored. These are hints about the relationship between the page and the link target, used mainly for search ranking. They are not instructions to avoid the target page, and treating them as crawl prohibitions would misread them.
A rel value of license works differently: it points to the license under which the page's content is offered. When a page links to an open license this way, the license text tells you what reuse is permitted and on what conditions, such as attribution. It is one of the few signals that can grant permissions rather than restrict them, and it is worth recording with the data.
Text and data mining reservations
Text and data mining (TDM) means the automated analysis of text and data to find patterns, trends and correlations. The European Union's 2019 copyright directive created an exception allowing TDM on lawfully accessible content, unless the rights holder has expressly reserved its rights; for content made publicly available online, the reservation is expected in an appropriate manner, such as machine-readable means. A separate exception covers research organizations and cultural heritage institutions doing scientific research.
That raised a practical question: what does a machine-readable reservation look like? One answer is the TDM Reservation Protocol, published by a W3C community group. It lets a site declare a reservation through an HTTP header, an HTML meta tag or a file in a well-known location on the server, using a tdm-reservation value and, optionally, a tdm-policy link that points to the terms under which mining is licensed. It is a community specification rather than a formal W3C standard, and other approaches, including entries in robots.txt and further proposals in standards bodies, are also in use or under discussion. Expect this area to keep changing, and check the current state of practice when planning a project that relies on the TDM exception.
How these signals relate to robots.txt and terms of service
| Signal | Where it lives | What it mainly addresses |
|---|---|---|
| robots.txt | One file at the site root | Which paths crawlers may request |
| Robots meta tag | Head of each HTML page | Indexing and display of that page |
| X-Robots-Tag | HTTP response header | Indexing and display, any file type |
| rel nofollow, ugc, sponsored | Individual links | Relationship to the linked page |
| rel license | Link in the page | License for reuse of the content |
| TDM reservation | Header, meta tag or well-known file | Copyright reservation against mining |
| Terms of service | Legal page in prose | Contractual conditions of use |
The table shows why one check is never enough. Robots.txt answers "may I request this path?", while a TDM reservation answers a copyright question about what you do with content you can already read. Terms of service sit apart from all of them: they are a contract question, and a site can restrict automated collection there even when every machine-readable signal is silent.
Whether collection is done in-house or by a provider such as ScrapingIsNotACrime, checking and recording these signals belongs in the project specification, alongside the terms review. The overview of ethical web scraping and data privacy standards covers how such checks fit with privacy obligations, and the explanation of copyright and database rights in scraped data places them in the wider legal picture.
Common mistakes with opt-out signals
- Reading silence as permission. The absence of a signal means the site has not used that mechanism. It does not settle the terms of service, copyright or privacy questions.
- Treating every directive as a prohibition on collection. A nofollow link or a nosnippet page says something specific about search behavior. Recording it as "collection forbidden" overstates it, and recording it as irrelevant understates the owner's intent; note it and weigh it with the other signals.
- Checking only the home page. Meta tags and headers are set per page or per section. A site can mark its product catalogue one way and its user forum another.
- Ignoring headers. A pipeline that stores only the parsed body throws away X-Robots-Tag and header-based TDM reservations. Keep the response headers, at least for a sample of each section.
- Checking once and never again. Signals change. A reservation added after the first collection applies to what you collect from then on, so re-check on a schedule and store the date each signal was observed.
- Not recording what was seen. If a question about a dataset comes up later, a record of which signals were present at collection time, with dates, is far more useful than a recollection that "the site seemed fine".
