By the end of these steps, every request your scraper sends will carry a stable name that leads a site owner to a page explaining what the scraper does, how to limit or block it, and how to reach a person who answers. That takes an afternoon. It matters because an owner who sees unexplained traffic in the logs has only one tool, blocking, while an owner who can identify you can also ask you to slow down, skip a section or use a feed instead. Each step below ends with a check you can run yourself.
Step 1: Choose a product token and keep it for good
Pick a short, unique name for the scraper, made of letters, digits and hyphens, such as acme-price-monitor. This token is what site owners will search for in their logs and what they will write in robots.txt to give you rules, so it should not change when you rewrite the code or move servers. The guide on how to read a robots.txt file and apply its rules explains how that token is matched against the file's groups.
Use one token per purpose rather than one per company. If the same organization runs a price monitor and a job listings collector, two tokens let an owner allow one and block the other.
Check: search the web and a few large robots.txt files for your token. If another crawler already uses it, choose again, or you may inherit rules written for someone else.
Step 2: Build the user agent string
The user agent string is the value of the User-Agent header sent with each request. A widely used convention for automated clients has three parts: the product token, a slash and a version number, followed by a comment in parentheses that contains a plus sign and the full address of your information page. The version helps owners tell your old and new behavior apart, and the address gives them somewhere to go.
Set the string in one place in your configuration and have every part of the scraper read it from there, including the code that fetches robots.txt, sitemaps and feeds. Leave out anything that imitates a browser. A string that claims to be a desktop browser makes the traffic harder to attribute, and an owner who discovers the disguise will reasonably assume the worst about the rest of your behavior.
Check: point the scraper at a page on a server you control and read the access log. The logged user agent should match your configured string character for character.
Common mistake: one component, often a retry handler, a headless renderer or an image downloader, quietly sends its library's default user agent. Run the log check for every type of request the scraper makes, not just page fetches.
Step 3: Decide whether to add a From header
HTTP also defines a From header, meant to carry an email address for the person responsible for an automated client. Few sites read it, so it does not replace the information page, but it costs nothing and gives an owner who inspects headers a second route to you.
If you add it, use a role address such as a shared crawler mailbox rather than a personal one, so it keeps working when people change jobs.
Check: send a message to that address from outside your organization and confirm it reaches the people who can change the scraper's behavior.
Step 4: Write the information page
The page linked from the user agent is where an owner decides what to do about you. Keep it short and specific. It should answer:
- Who operates the scraper and for what purpose, in one or two plain sentences.
- What it collects: the kinds of pages and fields, and that it does not collect pages behind logins.
- How it behaves: how often it visits, its usual request rate, and that it follows robots.txt.
- How to limit or block it: the exact token to use in robots.txt, and an example of disallowing it for a whole site or a single directory.
- How to contact a person: an address that is monitored, and how soon a reply can be expected.
- Where requests come from, if your scraper uses fixed network addresses, so owners can confirm that traffic using your name is really yours.
- When the page was last updated.
Describe only what the scraper really does. If the page says it visits once a day and the logs show hourly visits, the page becomes evidence against you rather than for you. When behavior varies by source, say so and give the upper limit, for example that no host receives more than one request every few seconds, rather than an average that hides the busiest case.
Check: give the page to a colleague who has never seen the project and ask them to write the robots.txt lines that would block the scraper from one directory. If they need to ask you anything, the page is missing it.
Step 5: Make the contact route work
A contact address nobody reads is worse than none, because the owner wrote to you, got silence, and now has a reason to escalate. Route the mailbox to a small group, give it an owner, and set a response target, such as one business day for an acknowledgment.
Prepare for the usual requests: slow down, stop collecting a section, stop entirely, use a feed or API instead, or explain what was collected. For each one, decide in advance who can approve the change and how fast it can be applied. A request to stop should be honored by pausing collection from that host first and discussing afterwards.
Check: send a test request asking the scraper to stop visiting one directory of a test site, and time how long it takes until the scraper actually stops. That number is the honest answer to put on the information page.
Step 6: Keep the identity consistent across machines
Owners recognize traffic by its pattern as much as by its name. If the scraper runs on several machines, all of them should send the same user agent, follow the same pace for each host combined rather than per machine, and, where possible, send requests from a small, fixed set of network addresses that you list on the information page.
Avoid any setup that spreads requests across many changing addresses or alternates user agents. Even when the intent is only to balance load, it looks like an attempt to avoid limits, and it makes the information page useless for verification.
Check: from your own server logs, filter one day of traffic by your token and confirm that it comes from the addresses you published, and that the combined request rate for one host stays within the pace you described.
Step 7: Test that a block actually works
An information page that tells owners how to block you is only honest if the block takes effect. Set up a test site you control, add a robots.txt group that disallows your token from one directory, and run the scraper against it.
Check: the scraper should not request any page in that directory, should log that it skipped them because of robots.txt, and should pick up a change to the file within the re-check interval you state on the page. Then remove the rule and confirm that collection resumes.
Common mistake: caching robots.txt for days or reading it only at startup, so a new rule is ignored until the next deployment. Owners who add a rule and still see traffic will conclude that you ignore it.
Step 8: Version the scraper and keep the page current
Raise the version number in the user agent string whenever the scraper's behavior changes in a way an owner would notice: a new section collected, a different visit frequency, a change in request rate. Record each version and what changed on the information page or in a linked changelog. When an owner writes about a specific day's traffic, the version in their logs tells you which behavior they saw.
The same applies when someone else collects for you. If a provider such as ScrapingIsNotACrime runs part of the collection, ask how its traffic identifies itself and which contact route site owners should use, and keep that answer with your own records. Identification works alongside the other polite habits described in ethical and effective social media scraping best practices.
Check: compare the version in the last day of your server logs with the latest version listed on the information page. They should match.
