By the end of this tutorial you will have a scraper plan that starts from the website's XML sitemap instead of following every link on every page. The scraper will find the sitemap, read the list of URLs it contains, keep only the pages the project needs and allowed by robots.txt, and on later runs fetch only pages whose content changed. For most sites that publish a sitemap, this means far fewer requests than crawling, a clearer view of what exists, and an easy way to notice pages that were removed. Each step below ends with a check you can verify before moving on.
Step 1: Find the sitemap
Start with the site's robots.txt file at the root of the domain. Sites often declare their sitemaps there on lines beginning with "Sitemap:" followed by a full URL, and there may be several. If robots.txt lists none, try the common locations at the root, such as /sitemap.xml and /sitemap_index.xml. Sites built on popular content management systems often generate one automatically at a predictable path.
Do not confuse an XML sitemap with an HTML sitemap page, the human-readable list of links some sites keep in the footer. The XML file is meant for machines and is much easier to process reliably.
Check: the file you found returns a 200 status and is XML whose root element is either urlset or sitemapindex. If the file name ends in .gz, it is compressed and must be decompressed before parsing.
Common mistake: treating a missing sitemap as a reason to crawl aggressively. If there is no sitemap, crawl slowly from category or archive pages, and consider whether the site offers a feed or an API instead.
Step 2: Tell an index from a URL list and walk the index
The sitemap protocol has two kinds of files. A URL set lists pages directly, each inside a url element with a loc element holding the address. A sitemap index lists other sitemap files, each inside a sitemap element with its own loc. Large sites almost always use an index, because a single sitemap file is limited to 50,000 URLs and 50 MB uncompressed under the protocol, so they split pages into many files, often by content type or by date.
Read the index first and look at the names of the child sitemaps before fetching all of them. Names such as products, posts, categories or a year and month usually tell you which files hold the pages you need, which can save fetching dozens of irrelevant files.
Check: record how many child sitemaps the index lists and how many URLs each one you fetch contains. A child file with zero URLs or a parse error should be logged, not silently skipped.
Common mistake: assuming the index is one level deep. It usually is, but write the walker so that it notices when a child file is itself an index, and set a limit on depth so a misconfigured site cannot send it in circles.
Step 3: Filter the URLs to what the project needs
A sitemap lists everything the site wants search engines to know about, which is usually much more than a project needs. Filter the list down by path patterns that match your target pages, for example everything under a products path but nothing under a blog or help path.
Then apply robots.txt to the filtered list. A URL appearing in a sitemap is not permission to collect it: the sitemap and the robots.txt rules are separate, and a site can list a page for search engines while disallowing other crawlers. Test each remaining URL against the rules for your user agent and drop the disallowed ones. The site's terms of service apply to the whole project regardless of what either file says.
Normalize the URLs as you go: keep one scheme, usually https, and one form of trailing slash, and strip tracking parameters if the sitemap includes them. Otherwise the same page can enter your queue twice.
Check: take a random sample of twenty filtered URLs and open them. All twenty should be the kind of page you want. If some are not, tighten the path pattern before going further.
Step 4: Store each URL with its lastmod value
Each url entry may carry optional fields: lastmod, the date the page last changed; changefreq, a hint about how often it changes; and priority, its relative importance within the site. Of these, lastmod is the one worth using. Changefreq and priority are hints, are often left at default values, and should not drive your schedule.
Create a table keyed by the normalized URL with at least these columns: the URL, the lastmod value from the current sitemap, the lastmod value at the time you last fetched the page, the date you last fetched it and the date the URL was last seen in the sitemap.
Check: count how many URLs have a lastmod value at all. If most do not, the site does not provide change dates, and you will need a fixed revisit schedule instead of the change detection in the next step.
Step 5: Fetch only pages that changed
On the first run, fetch every filtered URL at your normal polite pace: one request at a time or very low concurrency, a pause between requests, and a descriptive user agent. On later runs, compare each URL's current lastmod with the value you stored when you last fetched it, and fetch only the pages where it moved forward, plus any new URLs.
Suppose, as a hypothetical, a catalog of 8,000 product pages where about 300 change on a typical day. A crawler revisiting everything daily makes 8,000 requests; a sitemap-driven scraper makes the sitemap requests plus roughly 300 page requests. The load on the site and the run time both drop accordingly.
Check: after a run, spot-check five pages the scraper skipped as unchanged by fetching them once and comparing them with your stored copies. They should match in the fields you collect.
Common mistake: trusting lastmod blindly. Some sites set every lastmod to the time the sitemap was generated, so every page looks changed on every run. If all or nearly all lastmod values are identical, treat them as unreliable for that site and fall back to a revisit schedule, for example refreshing a fixed share of pages each day.
Step 6: Detect pages that were removed
When a URL that appeared in previous sitemaps is missing from the current one, the page may have been removed. That is useful information for many projects, such as a product being discontinued, but a sitemap can also be incomplete for a day because of a generation problem on the site's side.
Mark a URL as possibly removed when it is missing from the sitemap, then confirm by requesting the page once. A 404 or 410 status confirms removal; a redirect to another product suggests a replacement worth recording. Only mark the record as removed after that confirmation.
Check: if more than a small share of URLs disappear at once, stop and look at the sitemap before marking anything. A sudden drop is more often a broken sitemap than a mass deletion.
Step 7: Re-read sitemaps politely on a schedule
Sitemaps themselves are files on the site's server, and large indexes can hold dozens of child files. Fetch them no more often than your project needs, usually once per run. Use conditional requests: store the ETag or Last-Modified header from each sitemap response and send them back on the next request as If-None-Match or If-Modified-Since. If the file has not changed, a server that supports this answers with a 304 Not Modified status and no body, and you can skip parsing that file.
Respect any rate signals while reading sitemaps exactly as you would for pages: back off on 429 or 503 responses and honor a Retry-After header. The same pacing principles appear in the guide to rate limiting, crawl rates and backoff.
Check: on a second run with no site changes, most sitemap requests should return 304, and the scraper should fetch almost no pages.
Step 8: Compare the sitemap with the site to catch gaps
A sitemap is the site's own list, and it is not always complete. Pages can be missing because of how the site generates the file. Once a month, take a small sample of listing or category pages, collect the product or article links they show, and check whether each one appears in your sitemap-derived table.
If the overlap is high, the sitemap is a reliable starting point and the sample check can stay infrequent. If many pages are missing, combine both approaches: use the sitemap for change detection and a slow, limited crawl of listing pages to find what the sitemap leaves out. If the collection is handled by a managed service such as ScrapingIsNotACrime rather than in-house, the same questions apply: ask how sitemaps, change detection and removed pages are handled for each source.
Check: record the overlap percentage for each sample. A falling overlap over time is an early sign that the site changed how it builds its sitemap.
