Skip to content
Home Posts About us Privacy Policy
How to Use ETag and Last-Modified to Skip Unchanged Pages When Scraping

How to Use ETag and Last-Modified to Skip Unchanged Pages When Scraping

Share ๐• in f W

By the end of these steps your scraper will ask each server whether a page has changed before downloading it again, and will skip the download when it has not. The mechanism is the HTTP conditional request: the scraper sends back a validator it received on the last visit, an ETag or a Last-Modified date, and the server answers either with the full page or with a short 304 Not Modified response that has no body. For a site whose pages rarely change, most revisits become a few hundred bytes instead of a full page, which reduces load on the site and time spent by your scraper. Conditional requests do not replace a polite crawl rate; they make each request at that rate cheaper. Every step below includes a check you can run, and the mistake that most often breaks it.

Step 1: Check whether the site sends validators

Pick ten pages from the site that represent the kinds of pages you collect, such as listings, detail pages and archive pages, and fetch each one once while recording the response headers. Look for two headers: ETag, which carries an opaque identifier for the current version of the page, and Last-Modified, which carries a date.

The check is to fetch the same ten pages again a minute later and compare. A useful ETag stays the same when the page has not changed. A useful Last-Modified date stays the same and lies in the past. If the ETag differs on every fetch, or the Last-Modified date is always the current time, the server generates them per response and they cannot be used. Note the result per type of page, because static detail pages and dynamic listing pages on the same site often behave differently.

Before any of this, confirm that the pages are in scope: the robots.txt rules and the site's terms of service decide whether you collect them at all. The guide on how to read a robots.txt file and apply its rules before you scrape a website covers that check.

Step 2: Store the validators with each page

For each URL, store the ETag exactly as received, including any W/ prefix and the quotation marks, and the Last-Modified value exactly as received, as text. Store them next to the copy of the page or the extracted record, together with the date and time of the fetch.

The mistake that usually breaks this step is parsing the Last-Modified date into your own date format and writing it back differently. Servers compare the value they receive, and a reformatted date, a different time zone or lost seconds can make every request look like a new one. Keep the original string and, if you need a date for analysis, store a parsed copy in a separate field.

Check the stored data by picking a few URLs and comparing the saved validators with what the server sends on a fresh fetch. They should match character for character when the page has not changed.

Watch redirects as well. If a requested URL redirects to another address, store the validators against the final URL, the one that actually returned the page, and keep a note of the redirect. Validators stored against the original address will be sent to the wrong resource on the next visit and never produce a 304.

Step 3: Send the conditional headers on the next visit

When revisiting a URL that has a stored ETag, send it in an If-None-Match header. When it has a stored Last-Modified value, send it in an If-Modified-Since header. Sending both is fine. Under the HTTP standard, a server that receives both evaluates If-None-Match and ignores the date, so the ETag decides when it is present.

Weak ETags, the ones starting with W/, work with If-None-Match, because that header uses a comparison that treats weak and strong forms of the same tag as equal. Send them back unchanged. Check this step by revisiting a page you know has not changed and confirming that the response status is 304.

If the check fails on a page whose ETag looked stable in Step 1, compare the request headers you send now with the ones you sent then. Some servers produce a different ETag for each compressed version of a page, so a scraper that changes its Accept-Encoding header between visits will never match. Keep request headers the same from visit to visit, and the comparison becomes reliable.

Step 4: Handle a 304 response correctly

A 304 response means your stored copy is still current. It has no body, so the scraper must not run the extractor on it or overwrite the stored record with empty fields. Instead, keep the existing record, update its last-checked time, and update any headers the 304 carries, such as a refreshed ETag or caching directives.

The common failure here is a pipeline that treats any response as a new page. Suppose, hypothetically, the extractor runs on a 304 and writes empty values: the next day's dataset shows half the catalog with missing prices, which looks like a site change rather than a scraper bug. Route 304 responses to a separate branch that touches only the metadata, and add a test that feeds a 304 through the pipeline and checks that the record is unchanged.

Step 5: Handle a 200 response and replace the validators

A 200 response to a conditional request means the page changed, or the server ignores the conditional headers. Extract the page as usual, store the new copy and replace the validators with the ones in the new response. If the new response carries no validators at all, delete the stored ones, so that you do not keep sending values the server has stopped recognizing.

Some servers return 200 with identical content every time. To catch that, compute a hash of the page content, or of the main content area if the page includes changing elements such as dates in a footer, and compare it with the previous hash. If they match, skip extraction and downstream processing. This does not save the download, but it saves the rest of the pipeline, and the share of identical 200 responses tells you whether the site's validators are worth relying on.

Step 6: Respect caching hints and choose revisit intervals

Responses may carry a Cache-Control header with a max-age value, or an Expires date, which state how long the server considers the response fresh. A polite scraper can treat these as a minimum interval before it revisits the URL at all, which saves even the small conditional request.

Combine this with what you know about each source. Pages that list a lastmod date in the site's XML sitemap can be checked against that date first, and only revisited when it moves. Pages that rarely change can be revisited weekly with a conditional request, while pages that change daily keep a daily schedule. The hypothetical job listings site that sets its scraping schedule shows how revisit schedules fit into a larger collection plan.

Step 7: Measure the effect per source

Log the status code and the response size for every request, and report per source and per run the share of 304 responses, the share of 200 responses with an unchanged hash, and the total bytes downloaded. A hypothetical source of 5,000 product pages where 80 percent come back as 304 on a daily run needs about 1,000 full downloads instead of 5,000, and the log shows it.

Use the figures to spot problems. A share of 304 responses that falls to zero overnight usually means the site changed how it generates validators or that your stored values were corrupted, and the scraper is quietly downloading everything again. A share of 304 responses close to 100 percent on a source that should change daily may mean the server answers 304 by mistake, so compare a sample of pages fetched without conditional headers. If you hand collection to a managed service, such as ScrapingIsNotACrime or another provider, these are reasonable figures to ask for, since they show how much load the collection places on each site.

Share ๐• in f โœˆ