A scraper that returns empty fields without throwing an error is almost always looking in the wrong place, or receiving a different page from the one you inspected in your browser. The usual causes are content loaded by JavaScript, markup that changed, content that loads on scroll, a consent or interstitial page, an error response that was treated as a normal page, and encoding problems. The fastest diagnosis is to save the raw response your scraper actually received and compare it with what you expected. Most of the time, the answer is visible in that file within a minute.
The symptom: the job succeeds, the data does not
The pattern looks like this: the run finishes, the row count is roughly normal, and one or more fields are blank, or every row holds the same value. Sometimes it affects a single source; sometimes every page of one site at once. Before changing any code, collect three things for one failing URL:
- the HTTP status code your scraper received;
- the raw HTML or JSON body it received, saved to a file;
- the selector or path your extractor uses for the empty field.
Open the saved file and search for a value you can see on the live page, such as the product title. Whether it is there, and where, points to the cause.
Cause 1: the content is loaded by JavaScript
If the value appears in your browser but not in the saved response, the page probably builds that part of its content with JavaScript after loading. Your HTTP client receives the initial HTML shell, without the data.
Fix: open the browser's developer tools, go to the Network tab and reload the page. Look for a request that returns the data, often JSON. Before using it, check whether the site documents it as a public API, and read the terms of service: an undocumented internal endpoint is not an invitation to use it. Many pages also embed the same data as structured JSON inside the HTML, for example in a script tag of type application/ld+json for product or article details, which is easier to parse and more stable. If neither option exists and the terms permit automated access, a headless browser can render the page, at a much higher cost per page, so slow the pace accordingly.
Cause 2: the markup changed
If the value is present in the saved response but your selector finds nothing, the site has probably changed its structure. Class names generated by front-end build tools change with every deploy, and a selector such as a div with a long random-looking class breaks without warning.
Fix: rewrite the selector against something more meaningful, in this order of preference: structured data in the page, stable attributes such as itemprop or data attributes that describe content, semantic tags and headings, and only then class names. Test the new selector on several pages of different types, since category, product and sale pages often differ.
A quick way to confirm this cause: compare a saved response from a run that worked with one from today. If the value sits in both files but inside different tags or attributes, the markup changed. Keeping one known-good page per template makes this comparison possible at any time, and it doubles as a regression test after the fix.
Cause 3: lazy loading and pagination
If the first few items have data and later ones are empty, or a list stops at 20 items when the page shows 200, the page loads more content as the user scrolls or clicks a button.
Fix: find how the page requests the next batch, usually a numbered page parameter or a next-page link, and follow it at your normal polite pace. Check whether the site offers a sitemap or a feed that lists the same items more efficiently. Set a stop condition, such as a maximum page count or the absence of new items, so a pagination change does not send the scraper into a loop that fetches the same page repeatedly.
Cause 4: a consent page or regional variant
If the saved response contains a cookie consent message, a region selector or a different language, the site served a variant page. This depends on the location your requests come from and on whether cookies are set.
Fix: make sure the extractor recognizes this page type and does not record it as a normal result. Where the site offers a regional URL or a language parameter for public content, request that version directly. Consent choices are made by the person using the site; if the content you need is only available after accepting terms or creating an account, treat that as a signal that it is out of scope for automated collection and look for an official API or data agreement instead.
Cause 5: an error or block page treated as success
If the saved response is short and the status code is 403, 429 or 503, or the body is a generic message, the server did not serve the page. Some sites return such messages with a 200 status, which is why checking the body matters.
Fix: this is not a parsing problem, and it should not be solved by working around the site. A 429 or 503 means slow down: honor any Retry-After header, reduce concurrency and lengthen delays. A 403 or a challenge page means the site does not want this traffic, and the right response is to stop collecting from that host, re-read its robots.txt and terms, and look for an official API or contact the owner. Disguising traffic or rotating addresses to get past a block is outside what a legitimate project does. The guide to rate limiting, crawl rates and backoff explains how to slow down after these responses.
Cause 6: encoding and whitespace
If the field is not empty but looks empty, contains only spaces, or shows broken characters, the problem is text handling. Non-breaking spaces, zero-width characters and wrong character encoding are the usual suspects.
Fix: decode the response using the charset the server declares in its headers or meta tag, normalize Unicode and replace non-breaking spaces before trimming. Then check how many values become empty after trimming: that is your true null rate.
How to prevent empty fields from reaching your data
Prevention comes down to detecting the problem in the run, not in the report:
- Save raw responses for a sample of pages on every run, so diagnosis starts from evidence.
- Classify every response as normal page, error, consent or interstitial, and empty result before extraction.
- Track null rates per field and per source and alert when they move away from their usual level.
- Keep a test set of saved pages with known correct values and run the extractor against it after each change.
- Fail loudly: a run where a required field is empty for more than a set share of records should stop and alert, not publish.
The data quality checks to run on every scraped dataset help catch empty fields before the data is used. If maintaining extractors across many changing sites takes more time than the project can spare, handing the collection to a managed service such as ScrapingIsNotACrime is one option; the same monitoring rules should apply to whatever it delivers.
