Skip to content
Home Posts About us Privacy Policy
How to Extract Data from JSON-LD Structured Data Instead of HTML Selectors

How to Extract Data from JSON-LD Structured Data Instead of HTML Selectors

Share ๐• in f W

By the end of this tutorial you will have an extractor that reads the structured data a page publishes about itself, in the JSON-LD format, and turns it into clean records, with a fallback for pages that do not have it. Many product, article, event, recipe and job posting pages include a block of JSON-LD that lists the name, price, date, author or location in a predictable shape, using the shared schema.org vocabulary. Reading that block is often sturdier than pointing selectors at the visible layout, because the block follows a published vocabulary rather than the page design, so a redesign does not necessarily change it. The steps below go from checking whether the data exists to monitoring it in production, each with a check you can verify.

Step 1: Confirm the page is in scope before you parse anything

Structured data does not change what you are allowed to collect. Before writing the extractor, read the site's robots.txt and terms of service for the paths you plan to visit, exactly as you would for any other scraper. The tutorial on how to read a robots.txt file and apply its rules walks through the first part. JSON-LD only changes how you read a page you were already entitled to fetch, not whether you should fetch it.

Check: you have a short written note per site stating which paths are allowed, the pace you will use and the date of the review.

Step 2: Find out whether the page publishes JSON-LD

Open a sample page, view its source (not the inspector, which shows the page after scripts have changed it) and search for the text application/ld+json. Each match is a script element whose content is a JSON document. Some pages have none, some have one, and many have several: one for the organization, one for the breadcrumb trail, one for the main item.

Repeat this on at least one page of each template you plan to collect, such as a product page, a category page and a sale page. Templates on the same site often differ: product pages may carry full structured data while category listings carry none.

Check: for each template, you can say how many JSON-LD blocks it has and which one describes the item you need. Common mistake: checking only in the browser inspector. If the block is inserted by JavaScript, it will not appear in the raw HTML your HTTP client receives, and your extractor will find nothing.

Step 3: Extract and parse every block on the page

In your parser, select every script element whose type attribute is application/ld+json, take its text content and parse each one as JSON separately. Parse them separately because one broken block should not stop you from reading the others.

Expect three shapes. A block can hold a single object, a list of objects, or an object with a property called @graph that holds a list. Flatten all three into one list of objects per page before looking for anything. Also expect small faults: trailing commas, HTML entities inside strings, or comments that a strict JSON parser rejects. When a block fails to parse, log the URL and the error and move on rather than patching the text with ad hoc string replacements that may corrupt values.

Check: run the parser over your sample pages and print, for each page, the number of blocks found, the number parsed and the list of @type values. Every sample page should show the type you expect.

Step 4: Pick the object with the right type

Each object declares what it describes in its @type property: Product, Article or NewsArticle, Event, JobPosting, LocalBusiness, BreadcrumbList and so on. Select the objects whose type matches what you collect. Note that @type can be a single string or a list of strings, so test for membership rather than equality.

When a page has several objects of the same type, decide which is the main item. A product page may list related products as Product objects too. Useful tie-breakers are an @id or url property that matches the page's canonical URL, or the object that carries the most complete set of the properties you need.

Check: on a page with related items, your selector returns the main item and not a recommendation. Common mistake: taking the first Product object on the page, which is sometimes a carousel item.

Step 5: Map properties to your own fields

Write down a mapping from schema.org properties to the columns in your dataset, and keep it in one place. A product mapping might read: name to title, sku or gtin to identifier, offers.price to price, offers.priceCurrency to currency, offers.availability to stock status, and brand.name to brand. An event mapping might take name, startDate, endDate, location.name and location.address.

Watch the details that commonly cause wrong values:

  • Nested values can be objects or plain text. brand may be an object with a name or just a string; author may be a person object, an organization or a list of both.
  • Offers can be one object, a list, or an AggregateOffer with lowPrice and highPrice instead of a single price. Decide in advance how a price range is stored, rather than letting the code silently take the first value.
  • Availability is a vocabulary value, usually a schema.org address ending in InStock, OutOfStock or PreOrder. Map it to your own short codes and keep the original.
  • Prices may be strings or numbers. Read them as text and convert them with your normal price parser, so that decimal handling stays consistent with the rest of the dataset.
  • Dates follow ISO 8601 but may omit the time zone. Record whether a time zone was present instead of assuming one.

Check: for five sample pages, put the mapped record next to the visible page and confirm each field by eye.

Step 6: Compare structured data with the visible page

Structured data is published by the site, but it is not always in sync with what visitors see. A sale price may show on the page while the JSON-LD still carries the regular price, or an event may be rescheduled on the page before its structured data is updated. Before relying on it, measure how often the two agree.

Take a sample of pages, extract the key fields from both the JSON-LD and the visible HTML, and count disagreements per field. If price agrees on nearly every page but availability often does not, use structured data for price and keep a visible-page selector for availability. Repeat the comparison on a small sample in later runs, because a site can change how it fills its structured data without changing anything visible.

Check: you have a table of agreement rates per field and per site, and a decision recorded for each field. The data quality checks to run on every scraped dataset cover more ways to measure accuracy across sources.

Step 7: Handle microdata, RDFa and pages without structured data

Some sites publish the same schema.org vocabulary as microdata or RDFa, which are attributes inside the visible HTML (itemprop, itemscope, property) rather than a separate JSON block. Libraries for most languages can read both into the same kind of object you built in Step 3, so the mapping from Step 5 can be reused.

For pages with no structured data at all, keep an HTML selector extractor as a fallback. Record in each output row which method produced it: JSON-LD, microdata or selectors. That one column makes later debugging much faster, because a sudden quality drop can be traced to the method rather than to the whole source.

Check: every record in a test run carries an extraction method, and the share of each method per site matches what you found in Step 2.

Step 8: Monitor the structured data over time

Structured data breaks less often than selectors, but it does break: a site can drop a property, move from a single offer to a list, or remove the block from a template. Add three checks to every run:

  • the share of pages per site where a JSON-LD block of the expected type was found;
  • the null rate of each mapped field;
  • the share of records produced by the fallback method.

A drop in the first number or a rise in the third usually means a template changed. Alert on it and look at a saved page before changing the code. Save the raw block alongside each record for a sample of pages, so you can re-run the mapping after a fix without fetching the pages again.

Whether you run this extractor yourself or receive records from a provider such as ScrapingIsNotACrime, ask for the extraction method and the raw structured data to be kept with each record; the guide to receiving scraped data as CSV, JSON Lines, database loads or an API discusses what else to agree for any delivery.

Check: deliberately remove the JSON-LD from one saved test page and run the extractor against it. The fallback should take over, the method column should change, and the per-site share in your monitoring should move enough to trigger the alert.

Share ๐• in f โœˆ