Skip to content
Home Posts About us Privacy Policy
7 HTML Table Quirks That Break Scraped Data and How to Handle Each

7 HTML Table Quirks That Break Scraped Data and How to Handle Each

Share ๐• in f W

By the end of this list you will have a table extractor that turns HTML tables into rows where every cell sits under the right column, with totals, notes and missing values kept apart from real data. HTML tables look like the easiest thing on a page to scrape, because the structure is already there. The trouble is that tables are built for people to read, and the shortcuts that make them readable, such as merged cells, stacked headers and footnote marks, break a naive extractor that reads cells in order. The seven quirks below cause most of the damage. Before handling any of them, check whether the site offers the same table as a download, and confirm that its robots.txt and terms of service allow automated collection of the page.

1. Merged cells spanning rows or columns

A cell with a rowspan or colspan attribute covers more than one position in the grid. Read the cells in order and every value after the merge shifts one column to the left, so a price lands under the quantity column without any error being raised.

The fix is to build the grid explicitly. Walk the rows, keep a record of which positions are already occupied by a cell from an earlier row, and place each new cell in the next free position. When a cell spans several positions, copy its value into each of them. Suppose a hypothetical schedule table lists a region once with a rowspan of three, followed by three city rows: after expansion, all three rows carry the region name, which is what the analyst needs.

Check it by counting cells per row after expansion. Every row in the body should have the same count as the header. A row with fewer cells means a span was missed. A second check is to pick one merged value on a saved page and confirm by eye that it appears in each row it covers on screen.

The mistake that usually breaks this step is treating a span value written as text, or a span larger than the table, as valid. Read the span as a whole number, treat anything missing or unreadable as one, and cap it at the remaining number of rows or columns so a malformed attribute cannot stretch a single cell across the rest of the table.

2. Headers stacked over two or more rows

Many statistical and financial tables use a top header row for groups, such as years, and a second row for measures, such as revenue and margin. An extractor that only reads the last header row ends up with two columns called revenue and no way to tell 2024 from 2025.

Apply the same grid expansion to the header rows, then join the labels from top to bottom for each column, producing names like "2025 revenue". Keep the original labels as separate fields in a mapping table, so a later change to the grouping can be spotted. The mistake that usually breaks this step is treating the first body row as a header, or the last header row as data, when the page does not use thead and th consistently. Decide which rows are headers by their content and position on a few saved pages, not by tag alone.

For the check, compare the joined column names with a list you expect for that source. A hypothetical annual report table with three years and two measures should yield exactly six data columns plus the row label. If the next run yields eight, a year was added or the grouping changed, and the output schema should be reviewed before the data is loaded rather than after.

3. Group labels and repeated headers inside the body

Long tables often insert a row that spans the full width and names a section, such as a category or a month, followed by the rows that belong to it. Some also repeat the header row every few dozen lines so readers can keep their place.

A section row should become a value, not a record. Detect rows with a single cell spanning all columns, store that text as the current group, and add it as a column to each following row until the next group row appears. Repeated header rows should be recognized by comparing their text with the real header and dropped. Check the result by confirming that no output row has the header text as its values and that every row has a group when the table uses groups.

Watch for the opposite failure as well. Some tables style a normal data row with a bold first cell that looks like a group label but has values in the other columns. Use the span across all columns, or empty remaining cells, as the signal for a group row, not the styling, since styling is the first thing to change in a redesign.

4. Totals and subtotal rows

A totals row is useful on the page and harmful in a dataset. If it stays in, any later sum counts everything twice, and averages are skewed by a value that is not an observation.

Do not simply delete the last row. Flag rows whose first cell says total, subtotal, all or similar words in the page's language, and move them to a separate field or table. Then use them as a free check: sum the detail rows yourself and compare the result with the published total. Suppose a hypothetical table has twelve monthly values and a total of 1,840; if your twelve values add up to 1,790, one month was missed or misread. A small difference may come from rounding on the page, so allow a tolerance that matches the number of decimals shown.

5. Footnote marks and annotations inside cells

Cells such as "12.4*", "1,200 (est.)", "35 p" or a number followed by a superscript note marker carry two pieces of information: the value and a qualifier. Parsing the whole cell as a number either fails or, worse, silently drops the qualifier.

Split each cell into a value and a note. Strip known markers, such as asterisks, daggers, superscript digits and bracketed letters, into the note field, then parse what remains. Collect the footnote text from below the table and attach it by marker, so a reader of the dataset can see that a value was provisional or estimated. Check this step by listing every distinct note marker found across the run. A new marker appearing is a sign that the site changed its notation and the parser needs a look.

6. Placeholders for missing values

Tables mark missing data in many ways: an empty cell, a dash, "n/a", two dots, "not available", a zero shown as a dash, or a value hidden for confidentiality. Each one means something different, and loading them all as zero or all as text corrupts the column.

Write a mapping for each source from placeholder to meaning: not applicable, not available, suppressed or genuinely zero. Store the value as null with a reason code rather than as a number. Read the source's own notes on symbols where it has them, because conventions vary between publishers and countries. The check is a count of each placeholder per column and per run. If a column that was mostly numbers suddenly shows a large share of dashes, the source may have stopped publishing that figure, which is worth knowing before anyone uses it. The data quality checks to run on every scraped dataset show how such checks fit into a wider accuracy routine.

7. Layout tables and tables nested inside tables

Older pages often use table elements for layout, and some data tables contain smaller tables inside cells. An extractor that grabs the first table on the page, or every table, can end up parsing the navigation menu or flattening a nested table into one long string.

Select the target table by what it contains, not by its position: look for a header row with the column names you expect, or for a caption or nearby heading that names the table. When cells contain nested tables, decide whether the inner table is part of the record, in which case parse it into a child list, or decoration, in which case take only its text. Check the selection by asserting that the expected headers are present on every run. If they are missing, stop and report rather than parsing whatever table happens to be first.

A related trap is a single logical table split across several pages or several table elements, for example one per letter of the alphabet. Collect the parts, confirm that they share the same headers, and only then append them, keeping the page or part each row came from as a field so gaps can be traced back.

Testing the extractor against a handful of saved pages that include each of these quirks catches most regressions before they reach the data. When tables are the main source for a recurring project, a managed provider such as ScrapingIsNotACrime is one option for taking on that maintenance, and the same checks apply to whatever it delivers. The guide to testing a scraper against saved pages shows how to build and maintain that set of pages.

Share ๐• in f โœˆ