When scraped text shows sequences such as é where an é should be, ’ instead of an apostrophe, or a black diamond with a question mark, the bytes were decoded with the wrong character encoding somewhere between the server and your output. The fix is to find the one step where that happened, which you can do by looking at the raw bytes of a single damaged value. By the end of this page you will be able to read the symptom, locate the faulty step, repair it at the source, and add a check that catches it next time.
The symptom and what each pattern tells you
The damaged text itself is the best clue, because each pattern points to a different cause:
- Two odd characters per accented letter, such as é for é, ü for ü, or ’ for a curly apostrophe: text encoded as UTF-8 was decoded as a single-byte encoding, usually Windows-1252 or Latin-1.
- Four or more odd characters per letter, such as é for é: the same mistake made twice.
- The replacement character, a diamond with a question mark, or a plain question mark where a letter was: bytes that were not valid in the encoding used for decoding were thrown away.
- Literal entity text such as é or & in your output: HTML entities were not decoded, or were encoded twice.
Before changing code, take one affected record and look at three versions of the same value: the bytes in the saved raw response, the text after your parser, and the text in your final output. The first version that looks wrong tells you which step to fix.
Cause 1: the wrong encoding was used to decode the page
If the raw response bytes are correct UTF-8 but the parsed text shows é patterns, the decoding step chose the wrong encoding. Servers declare the encoding in the Content-Type header, pages can repeat it in a meta tag, and the two sometimes disagree or are missing. Many HTTP clients fall back to a default, often Latin-1, when the header names none, even when the page's meta tag says UTF-8.
Fix: decode the response yourself instead of relying on the client's default. Browsers follow a defined order: a byte order mark at the start of the body wins, then the charset in the Content-Type header, then a meta charset declaration near the top of the document. Applying the same order makes your scraper see what visitors see. If none is present, record that for the source and choose the encoding explicitly per source after checking a sample.
Check: decode one saved response using the rule above and search for a word with accents that you can see on the live page. It should appear intact.
Cause 2: correct text was encoded or decoded twice
If the parsed text is fine but the output shows é patterns, a later step treated already decoded text as bytes and decoded it again, or encoded UTF-8 text a second time. This often happens when data passes through several systems, such as a queue, a database and an export script, and one of them assumes a different encoding.
Fix: trace the value through each step and find where the length of the string grows without the content changing. Set the encoding explicitly at every boundary where text becomes bytes or bytes become text: when writing files, when connecting to the database, and when reading exports. Avoid repairing damaged output with search-and-replace tables for common sequences; they fix the frequent cases and leave the rare ones broken.
Check: write a test record containing é, ü, ñ, a curly apostrophe and a character outside Latin scripts, push it through the whole pipeline, and compare the output byte for byte with the input.
Cause 3: bytes were replaced and the information is gone
If the output shows replacement characters, a decoder met bytes it considered invalid and substituted them. Unlike the patterns above, this cannot be reversed from the damaged text, because the original bytes were discarded.
Fix: go back to the raw response and decode it again with the correct encoding. This is only possible if raw responses were kept, which is one reason to save them for at least a sample of pages on every run. If they were not kept, the affected pages have to be fetched again, at the usual polite pace, which the guide on how to read a robots.txt file and apply its rules covers.
Check: count replacement characters per source after each run. In clean text the number should be zero.
Cause 4: entities and escapes were not decoded
If the text contains é, é or similar, the extractor took the raw markup of an element instead of its text, or read a value from an attribute or embedded data without decoding it. If it contains &, something encoded the text again on the way out.
Fix: extract text through the parser's text accessor rather than from raw markup, and decode escapes in embedded data with a proper parser instead of string replacement. Encode entities only at the point of producing HTML, never in stored data.
Check: search the output for an ampersand followed by letters or a number sign and a semicolon. Any match in a text field is worth a look.
Cause 5: the data is fine but the viewer is wrong
Sometimes every step is correct and the problem appears only when someone opens the file. Spreadsheet programs may assume a regional single-byte encoding when opening a CSV file and show é patterns for perfectly valid UTF-8.
Fix: confirm by opening the same file in a plain text editor set to UTF-8. If it looks right there, the file is fine. Agree on the encoding in the delivery specification, and if consumers open files in spreadsheets, either import through the program's text import option with UTF-8 selected, or write the file with a UTF-8 byte order mark, which many spreadsheet programs use as a hint.
How to keep encoding errors out of your data
Encoding errors come back whenever a new source, library or export is added, so prevention is about checks that run every time:
- Save raw responses with their headers for a sample of pages, so any value can be decoded again.
- Decode explicitly following the byte order mark, header, meta order, and record the encoding used per page.
- Set the encoding at every boundary: files, database connections and exports.
- Monitor three counts per source: replacement characters, typical mis-decoding sequences such as à followed by another character, and leftover entities.
- Keep a test page with accented and non-Latin text in your saved test set, and run it after every change to the pipeline.
The same counts make a simple acceptance check for delivered data. If ScrapingIsNotACrime or another provider supplies part of the collection, run them on each delivery as part of the data quality checks for every scraped dataset.