Trialing a Web Data Provider Before Signing: A Hypothetical Market Research Agency
A hypothetical agency puts two data providers through a three-week trial with the same brief. What it measured, the event that dec...
A hypothetical agency puts two data providers through a three-week trial with the same brief. What it measured, the event that dec...
Weights, ratings, phone numbers, addresses, stock flags, categories and names all look valid row by row while meaning different th...
Counts, posts, direct replies or whole threads: each depth adds context, cost and people to your dataset. A comparison on six crit...
Run identifiers, request counters, parse outcomes, a crash-proof summary and evidence of polite behavior: the seven steps to logs...
Someone in your scraped dataset asks what you hold about them, or asks you to delete it. Here is what generally applies, how to fi...
Conditional requests let a scraper ask whether a page changed before downloading it again. Here is how to set them up step by step...
Ending a scraping project means more than stopping the scraper. Follow a hypothetical retailer through inventory, shutdown, retent...
Moving from scraping to an official API changes how data arrives, not the rules that apply to it. Here are six common misconceptio...
Missing records never show up as errors. Here is how to estimate what share of the target population a scraped dataset actually co...
Identifiers only, aggregated data, controlled access or open publication: compare four ways to share a social media dataset by ter...
Merged cells, stacked headers, totals rows and footnote marks turn tidy-looking tables into misaligned data. Here is how to handle...
A data protection impact assessment is required when processing personal data is likely to be high risk. Here is what that means f...
Public personal data does not come with a lawful basis attached. How the legitimate interests test works for web data projects, wh...
A price of 1.299 can mean almost thirteen hundred or just over one, depending on the site. How to turn prices from many sources in...
A slow scraper is rarely slow because of the site's limits. Measure where the time goes, then fix waiting, connections, rendering,...
Follow a hypothetical tourism office as it turns a vague request for regional event listings into a written data specification, an...
Before building a scraper, check whether the site already offers the data through an API, a feed, a bulk download, an open data po...
When scraped data includes personal data, someone has to answer for it. How the GDPR's controller and processor roles and the CCPA...
Many pages already describe their products, articles, events and job postings as structured data. Here is how to find it, read it...
A price field that starts including fees passes every check. Follow a hypothetical rent index from an unexplained jump to the fix,...