How to Read a Website's Terms of Service Before You Scrape It: A Practical Guide
Robots.txt says which paths crawlers may visit; the terms of service say whether automated collection is allowed at all and what y...
Posts
How to Use a Website's XML Sitemap to Plan a Scraper Instead of Crawling Every Link
A sitemap tells you which pages exist and when they changed. Here is how to find it, filter it, respect robots.txt and fetch only what changed.
// Posts
Posts
Planning a Competitor Price Monitoring Project: A Hypothetical Retailer, Step by Step
Follow a hypothetical cookware retailer as it scopes a price monitoring project, checks permissions, matches products and decides...
6 Myths About Collecting Public Personal Data Under the GDPR and CCPA
Public does not mean free to use, and removing names does not make data anonymous. Six common misconceptions about scraping person...
Scraper Returning Empty Fields? Common Causes and How to Fix Each One
When a scraper runs without errors but the fields come back empty, the cause is usually one of six things. Here is how to tell whi...
9 Data Quality Checks to Run on Every Scraped Dataset Before You Use It
Schema, null rates, duplicates, units, ranges and drift: nine checks that catch the silent errors scraped data picks up between th...
Official APIs vs Scraping Public Pages for Social Media Data: How to Choose
Official APIs, researcher programs, scraping public pages and licensed datasets each fit different projects. Compare them on the s...
Rate Limiting in Web Scraping Explained: Server Limits, Crawl Rates and Backoff
Rate limiting works in two directions: the limits a server enforces and the pace a scraper sets for itself. Here is what each term...
How to Read a robots.txt File and Apply Its Rules Before You Scrape a Website
Find the file, match your user agent to the right group, resolve Allow and Disallow conflicts, and turn the result into rules your...
Ethical Web Scraping: A 9-Point Checklist for Responsible Data Scraping
Is web scraping ethical? It can be. Nine checks to run before and during every project, each with a way to verify it and the mista...
Is Web Scraping Legal? 6 Myths About Scraping Websites and Instagram
Public does not mean free to use, and scraping is not illegal by default. Six common myths about web and Instagram scraping, and w...