The first decision in any web data project is not which scraper to build but whether you need one. Many websites and platforms already offer their data through an official channel: a documented API, a feed, a bulk download, an open data portal, a partner feed or a license. An official channel usually comes with clear terms, a stable format and no guesswork about acceptable load, which makes it the better choice whenever it covers the fields and the freshness the project needs. The six sources below are worth checking, in roughly this order, before any page is scraped. Each one is assessed against the same four criteria: freshness, completeness, terms and the effort needed to use it.
1. A documented public API
An API is the most direct official route: the site exposes its data in a structured format, usually JSON, through documented requests. Look for links labelled developers, API or partners in the site footer, or search the site's help pages.
- Freshness: usually close to real time.
- Completeness: depends on what the provider chooses to expose. APIs often leave out fields that appear on the page, or offer them only on paid tiers.
- Terms: explicit, often with rate limits, quotas, attribution rules, limits on storing data and rules about displaying it.
- Effort: registration and keys, then a client that respects the quota.
On social platforms, the API is often the only route the terms allow, and access rules change over time, so check the current developer terms rather than relying on what applied a year ago. If an API exists but lacks a field you need, ask the provider before deciding to scrape for it; the answer is sometimes a different endpoint or tier. A common mistake is to compare an API with scraping on cost alone: a free scraper that breaks every few weeks and runs outside the terms is rarely cheaper than a paid tier with a stable contract. The article on mining social media data legally and effectively covers how access routes shape what a social project can collect.
2. RSS and Atom feeds
Feeds are lists of recent items, such as articles, posts, listings or updates, published in a structured XML format. News sites, blogs, podcasts, job boards and many public bodies still publish them. Look for a feed link in the page head (a link element with an RSS or Atom type), a feed icon, or common paths mentioned in the site's help pages.
- Freshness: good; feeds are designed to be checked regularly.
- Completeness: usually limited to the most recent items, and sometimes to a title and summary rather than the full text.
- Terms: feeds are offered for automated reading, but the content is still subject to the site's terms and to copyright, especially if you republish it.
- Effort: low. A feed reader or a short parser is enough.
Feeds work best for monitoring new items. Because they show only the latest entries, a project that needs history has to start reading early and store each item, or combine the feed with another source for the backlog.
Suppose, hypothetically, a team tracks new public tenders from 30 municipal websites, and 22 of them publish a feed of new notices. Reading those 22 feeds a few times a day covers most of the sources with a handful of light requests, leaving page collection only for the eight that have none. Feeds also make a useful cross-check: if an item appears in the feed but not in your page-based dataset, the extractor missed something.
3. Bulk downloads and data exports
Some sites offer their data as downloadable files: full database dumps, periodic CSV exports, archives of past records or an export button on search results. Reference sites, statistical agencies, public registries and community projects are typical places to find them, and a file spares both sides the cost of crawling pages.
- Freshness: as current as the publication schedule, which can be daily, monthly or less often.
- Completeness: often the most complete option, including history and fields not shown on pages.
- Terms: usually stated alongside the download, often with a license.
- Effort: the work moves from collection to processing large files and loading them correctly.
If a site offers a bulk download and also asks crawlers to stay away from its pages, use the download. Downloading a single file costs the site far less than requesting thousands of pages, and the data is usually cleaner.
Before choosing between a download and live collection, compare the download's publication schedule with how fresh the project actually needs the data to be. A monthly export is enough for trend analysis but not for daily price checks. In that case, a sensible split is to load the bulk file for history and use an API or a light, polite collection only for the recent changes, if the terms allow it.
4. Open data portals
Governments, cities, international organizations and research bodies run portals that publish datasets under open licenses: business registries, procurement notices, transport timetables, statistics, environmental readings. When the information you want originates from a public body, its portal may hold the original data that other websites republish.
- Freshness: varies by dataset; each one should state its update frequency.
- Completeness: often the authoritative source, though some datasets omit fields for privacy reasons.
- Terms: an explicit license, often requiring attribution.
- Effort: low to moderate; many portals offer both files and an API.
Read the license and the dataset documentation together. A license may allow commercial reuse while the documentation warns that some fields are estimates or that records are revised later, which affects how you can use the data.
A common mistake is to scrape an aggregator that republishes public-sector data instead of going to the portal behind it. The aggregator may have its own terms, add its own errors and lag behind the original. Following the trail back to the publishing body usually gives cleaner data under a clearer license.
5. Product and partner feeds
Retailers, travel sites, marketplaces and job boards often maintain structured feeds of their catalog or listings for partners: comparison sites, affiliate publishers, advertising platforms and aggregators. These feeds are not always public, but many are available by joining an affiliate or partner program, or by asking.
- Freshness: typically updated at least daily, because partners depend on accurate prices and stock.
- Completeness: strong for the fields partners need, such as identifiers, prices, availability and images; weaker for reviews or content outside the catalog.
- Terms: set by the program agreement, which may limit use to promoting the products or require links back.
- Effort: an application or a conversation, then routine file processing.
Check whether the program's terms fit your purpose. A feed provided for promotion may not permit use for competitor analysis, and using it that way can breach the agreement even though the data itself is accurate. Keep a copy of the program terms you accepted, with the date, alongside the dataset.
6. A data license or a direct agreement
When none of the channels above fits, the site may still be willing to provide data on request. Many organizations sell or license their data, and smaller sites sometimes agree to a simple arrangement: permission to collect at a stated pace, a periodic export, or access to an internal endpoint, in exchange for attribution or a fee.
- Freshness: negotiated.
- Completeness: negotiated, and often better than what pages show.
- Terms: written and specific to your project, which removes the ambiguity of general terms of service.
- Effort: time to find the right contact and agree on terms.
A short, specific request works better than a general one: what data, for what purpose, how often, how it will be credited and how much load collection would create. If the answer is no, that answer is also useful, because it tells you the source should not be scraped either.
If none of the six channels exists and the site's robots.txt and terms permit collection, scraping public pages at a polite pace is the remaining route. Whether you build that collection in-house or ask a provider such as ScrapingIsNotACrime, record which official channels you checked and why each did not fit; that record is part of a defensible project. The guide to rate limiting, crawl rates and backoff covers how to keep that pace polite once collection is the chosen route.