The way scraped data reaches you decides more about the project's running cost than the scraper itself. Flat files such as CSV or JSON Lines are the cheapest to start with and the easiest to inspect; columnar files suit large analytical loads; direct database loads remove a manual step but tie the collection to your infrastructure; and an API or push delivery suits applications that need the latest records rather than a daily batch. The right choice depends on who consumes the data, how fresh it must be and who will fix the delivery when it breaks, so this comparison weighs each option against those questions instead of against a feature list.
The criteria used for every option
To keep the comparison honest, each option below is measured against the same six questions:
- Consumer: who opens the data first, an analyst with a spreadsheet, a data engineer with a warehouse, or an application.
- Freshness: whether a daily batch is enough or records must arrive within minutes.
- Volume and shape: how many records per delivery, and whether they are flat rows or nested structures such as a product with many variants.
- Schema change: what happens when a field is added, renamed or changes type.
- Failure handling: how a missed, partial or duplicated delivery is detected and repaired.
- Maintenance: who owns the moving parts, and how many hours a month they take.
To make the trade-offs concrete, consider a hypothetical mid-sized retailer that collects about 200,000 public product records a day from 30 competitor and marketplace pages, with a pricing analyst, one data engineer and an internal dashboard as consumers. That project reappears in each option.
Option 1: CSV files
CSV is the default for good reasons: every analyst can open it, every database can import it, and a person can read a sample without special software. For flat data, such as one row per product per day with a price, a currency and a stock status, it is often all a project needs.
Its weaknesses show up with real data. CSV carries no types, so a price, a postal code with a leading zero and a date are all plain text until something interprets them, and spreadsheet programs interpret aggressively: leading zeros vanish and long identifiers turn into rounded numbers. Commas, quotes and line breaks inside product descriptions must be quoted correctly, and the character encoding has to be agreed in advance. Nested data, such as a product with twelve size and color variants, has to be flattened into many rows or squeezed into a single cell.
In the retailer project: CSV works well for the analyst's daily extract of prices, and badly for full product records with variants and attributes.
Choose it when the data is flat, the consumers are people, and volumes stay in the range a spreadsheet or a simple import can handle. Write down the delimiter, the encoding, the quoting rule and the date format in the delivery specification, because every one of them will otherwise be guessed.
Option 2: JSON Lines files
JSON Lines puts one complete JSON record on each line of a file. It keeps nested structures intact, so a product can carry its variants, images and attributes as a single record, and because each line stands alone, a file can be processed one record at a time and a single malformed line does not ruin the rest.
JSON has fewer types than a database: strings, numbers, booleans, null, lists and objects, with no date or decimal type. Dates travel as text and need an agreed format, and prices are safer sent as strings or as integer amounts in the smallest currency unit, to avoid rounding surprises. Files are also larger than the same data in CSV, since every record repeats its field names, although compression reduces most of that difference.
In the retailer project: JSON Lines is the natural format for full product records, and the data engineer can load it into the warehouse without flattening anything first.
Choose it when records are nested, a data engineer or a script is the first consumer, and you want a format that survives new optional fields without breaking older readers.
Option 3: Columnar files
Columnar formats such as Parquet store data column by column with a typed schema embedded in the file. They compress well, and analytical queries that read only a few columns, such as the average price per category, skip everything else. For large histories, the savings in storage and query time are substantial.
The cost is accessibility. A columnar file cannot be read in a text editor or opened by most spreadsheet users, and producing one correctly requires settling the schema: whether a price is a decimal with two places, whether a timestamp carries a time zone, which fields can be null. Schema changes need care, since readers may reject files whose types differ from earlier ones.
In the retailer project: a columnar copy suits the long price history the dashboard queries, while the daily files for the analyst stay in CSV.
Choose it when volumes are large, consumers are analytical systems rather than people, and someone owns the schema and its versions.
Option 4: Direct database or warehouse loads
Here the collection writes straight into tables the consumers already query. There is no file to pick up and no import job to maintain, and the data is queryable as soon as the load finishes.
The trade-off is coupling. Whoever runs the collection needs credentials, network access and write permission on your systems, which means a security review and a decision about which tables they may touch. A schema change on either side can break the load, and a faulty run writes bad data directly where people are using it, unless loads go first into a staging table that is checked before records are promoted. If the collection is outsourced, this option also makes switching providers harder, because the load logic lives in two places.
In the retailer project: loading into a staging schema, then promoting after checks, saves the data engineer a daily import step, at the cost of an access setup that takes longer to approve than the scraper takes to build.
Choose it when the consumer is a data team with a warehouse, the access model is clear, and a staging step catches problems before they reach reports.
Option 5: An API or push delivery
With a pull API, the consumer requests records when it needs them, usually filtered by time or source and returned in pages. With push delivery, often called a webhook, the collection sends each new batch or record to an address the consumer exposes. Both suit applications: a repricing tool that needs today's competitor prices within minutes, or an alert that fires when a product goes out of stock.
Both also add moving parts. A pull API needs pagination, authentication and its own rate limits, and the consumer has to keep track of what it already fetched. Push delivery needs a reliable receiving endpoint, retries when it is down, and protection against processing the same record twice when a retry repeats a delivery. Neither keeps a long history by default, so a separate archive is usually still needed.
In the retailer project: only the stock alerts justify push delivery; everything else is well served by daily files.
Choose it when freshness is measured in minutes and an application, not a person, consumes the data.
The five options side by side
| Option | Best first consumer | Freshness | Nested data | Main maintenance cost |
|---|---|---|---|---|
| CSV files | Analyst | Batch | Poor | Format agreements, import fixes |
| JSON Lines files | Engineer or script | Batch | Good | Agreed date and number formats |
| Columnar files | Analytical system | Batch | Good | Schema ownership and versions |
| Database load | Data team | Batch or frequent | Depends on tables | Access, staging, coupling |
| API or push | Application | Minutes | Good | Endpoints, retries, duplicates |
Where the hours go in the retailer project
Features rarely decide this choice; maintenance does. A rough, hypothetical estimate for the retailer project shows why. Setting up CSV and JSON Lines delivery to a shared storage location takes a day or two, and afterwards the recurring work is mostly answering format questions and fixing the occasional import, perhaps an hour or two a month. A staging load into the warehouse adds a few days of setup and a few hours a month of attention to failed loads and schema updates.
A push feed for stock alerts is the expensive part. The receiving endpoint has to be monitored, retries have to be tested, and duplicate alerts have to be suppressed, because a retailer that sends two price changes for one stock event will notice quickly. Even for a single feed, budget a week of setup and a few hours a month of upkeep. Extending push delivery to all 200,000 daily records would multiply that work for no benefit, since the analyst and the dashboard only look at the data once a day.
Failure handling also differs in practice. With files, a broken delivery is visible as a missing or short file, and the fix is to re-send it. With a database load, a broken run can leave half-written data in tables people are querying, which is why the staging step matters. With push delivery, a failure can go unnoticed for hours if nobody compares what was sent with what was received. The same estimate also points to the risk of each choice: files fail loudly, loads fail in the open only if staging exists, and push feeds fail quietly unless someone builds the comparison.
What every delivery needs, whatever the format
Most delivery failures have nothing to do with the format. They come from missing agreements around it. Whichever option you choose, settle these points in writing:
- A manifest per delivery listing the files or batches, their record counts and a checksum, so a partial upload is detected before anyone loads it.
- Predictable naming with the source, the collection date and a sequence number, so a re-delivery replaces the right file instead of adding a duplicate.
- Idempotent loading: loading the same delivery twice should leave the data unchanged, which usually means a stable record key and an upsert rather than a blind append.
- A schema version carried in every delivery, with advance notice of changes, so consumers can adapt before a renamed field breaks a report.
- Provenance fields on every record: the source URL, the collection time and the extractor version.
The same points apply when data comes from a provider. If a managed service such as ScrapingIsNotACrime delivers part of the collection, the delivery specification is where these agreements belong, alongside the data quality checks to run on every scraped dataset. Large collections make these rules more important.
Which delivery fits which team
For a small team where analysts are the main users, start with CSV for flat extracts, with the format written down, and add JSON Lines only when nested records appear. The maintenance cost stays close to zero, and nothing prevents a later move.
For a team with a data engineer and a warehouse, deliver JSON Lines or columnar files to a storage location and load them through a staging table. Files give you a replayable archive; staging gives you a place to run checks before data reaches reports. Direct loads make sense once the access model is settled and the schema is stable.
For a product that reacts to changes, such as repricing or stock alerts, use push delivery or a pull API for the few fields that need minutes of freshness, and keep the full records in daily files. Building everything around real-time delivery costs more than most projects need.
For a team buying data from a provider, prefer file delivery with manifests at the start, even if an API is offered, because files are easier to audit during a trial. In the hypothetical retailer project, that combination, CSV for the analyst, JSON Lines through staging for the warehouse and a single push feed for stock alerts, covers every consumer with the least to maintain.
