The decision is whether to sign a yearly contract with a web data provider on the strength of a sales conversation and a sample file, or to spend a few weeks on a structured trial first. The trade-off is plain: a trial costs internal time and delays the project, while skipping it risks a year of paying for data that does not fit the report it was meant to feed. Let us follow a hypothetical market research agency through a trial, from the brief to the signature, watching which decisions paid off and which the agency would make differently.
The agency and the project
The hypothetical agency produces a quarterly report on consumer electronics assortment for a manufacturer client: which models each of twenty-five online retailers carries, with the published specifications, and how the assortment changes over time. Prices are not part of the brief. The agency has two analysts, no engineers, and a deadline for the first report in four months. It has already decided against building in-house, following the reasoning in choosing how to scrape; the open question is which provider, and on what terms.
Writing the brief before talking to anyone
The agency's first decision is to write the brief before the first vendor call, and it is the decision that shapes everything after. The brief lists the twenty-five sites, the fields per product (model identifier, brand, product name, category path, the specification table as key-value pairs, availability state, product page URL), the required freshness (one full collection per week, delivered by Monday), the delivery format (one file per retailer in a line-delimited format plus a run summary), and the acceptance criteria: coverage of at least ninety-five percent of each retailer's category listing, measured against the retailer's own category page counts; no more than two percent of records with a missing model identifier; and a hand-checked accuracy sample with no more than three wrong fields in a hundred records.
It also lists the questions that are not about data: how the provider handles robots.txt and terms of service for each site, whether any personal data is collected (the answer should be none, since product pages carry none), how the provider identifies itself to the sites, and what happens when a site objects. Writing the brief takes the two analysts three days. Without it, the trial would have measured whatever each vendor chose to show. The structure of such a document is laid out in writing a data specification for a scraping project.
Choosing the trial scope
The agency shortlists two providers and sends both the same brief. It then decides not to trial all twenty-five sites, which would make the trial as large as the project, but to pick four: the two largest retailers, the one whose product pages load their specification tables in a way the sales demonstrations never showed, and the one with the strictest terms of service. The reasoning is that a provider who handles the hardest cases will handle the easy ones, and that the trial should expose the provider's behavior on a site where the legal question is not trivial. The trial runs for three weeks, which allows three weekly deliveries, enough to see whether the second and third match the first.
Running the trial and what to measure
For each delivery, the analysts measure four things against the acceptance criteria. Coverage: the record count per retailer against the category page counts, collected by hand the same day. Completeness: the share of records with each required field present. Accuracy: a sample of one hundred records per provider, chosen at random, with every field compared against the live product page by an analyst. Consistency: the week-over-week difference in record counts, with any drop explained or flagged. The method for the coverage figure follows measuring whether a scraped dataset is complete.
Suppose the results after three deliveries. Provider A reaches ninety-seven percent coverage on all four sites, with one percent of records missing the model identifier and two wrong fields in the hundred-record sample, both in specification values where the site shows two variants of a product on one page. Provider B reaches ninety-one percent coverage on the two large retailers and ninety-eight on the others, with four wrong fields in the sample and a delivery on week two that arrived a day late with no explanation in the run summary. Neither result is disqualifying on its own; the gap is in what each provider did when asked about it.
The event that decided it
In the second week, the retailer with the unusual specification tables changes its page layout. Provider A's week-two delivery for that site contains the right number of records but empty specification tables, and the run summary flags the site with a warning and a note that the extractor is being updated; week three is complete. Provider B's week-two delivery for the same site contains the right number of records with specification tables filled from the wrong part of the page, and nothing in the summary says anything changed. The analysts notice only because the accuracy sample happens to include six records from that site.
The agency weighs this heavily, and should. Over a year, every site will change layout at least once. A provider that detects the change and reports it costs the agency a week of partial data per incident; a provider that delivers wrong data silently costs the agency the credibility of the report, which is the product. On the legal questions, both providers describe their robots.txt handling clearly; one of them answers the terms-of-service question for the strict site with a generic statement, the other with a specific account of which clauses it read and how it concluded that collecting product specifications is within its policy, while noting that the agency should have its own counsel confirm. The specific answer is worth more than the reassuring one.
Questions beyond the data
Before signing, the agency asks the remaining provider a set of contract questions that the brief did not cover. Who owns the delivered data and whether the provider may reuse it. What is retained on the provider's side and for how long. Whether sub-processors are involved and where the data is stored. What happens if a site objects or blocks the provider: whether collection pauses, whether the agency is informed, and who decides. What the exit looks like: notice period, final delivery, and whether the agency gets the extractor configuration or only the data. The answers go into the contract rather than into an email, because the person who gave them may not be there when the question matters. Whichever provider is being evaluated, ScrapingIsNotACrime included, these are the questions that distinguish a data purchase from a dependency.
The first month after signing
The agency signs with provider A for the full twenty-five sites, with the acceptance criteria written into the agreement and a monthly review of the run summaries. It keeps the hand-checked accuracy sample as a permanent monthly routine, one hundred records across all sites, because the trial showed that the sample is what catches silent errors. In the first month, two of the twenty-one sites not covered by the trial fall short of the coverage threshold; the provider explains that both paginate their category listings in a way that hides part of the catalog, and proposes using the sites' sitemaps as the source of product URLs instead. Coverage rises above the threshold in month two. The agency notes that it would have preferred to find this in the trial, and that the fix was to include, in future trials, one site chosen for awkward pagination rather than only for size and legal strictness.
What to take from the agency's trial
Write the brief and the acceptance criteria before the first conversation, and send the same brief to every provider, so that the trial measures the provider rather than the demonstration. Trial the hardest sites, not the easiest, and choose them for different kinds of difficulty: scale, page structure, legal terms and pagination. Run long enough to see repeated deliveries, because a single good file proves little. Measure coverage, completeness, accuracy and consistency with your own hands, and keep the accuracy sample as a routine after signing. Weigh how the provider behaves when a site changes above how it performs when nothing does. Put the non-data answers in the contract. The trial in this case cost the agency roughly three weeks of two analysts' part-time attention; the alternative was learning the same lessons from the client who received the report.