Skip to content
Home Posts About us Privacy Policy
Writing a Data Specification for a Scraping Project: A Hypothetical Events Case

Writing a Data Specification for a Scraping Project: A Hypothetical Events Case

Share ๐• in f W

When a scraping project starts without a specification, the scope is settled too late: someone asks for "all the data" from a list of sites, a scraper is built, and the arguments about what the data should contain start after delivery. A data specification settles those arguments first: what one record is, which fields it has and what each field means, which sources are in scope and on what terms, how fresh the data must be and how the result will be accepted. The trade-off is a week or two of thinking before any collection starts, in exchange for less rework later. This article follows a hypothetical project from a one-line request to a specification that a team, or an outside provider, could build against.

The starting point: a one-line request

Consider a hypothetical regional tourism office. It runs a public calendar of things to do in the region and wants it filled automatically from the websites of about 60 venues, museums, festivals and municipalities. The request to the data team reads: collect all events in the region every day.

Every word in that sentence hides a decision. "All" could mean every listing on 60 sites or only those that matter to visitors. "Events" could include weekly yoga classes and permanent exhibitions. "Every day" might be necessary or might triple the cost. The team decides to write a specification before estimating anything, and to have the calendar's editor sign it.

Step 1: Name the decision the data serves

The team starts with use, not with sites. The editor explains that the calendar exists to help visitors plan a trip in the coming four weeks, and that the most damaging error is listing an event that was cancelled.

That one conversation shapes the rest. Events more than three months ahead are low priority. Cancellations and date changes matter more than descriptions. Recurring classes for residents are out of scope. The specification opens with a two-sentence statement of purpose, so that later disagreements can be settled by asking which option serves it better.

Step 2: Define what one record is

Next comes the unit. The team writes that one record is one occurrence of a public event at one place and start time. A concert on two evenings is two records; a festival running for five days is one record with a start and an end date; a permanent exhibition is excluded, while a temporary one with fixed dates is included.

They also define identity. A record is identified by source, the event's own page address and the start time, so the same concert listed by the venue and by the municipality appears twice in the raw data and is merged later by a matching rule written into the spec. Without this paragraph, two developers would have counted events in two different ways, and coverage figures would have been meaningless.

Step 3: List the fields with definitions

The field list is a table, and each field has a definition, a format, a required or optional flag and an example. Abbreviated, it reads:

FieldDefinition and formatRequired
titleEvent name as shown on the source page, trimmedYes
startStart date and time with the region's time zone; date only if no time is publishedYes
endEnd date and time, if publishedNo
venueVenue name, matched to the office's own venue listYes
statusScheduled, cancelled, postponed or rescheduledYes
price_textPrice information as published, not parsedNo
categoryOne of the calendar's eight categories, mapped from source labelsYes
source_urlAddress of the event's own pageYes
fetched_atTime the page was collectedYes

Two choices here save money. Price is kept as text, because parsing dozens of ticket formats would cost more than the calendar gains from it. Descriptions are not collected at all: the calendar links to the source page instead, which avoids both the effort of cleaning long texts and questions about copying content that venues wrote themselves.

Step 4: Review each source and what it allows

For each of the 60 sites, the team records the robots.txt rules for the event pages, the relevant terms of service clauses, and whether an official channel exists. The review finds that 14 sites publish a feed or a calendar export, 21 include structured event data in their pages, and the rest offer only ordinary HTML. Three sites prohibit automated collection in their terms; the office writes to them, two agree to send a monthly export, and the third is dropped.

The specification lists every source with its access method and the date of review, and states that the method may be changed only after the review is repeated. That clause matters for maintenance, because it stops a developer from quietly switching a source to page collection when its feed breaks.

Step 5: Set freshness and coverage targets

Instead of "every day", the spec sets targets tied to the purpose. Events in the next four weeks must reflect the source within 24 hours. Events further ahead may be up to a week old. Coverage is defined as the share of events visible on a source's listing page that appear in the dataset, measured on a sample.

The tiered schedule cuts the expected number of requests to well under half of what daily full collection would have needed, by the team's own estimate. It also gives the editor a target that can be checked, rather than a general promise of completeness.

Step 6: Write the rules for changes and personal data

Because cancellations are the costliest error, the spec says how they are detected: the status field is read on every visit within the four-week window, and an event that disappears from a source is marked "not found" for review, never deleted automatically. Rescheduled events keep their identifier and record the old date.

On personal data, the spec is short and explicit: organizer contact names, phone numbers and personal email addresses are not collected, even when they appear on event pages. A general venue contact is enough for the calendar, and excluding the rest removes a whole category of privacy obligations.

Step 7: Agree on acceptance and ownership

The last section defines when a delivery is accepted. Each week, the editor checks a random sample of 40 records against the live pages; the delivery passes if required fields are correct on at least 38 of them and no cancelled event in the sample is shown as scheduled. Failures go back to the team with the record and the page.

Ownership is named per task: who maintains the source list, who answers venues that object or ask questions, and who updates the category mapping. With that in place, the team estimates the build and asks two outside providers for quotes against the same document; the office compares a ScrapingIsNotACrime quote and another provider's on equal terms because both answered the same specification.

What changed because the spec existed

In this hypothetical, the first month brings three disputes, and each is settled in minutes by pointing to the document: a festival listed as five records instead of one, a venue whose descriptions the editor suddenly wants, and a source that started blocking requests. The first is a bug against Step 2, the second is a change request with a cost, and the third triggers the review in Step 4 rather than a workaround. Without the specification, all three would have been arguments about what "all events" meant.

What to take from this case

A useful data specification is short and decision-driven. Start from the purpose and the most expensive error, define the record and its identity before the fields, give each field a definition and an example, record each source's access method and terms with a date, set freshness and coverage targets that can be measured, and write the acceptance test before the first delivery. Leave out fields nobody will use, especially personal data. Have the person who depends on the data sign it, and change it only through a recorded change request.

The same document works whether the collection is built in-house or bought, and it is the most reliable basis for comparing quotes.

Share ๐• in f โœˆ