Collection frequency should be set per source, from how often that source actually changes and how quickly users need to see the change, not from a single schedule applied to everything. The trade-off is freshness against cost and load: every extra run costs compute, adds requests to someone else's server and increases the chances of noticing breakage at the worst moment. To show how the decision plays out, this article follows a hypothetical project from its first instinct, "collect everything every hour", to a tiered schedule that is cheaper and, for its users, fresher where it counts.
The hypothetical project and its first instinct
A small company runs a regional job listings site. It plans to collect public job postings from the career pages of 60 local employers, with each employer's posting reproduced as title, location, a short summary and a link back to the original. Before choosing any frequency, the team checks each career page's robots.txt and terms of use, contacts several employers who are happy to be listed, and drops three whose terms exclude automated collection. That leaves 57 sources.
The founder's first instinct is to collect every source every hour. Job seekers hate stale listings, the reasoning goes, so maximum frequency means maximum quality.
Suppose, as a hypothetical, each career page has a listing page plus an average of 15 posting pages. A full pass is then about 57 ร 16 = 912 requests. Hourly, that is roughly 21,900 requests a day, most of them to pages that have not changed. For small employers whose sites run on modest hosting, an hourly crawl is far more traffic than their own visitors generate.
Step 1: Measure how often each source changes
Instead of guessing, the team runs a two-week observation at a modest daily pace, recording for each source how many postings appeared and disappeared each day. In the hypothetical results:
- 8 large employers change almost every weekday;
- 31 employers change once or twice a week;
- 18 employers changed nothing in two weeks.
The observation also shows something the hourly plan would have hidden: almost nothing changes at weekends, and the publication times shown on posting pages cluster on weekday mornings.
Where a posting page shows its publication time, the team records it too. A once-a-day observation cannot see time of day on its own, but those published times are enough to show that checking listing pages in the early afternoon would catch most of each day's new postings on the same day, while a pre-dawn run would always be a day behind.
Decision: frequency will follow measured change, not intuition. The observation itself costs about 912 requests a day for two weeks, a fraction of the hourly plan.
Step 2: Tie frequency to what users need
The team then asks what freshness means to a job seeker. Two needs come out. New postings should appear within a day, since many applications close within weeks and early applicants have an edge. Closed postings should disappear quickly, because sending someone to a vacancy that no longer exists is the complaint users remember.
Neither need requires hourly checks. A posting that appears at 10 a.m. and is listed by early afternoon serves the job seeker as well as one listed at 11 a.m. The second need, removal, turns out to matter more than the first and needs its own approach (Step 4).
Decision: the target is "new postings visible the same working day, closed postings removed within a day", written down as the service level the schedule has to meet.
Step 3: Group sources into tiers
With change rates and a target in hand, the team sorts sources into three tiers:
| Tier | Sources | Listing page checked | Posting pages fetched |
|---|---|---|---|
| A: changes most weekdays | 8 | Twice each weekday | Only new postings |
| B: changes weekly | 31 | Once each weekday | Only new postings |
| C: rarely changes | 18 | Twice a week | Only new postings |
The key move is checking only the listing page on each run and fetching a posting page only when its link is new. Where an employer publishes a sitemap or a feed of its vacancies, the team uses it instead of the listing page, since it states change dates directly.
In the hypothetical numbers, listing checks come to about 16 + 31 = 47 requests per weekday for tiers A and B, plus 18 twice a week for tier C. New posting pages add perhaps 30 a day across all sources. The daily total lands under a hundred requests, compared with nearly 22,000 for the hourly plan. Tier C sources face a risk, a posting missed for up to three or four days, which the team accepts because those employers post rarely and their vacancies tend to stay open for weeks.
Step 4: Handle removed postings separately
A posting that disappears from the listing page is probably closed, but not certainly. The listing fetch may have failed, the page may have shown a partial list, or the employer may have moved the posting to a new address.
The team sets a rule: a posting is marked closed when it is missing from two consecutive successful listing checks, or when its own page returns a not-found status. A listing check counts as successful only if it returned a normal number of items for that source. For tier A and B this meets the "removed within a day" target; for tier C, the twice-weekly check means a closed posting can stay visible for a few days, so the site labels tier C listings with the date they were last confirmed.
Decision: removal detection gets its own logic and its own monitoring, rather than being a side effect of the collection schedule.
Step 5: Re-tier on a schedule
Tiers are not permanent. An employer that starts a hiring drive moves from tier C behavior to tier A behavior overnight. The team logs the number of new postings per source per week and reviews the tiers monthly: a source that produced new postings on most checks in the past month moves up, and one that produced none moves down.
The monthly review also looks at the removal rule. If users report closed postings that stayed visible, or if many postings are marked closed and then reappear, the thresholds need adjusting. Both kinds of error are logged, so the review is a short look at two numbers per tier rather than a debate.
It also adds one safeguard: any tier C source where a check finds a new posting gets an extra check the next working day, which catches the start of a hiring burst without waiting for the monthly review.
Step 6: Build in-house or hand it off
With the schedule defined, the build-or-buy question has numbers attached. Under a hundred requests a day across 57 sources is light work for the infrastructure; the real cost is maintenance, because 57 different career page layouts will each change at their own pace. The team estimates, hypothetically, a few hours a week of upkeep once the extractors are stable.
They compare that with quotes from managed services, including ScrapingIsNotACrime, using the written schedule, the removal rule and the service level as the specification, so that every quote answers the same question. The comparison weighs the monthly cost against the upkeep hours and against how much control the team wants over tiering. The look at where scraper maintenance hours go and how to cut them covers the same build-or-buy decision from the upkeep side.
What to take from the hypothetical case
- Measure change before choosing frequency. A short observation at a modest pace replaces guesswork, and costs little.
- Write down the freshness target in users' terms, such as "same working day", then pick the lowest frequency that meets it.
- Check the cheap page often, fetch the expensive pages rarely. Listing pages, sitemaps and feeds tell you what changed; detail pages only need fetching when something did.
- Treat removals as their own problem, with a rule that tolerates a single failed check.
- Review tiers regularly, because sources change their behavior.
- Use the schedule as the specification when comparing building in-house with a managed service.
The hourly plan would have looked diligent and delivered little: most of its requests would have fetched unchanged pages, and it still had no answer for closed postings. The tiered plan puts effort where change happens and gives the users what they actually notice.
