Websites split long listings across pages in six common ways: numbered pages, offset and limit parameters, next links, cursor tokens, load-more buttons or infinite scroll, and date-based archives. Each one needs a different way of moving forward and, just as important, a different rule for when to stop. Most missing records and duplicates in scraped listings come from a stop condition that does not fit the pattern, so each item below describes how to follow the pattern, the check that confirms it works, and the mistake that usually breaks it.
Two rules apply to all six. Before following any pattern, confirm in the site's robots.txt and terms that the listing paths are open to automated collection. And keep a set of the item identifiers already collected in the current run, because every pattern on this list can produce duplicates under the right conditions.
1. Numbered pages in the URL
The most familiar pattern puts a page number in the address, either as a query parameter such as page=3 or as a path segment ending in /page/3. Following it looks trivial: increase the number until the pages run out.
The trouble is knowing when they have run out. Some sites return an empty listing past the last page, some return a not-found status, and some quietly redirect any out-of-range number back to page 1 or repeat the last page forever. A scraper that stops only on an empty page will loop on those last two.
How to follow it: stop when any of three things happens: the page has no items, the status is not a success, or the first item on the page is one you already collected in this run. If the pagination control shows a last page number, read it on page 1 and use it as an upper bound, but keep the other checks, because the count can change during the run.
Check: request a page number far beyond the end once, by hand, and note what the site does. That one request tells you which stop condition will actually fire.
Common mistake: generating page addresses from a template without checking robots.txt. Some sites allow the first listing page but disallow paginated or filtered variants, often precisely to keep crawlers out of endless combinations of page numbers and filters. Check the actual paths you will generate, not just the first page.
2. Offset and limit parameters
Here the address carries a starting position and a page size, such as start=40 and count=20. It behaves like numbered pages with one important difference: positions are relative to the current state of the listing. If new items are added at the top while you crawl, everything shifts down. Items you already collected reappear on the next page, and in the reverse case, when items are removed, some are skipped entirely.
How to follow it: deduplicate by item identifier, and if the listing offers an oldest-first sort, use it, since new items then land at the end instead of shifting everything. For listings that change fast, a full pass should be repeated or reconciled against the identifiers from the previous run rather than trusted on its own.
Common mistake: raising the page size far above what the site's own interface uses, to save requests. Very large pages can be expensive for the server to build. Stay within the sizes the site offers to its own visitors.
3. Next links
Many listings include a link to the next page, either as a visible button or as a link element marked with rel next in the page head. Following it means reading that link on each page and requesting it, until a page has no next link.
This pattern adapts to whatever address scheme the site uses, which makes it robust. It can still loop. Next links sometimes carry tracking or session parameters that make the same page look like a new address, and a misconfigured site can link the last page back to the first.
How to follow it: normalize each next address before requesting it, dropping parameters that do not change the content, and keep a set of addresses already visited in this run. Stop when the next link is missing, when it points to an address already visited, or when a page yields no new items.
Check: log the sequence of page addresses for one full run. A sequence that repeats, or one that grows much longer than the visible page count suggests, points to a loop or to parameters that should have been normalized.
4. Cursor tokens
Cursor pagination is common in official APIs and in structured data feeds. Each response contains a batch of items and an opaque token; you send the token with the next request to get the following batch. When there are no more items, the token is missing or empty.
Cursors handle changing listings better than offsets, because the token marks a position in the data rather than a count. They come with their own rules, which the API documentation sets: tokens may expire after a time, may be tied to the original query parameters, and may not be reusable.
How to follow it: treat the token as opaque, never construct or modify it, and send the same query parameters with every request. Stop when the response has no token or an empty batch. If the documentation says tokens stay valid for a while, store the last one so an interrupted run can resume instead of starting over. Respect the documented rate limits; cursor-based endpoints often state them explicitly.
Check: log the number of items in each batch along with whether a token came back. A run that ends with a full-size batch and no token deserves a second look, because a normal end usually comes with a short final batch; an abrupt stop can mean an error response that the scraper read as the end of the data.
Common mistake: treating an undocumented internal data endpoint as if it were a public API. If the cursor comes from an interface the site uses internally, its terms of use still apply, and the endpoint can change without notice.
5. Load-more buttons and infinite scroll
On these pages, more items appear when the visitor clicks a button or scrolls down. Behind the scenes, the page almost always requests the next batch from the server, using one of the patterns above, usually offsets or cursors, and inserts it into the page.
How to follow it: first look for a lighter route to the same items. Many sites that scroll endlessly also offer numbered archive pages, an XML sitemap, an RSS or Atom feed, or an official API. If none exists, driving a full browser to scroll is the heaviest option for both sides, because it loads scripts, images and tracking along with the data. When a listing is reachable only that way, that cost is a fair factor in deciding whether the source is worth collecting at all, or whether to ask a provider such as ScrapingIsNotACrime to assess and quote it.
Check: compare the number of items collected with the count the page itself shows, if it shows one. A large gap usually means the stop condition fired early, often because a batch request failed and was read as "no more items".
The comparison of official APIs and scraping public pages for social media data discusses when official channels are the better route.
6. Date-based archives
News sites, blogs and some public records portals organize content by period, with addresses such as /2026/10/ or a listing filtered by date range. This pattern is the friendliest for repeated collection, because you can fetch only the periods that are new since the last run instead of walking the whole listing again.
How to follow it: keep a record of the last period fully collected, and on each run fetch from that period onward, re-fetching the most recent completed period once to catch items published late. Inside each period, the listing may itself be paginated with one of the patterns above.
Common mistake: ignoring time zones at period boundaries. An item published just before midnight in the site's time zone can land in the next day in yours, and be skipped by both runs. Use the site's own time zone to decide which periods are complete, and overlap the boundary by one period.
Check: for one month, count items per day collected and look for days with zero or with roughly double the usual count. Both point to boundary errors. As collections grow, the same per-period counts become one of the simplest monitoring signals, a point also made in the guide to data quality checks to run on every scraped dataset.
