Skip to content
Home Posts About us Privacy Policy
Scraper Running Too Slowly? Where the Time Goes and How to Speed It Up Politely

Scraper Running Too Slowly? Where the Time Goes and How to Speed It Up Politely

Share ๐• in f W

When a scraper takes far longer than its polite crawl rate should allow, look inside the scraper before looking at the site. The common culprits are a design that waits on one site at a time, a new connection for every request, pages or browser rendering the job does not need, timeouts and retries set too generously, and parsing or database writes that block the next request. All of them can be fixed without sending more requests per second to any single site. By the end of this article you will know how to measure where the time goes and how to fix each cause while keeping the per-site pace exactly where it is.

The symptom: the numbers do not add up

Suppose, hypothetically, a job collects 20,000 pages from 50 sites, with a polite limit of one request every two seconds per site. If the sites were visited in parallel, each one holding to that limit, 400 pages per site would take a little over 13 minutes. Instead the job runs for most of a day. The gap between those two figures is the problem to solve; the per-site limit is not.

Before changing anything, add timing to every stage of a single request: waiting for a slot, opening the connection, waiting for the response, downloading the body, parsing, and writing the result. Log those durations for a few hundred requests, then sum each column. The largest column is where to start, and it is often not the one people suspect.

Cause 1: the scraper waits on one site at a time

A scraper that processes a list of URLs in order, sleeping between requests, spends almost all its time idle. While it waits two seconds before the next request to site A, it could be sending a request to site B, which has its own separate limit.

Fix: organize the work by host. Give each host its own queue and its own delay or token bucket, and let a pool of workers take the next request from whichever host is ready. The pace for each site stays the same, while the total throughput grows with the number of sites. Check: in the logs, the interval between consecutive requests to any one host never falls below your limit, while requests to different hosts overlap.

The mistake to avoid: raising concurrency globally without per-host limits. With one large list and ten workers, all ten can end up on the same site, multiplying the load on it, which is exactly what a polite scraper must not do.

Cause 2: a new connection for every request

If timing shows a large share spent connecting, the scraper is probably opening a fresh connection for each request, which repeats the DNS lookup, the TCP handshake and the TLS negotiation every time.

Fix: reuse a session or connection pool per host so that keep-alive connections carry many requests, and cache DNS results for the run. This also lowers the cost for the server, since each handshake costs it work too. Check: the connect time should be near zero for most requests after the first few to each host.

Cause 3: collecting pages and resources the job does not need

Many slow jobs simply do too much. They follow every link instead of the ones that lead to target pages, revisit pages that have not changed, or download images, stylesheets and scripts along with the HTML.

Fix: restrict link following to the URL patterns you need, use the site's sitemap or listing pages to find targets directly, and skip unchanged pages with conditional requests where the server supports them. Fetch only the HTML or JSON you parse. Fewer requests is the one speed-up that also benefits the site. Check: count requests per run by URL pattern; patterns that produce no records are candidates for removal.

Cause 4: rendering pages in a browser that do not need it

A headless browser loads scripts, styles, fonts and images, executes code, and waits for the page to settle. That can take many times longer than a plain HTTP request, and it multiplies the requests the site receives.

Fix: for each page template, test whether the data is present in the raw HTML, including any structured data the page embeds. Use the browser only for templates where it is not. Where a browser is truly needed, block resource types you do not use and wait for the specific element you extract, not for a fixed delay. Check: record which method each template uses and its average time per page.

Cause 5: timeouts and retries that are too generous

A 60-second timeout on a site that normally answers in half a second means one stalled request can hold a worker for a full minute. Add a retry policy that retries three times, each with the same long timeout, and a single unreachable page can cost several minutes.

Fix: set separate connect and read timeouts based on each site's normal response times, with some margin. Retry only temporary failures, with growing waits between attempts, and cap the number of attempts. Stop retrying and set the host aside after repeated failures. Never retry a 403 or a block page; treat it as a signal to stop and review that site's robots.txt and terms, as covered in the guide on how to read a robots.txt file and apply its rules. Check: the slowest one percent of requests should be explained by known slow sites, not by retries of hopeless URLs.

Cause 6: parsing and storage block the next request

If fetching is fast but the run is slow, look at what happens after each response. Heavy parsing on the same thread, or a database insert and commit for each record, can take longer than the request itself.

Fix: separate fetching from processing. Store raw responses or hand them to a queue, parse with separate workers, and write records in batches rather than one at a time. This also means a parser bug no longer forces you to collect pages again. Check: the fetch workers are never waiting on the database in your timing logs.

How to keep a scraper fast without adding load

Speed problems return as projects grow, so build the measurement into every run:

  • Log timing per stage and keep a summary for each run, so a slowdown can be compared with the previous week.
  • Track requests per record. A rise means the scraper is fetching pages that produce nothing.
  • Keep per-host limits in configuration, not hard-coded, and review them when a site slows down or returns too-many-requests responses.
  • Set a time budget per run and alert when it is exceeded, rather than discovering a slow job when the data is late.

For collections spread across hundreds of sites, scheduling and per-host queues become a project of their own; the guide to choosing between browser extensions, no-code tools, frameworks or a managed service covers when it makes sense to hand that work to a provider such as ScrapingIsNotACrime. Whoever runs it, the rule is the same: make the scraper faster by wasting less time, never by asking more of any one site.

Share ๐• in f โœˆ