By the end of these steps you will have a scraper that writes, for every run, a request log, a parse log and a one-page summary, so that when a dataset comes out short or wrong you can answer three questions in minutes: which pages failed, which pages returned fewer records than expected, and whether the run behaved politely toward the site. Most scrapers log something, usually a stream of messages that nobody reads until a problem appears, and at that point the stream turns out to lack the one field that would explain it. The fix is to decide in advance what each log line must contain and to check, after the first run, that the questions above can be answered from the files alone.
Step 1: Give every run an identity
Start by defining what a run is: one scheduled execution of one job against one site or one group of sites, from start to finish. Assign it an identifier made of the job name and the start time, and record alongside it the version of the extractor code, the version of the configuration (selectors, start URLs, limits) and the user agent string in use. Write that header before the first request goes out.
The check is to launch the job twice and confirm the two identifiers differ and that both appear in the header, in every request log line and in every stored record. The mistake that usually breaks this step is stamping records with the collection time instead of the run identifier; two runs that overlap in time then become impossible to separate, and a bad run cannot be rolled back without touching a good one.
Step 2: Count requests by outcome, per host
For each host the run touches, keep counters for requests attempted, successful responses, redirects, each client error code seen, each server error code seen, timeouts, and requests skipped before sending because robots.txt disallowed the path, the URL was on a suppression list, or a conditional request returned not-modified. Increment them as events happen and write them to the summary at the end.
The check is arithmetic: attempted must equal the sum of successful, redirected, errored and timed-out, and skipped must be reported separately rather than folded into attempted. The usual mistake is counting a retry as a fresh attempt without marking it. That hides the real request rate against the host: a run that reports two thousand attempts may have sent six thousand requests if every failure was retried twice. Keep a separate retry counter, and treat a high ratio of retries to attempts as a warning in its own right. The post on rate limiting and backoff explains why that ratio matters to the site.
Step 3: Log the essentials of each request, not the response body
One line per request, with the run identifier, the timestamp, the URL as requested, the final URL after redirects, the status code, the elapsed time, the size of the response, the declared content type, the retry number and the reason for any skip. That is enough to reconstruct what happened without storing the page. If you keep raw pages for debugging or history, store them in a separate archive keyed by the same run identifier and URL, so the log stays small and the archive can have its own retention.
The check is to take any line from the file and confirm it can be parsed back into those fields by a short script, with no field containing an unescaped delimiter. Lines that mix free text with structured fields fail this check the first time a URL contains a comma or a quote. Use a structured format, one object per line, and keep message text in its own field.
Step 4: Record parse outcomes for every page
For every successful page, log how many records the extractor found, how many of them passed validation, how many were rejected and for which reason codes: a required field empty, a value outside its allowed range, a type mismatch, a duplicate key within the page. A page that returns a successful status and zero records gets its own flag, because that combination is the clearest early signal that the site changed its layout and the selectors no longer match anything.
The check is to search the parse log for pages with zero records and read two of them by hand. If they are genuinely empty listing pages, fine; if they contain data the extractor missed, the flag has already paid for itself. The mistake here is logging only failures and not successes: without the count of records found per page you cannot tell a page that lost half its items from one that never had them. When this flag starts firing, the causes and fixes in scraper returning empty fields are the place to start.
Step 5: Write a summary that survives a crash
At the end of the run, write a single summary document: start and end time, duration, the counters from step 2 per host, the parse totals from step 4, the number of records stored, and a comparison with the previous run of the same job expressed as a difference and a percentage. Add a status field with three values: ok, warning or failure, decided by thresholds you set per job. Suppose a hypothetical job that normally stores around twelve thousand records; a run that stores seven thousand nine hundred is a drop of about a third, which under a threshold of ten percent is a warning worth a human look even though nothing errored.
The summary must exist even when the run dies halfway. Write a provisional summary with the status failure as soon as the run starts, and overwrite it at a clean finish; a crash then leaves behind a document that says the run started, when, and never completed. The check is to kill the process in the middle of a run and confirm a readable summary with the failure status is on disk afterward. The mistake is building the summary only in the success path, which guarantees that the runs you most need to understand are the ones with no record.
Step 6: Keep evidence of polite behavior
For each host, record the observed peak and average request rate during the run, the crawl delay you configured and whether it came from robots.txt or your own default, the time robots.txt was last fetched, the number of requests sent during the host's declared quiet hours if it has any, and the user agent and contact route presented. This is the part of the log you would show a website owner who asks what your scraper did, and it is also the part that lets you prove to yourself that a change in the scheduler did not quietly double the rate.
The check is to compute the request rate from the request log of step 3 and confirm it matches the figure in the summary; a mismatch means one of the two is sampling incorrectly. If you receive data from a provider such as ScrapingIsNotACrime rather than running the scraper yourself, these are the fields to ask for with each delivery. How to present identity and contact details to site owners is covered in identifying your scraper to website owners.
Step 7: Set retention and make the logs answer questions
Decide how long each log lives. Summaries are small and worth keeping for the life of the project, because the run-over-run comparison in step 5 depends on them and because they document behavior toward sites. Request and parse logs are large and lose value quickly; a few weeks is often enough, long enough to investigate a complaint or a bad delivery. Store each run's files under its identifier, append-only, and load the summaries into whatever table or dashboard you already use so that trends in record counts, error rates and request rates are visible without opening files.
The final check is to answer, from the files and in one query each, three questions about a run picked at random: which URLs returned server errors, which pages produced zero records, and what the peak request rate to the busiest host was. If any answer requires reading through lines by hand, the field it needs is missing from the earlier steps, and the fix is to add it there rather than to build a parser for the messages. The mistake that undoes this step is deleting the request log of a run before the data it produced has been checked by whoever uses it; tie the retention clock to the acceptance of the data, not to the date of the run.