A scheduled scraper that still runs months later
One resilience layer proven on two targets: retry with backoff, a circuit breaker, and blocks that fail loudly instead of quietly.
What was in the way
A scraper is easy to write and hard to keep alive. The first version works on the day you build it, then the markup shifts, the host starts refusing the server's IP, and the spreadsheet quietly stops filling.
On a Dutch housing-market site the failure mode was worse than an outage. The block came back as an ordinary HTTP 200 page with no listings on it, so the run logged "no new listings", exited clean, and told nobody. A total block and a quiet market looked identical for five days.
How it was built
Treat every failure as a state to classify, not an exception to swallow. Requests retry with exponential backoff and jitter, and a circuit breaker opens after repeated failures so a struggling target gets left alone. Any page that yields zero results is classified before it is believed: a real empty result set, a recognised block page, or markup that no longer matches.
The same core then moved to a login-gated target, where the constraints inverted. Some errors must never be retried at all, so ban signatures abort the run instead of backing off into it.
How it works
A block never looks like a quiet market
Zero-result pages are classified as blocked, unrecognised, or genuinely empty. Only the last one lets a run finish successfully, and the alert is throttled to one per day.
Some errors are worse to retry
On the logged-in target, detection signatures are hard non-retryable and abort the run. Backing off into a bot check deepens the flag rather than clearing it.
A partial crawl never writes a conclusion
The sweep that records listings leaving the market only trusts a pass that saw the whole inventory, and skips itself when most known listings are unaccounted for.