Silent success is not success
The most expensive incidents we have been called in to untangle were not crashes. A crash is loud: someone gets paged, the job is rerun, the gap is obvious. The expensive ones were jobs that exited with status 0 every night for months while doing slightly less than everyone assumed.
A typical shape: an importer that pulls a supplier's CSV, drops rows it cannot parse and loads the rest. When the supplier added a thousands separator to one column, roughly four percent of rows started failing to parse. Nothing broke. The dashboards were a little low. It took eleven weeks for a finance analyst to notice that one region's numbers had drifted.
Three numbers, every run
Every job we write ends with a summary line containing at least three numbers: rows read, rows written, rows rejected. Not in a debug log that nobody reads - on standard output, where the scheduler captures it, and in a small table where it can be graphed.
2026-08-18T02:10:04Z import-supplier-a read=48211 written=48190 rejected=21
2026-08-19T02:10:03Z import-supplier-a read=48377 written=46402 rejected=1975You do not need anomaly detection to see what happened on the second line. You need the line to exist.
Rejected rows go somewhere
Dropping a row is a decision, and decisions should leave a record. Rejected rows are written to a
side file next to the input, with the reason attached, and kept for as long as the input is. When
someone asks "why is this customer missing", the answer is a grep away instead of an
archaeology project.
Thresholds, not perfection
Demanding zero rejects makes jobs fail for reasons nobody will fix, and people learn to ignore the alerts. Instead we agree a threshold with whoever owns the data - often half a percent - and the job fails loudly above it. The threshold is written in the runbook, next to the name of the person who agreed to it.