Designing a Recall Report Engineers Actually Read
The hard part of measuring recall is not the maths. It is getting a tired on-call to believe the number.
We shipped a recall regression alert last year that fired on a genuine drop and nobody looked at it for twenty-two minutes. The measurement was right. The alert was clear. The on-call had been burned by noisy scores often enough to mute the channel before reading.
That was a product failure, not a statistics failure.
What we changed
The measurement got better, of course. The bigger change is that every recall number now arrives with three things attached, every time.
- The ground truth it was scored against, and when that ground truth was last recomputed. A recall figure without its reference set is a rumour.
- A plain-language band — healthy, watch, or regressed. Not a p-value, not a z-score, not a percentile of a percentile.
- The queries that lost the most ground. Twenty of them, in full, with the results the exact search would have returned. An engineer can argue with a number. Nobody argues with a query they can paste into a terminal.
What did not work
Suppressing low-confidence regressions automatically. We tried it for a quarter. Engineers stopped trusting every report, because they knew some had been filtered out and could not tell which. Keeping everything visible and ranking it honestly beat hiding the weak signals.
The 3am test
The review question on every change to this report is the same: what does this look like to someone who was asleep four minutes ago? If the answer needs a paragraph of explanation, the report is wrong, not the reader.
