Measuring the wrong thing
Average latency is a comfort metric. What users feel is the tail: the one request in a hundred that takes two seconds while the other ninety-nine take forty milliseconds. Optimise the histogram, not the mean.
Backups you have not restored are not backups
A snapshot nobody has mounted is a hypothesis. The only evidence that matters is a restore into a clean environment, done on a schedule, with the time it took written down.
Reading list
- SRE Book — chapters on cascading failures
- Dynamo paper — for the vocabulary, not the design
- Anything by Leslie Lamport, slowly