Robustness and resilience

Two dangerously addictive ideas:

  1. Keeping the system working through heroics.
  2. The dream of an autonomous system that requires no human intervention at all.

You need a combination of robustness (designing the system to handle known problems) and resilience (being ready to deal with the problems the system wasn’t designed to handle). Otherwise, you’re either brittle or you burn people out.

Source