“An alert is something which requires a human to perform an action.” - Pagerduty “Alerting Principles”
To understand why alerting can create a problem, you need to think about the purpose of alerts and how they differ from other monitoring components.
Actionable alerts are not:
Logs. Alerts are not records of events; that’s the role of logs.
Notifications. Alerts are not intended to announce non-critical occurrences such as the completion of a software build.
Alerts play an important role in your reliability monitoring strategy, but in order to be helpful, they must be properly constructed for situations that warrant immediate human attention, and they should be devised with simplicity, scope, and context in mind.
Preferences on how alerts are delivered can be designed using Action Groups in Azure.
You have learned how to monitor and interact with indicators of the reliability of your systems and create reliability goals, but there is also an important way by which reliability interacts with you.
To create effective actionable alerts, you must understand their components and characteristics. Actionable alerts have:
Simplicity is self-explanatory: make your alerts easy for you and others to understand, even if you’re reading them after being awakened at 2:00 a.m. Scope and context should be included in the content of the alert.
Let’s look at some elements that an actionable alert should always include: