Alerting and Anomaly Detection – Best Friends Forever? - Björn Rabenstein, Grafana Labs

Alerting and Anomaly Detection – Best Friends Forever? - Björn Rabenstein, Grafana Labs When Prometheus became publicly known starting in 2015, the Prometheus developers expected many questions. But one surprisingly stuck out: “Can you do anomaly detection?” Somehow, everyone expected a next-generation monitoring and alerting system to venture into anomaly detection. PromQL is powerful enough to support fundamental building blocks of anomaly detection, but the general direction of Promethean alerting is, in a way, exactly the opposite: Towards confident, non-noisy alerts based on your SLOs. In this talk, Beorn will share a few stories from the receiving end of the pager and why it is almost always a bad idea to put anomaly detection at the other end. He will talk about the “proper” Promethean way of alerting (including its limitations) and where anomaly detection (or even machine learning) might have its place in it after all.