OH GOD DO NOT DO THIS THING
The #1 sign that someone does not have observability is this: they lean on alarms for ⭐️EVERYTHING.⭐️
They cannot debug or introspect their systems any other way, so they abuse the FUCK out of monitoring checks and alarms.
I *just* wrote a post about this: https://charity.wtf/2019/09/20/love-and-alerting-in-the-time-of-cholera-and-observability/ …
Sorry if that was a little harsh. Flashbacks. Shudder.
Listen, I am not shaming anyone for doing the best they can do with the tools they have had.
It is NOT EASY trying to cobble together an understanding of a complex system using checks, alarms, dashboards, and your intuition. Sweetheart, I *know*. You do your best
But this is exactly why I feel so strongly about it. You will shred even the strongest of teams this way; it is just a matter of time. It is genuinely not compatible with human flourishing.
It worked okay for a while, in the days of the monolith and the LAMP stack. Not great, but "okay".
But your systems are getting exponentially more complex. Every dimension is high cardinality with a growing set -- services, apis, instances, containers, endpoints, storage...
Which means that the number of things that can go wrong, or combinations of things that can go wrong, or spaces between combination of things...
It's all growing a lot faster than *that* exponential curve. You cannot subject a team to this barrage of alerts and thresholds.
The ONLY way I know of to bend the cost curve down is with SLOs, observability tooling that lets you explore open endedly in real time, and limiting alerts to mirror customer impacting pain. Mostly e2e.
Page less. Not more. Page way, way, way less. WAY less.
You can follow @mipsytipsy.
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.
Enjoy Threader? Sign up.
Since you’re here...
... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.