OH GOD DO NOT DO THIS THING
The #1 sign that someone does not have observability is this: they lean on alarms for ⭐️EVERYTHING.⭐️
They cannot debug or introspect their systems any other way, so they abuse the FUCK out of monitoring checks and alarms.
I *just* wrote a post about this: https://charity.wtf/2019/09/20/love-and-alerting-in-the-time-of-cholera-and-observability/ …
Sorry if that was a little harsh. Flashbacks. Shudder.
Listen, I am not shaming anyone for doing the best they can do with the tools they have had.
It is NOT EASY trying to cobble together an understanding of a complex system using checks, alarms, dashboards, and your intuition. Sweetheart, I *know*. You do your best
But this is exactly why I feel so strongly about it. You will shred even the strongest of teams this way; it is just a matter of time. It is genuinely not compatible with human flourishing.
It worked okay for a while, in the days of the monolith and the LAMP stack. Not great, but "okay".
But your systems are getting exponentially more complex. Every dimension is high cardinality with a growing set -- services, apis, instances, containers, endpoints, storage...
Which means that the number of things that can go wrong, or combinations of things that can go wrong, or spaces between combination of things...
It's all growing a lot faster than *that* exponential curve. You cannot subject a team to this barrage of alerts and thresholds.
The ONLY way I know of to bend the cost curve down is with SLOs, observability tooling that lets you explore open endedly in real time, and limiting alerts to mirror customer impacting pain. Mostly e2e.
Page less. Not more. Page way, way, way less. WAY less.
You can follow @mipsytipsy.
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.
Enjoy Threader? Sign up.
Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.