Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Sep. 26, 2019 1 min read

A: Yes, there are many aspects of instrumentation and observability that are worth investing in, even for monoliths. Distributed systems are just where the pain of *not* doing so becomes prohibitive.

This is largely for historical reasons. Due to hardware cost, we optimized for skimpy metrics data, and generations of engineers grew up fitting their thinking and tooling around scarce data types like metrics instead of richer event-trace data.

It is categorically easier to debug and understand your systems with o11y than with scattered, aggregated/averaged/aggregated metrics.

It's incredibly hard to form a mental reconstruction (or hazard a few guesses) about why events are erroring when all you have are dashboards.

It's much easier to simply ask a straightforward question. Which you can do if you gathered the data by event in the first place.

Instead of seeing some spikes and running through a library of guesses as to the cause, you just ask. Compare:

Old way being: "I see a latency spike. Is it mysql timing out? Is it a memcache issue? Is it us-east-1? Is it coming from apps running an old version of the code?"

as you flip through dashboards one by one to check for data to support your hypotheses

With events, it goes more like this: "I see a latency spike. I will circle it with the mouse, and -- aha, those errors are all timeouts to the /export endpoint from a test user exceeding the upload limit"

(ok I sorta cheated, that's using bubbleup instead of iterating)

The point is: you don't have to guess or imagine sprawling abstractions. The data
Is right in front of you.

It takes a little work to unlearn old habits. But new grads and people who haven't done hard time in ops tend to find it much easier and more intuitive than the old ways.


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.