Charity Majors+ Your Authors @mipsytipsy CTO @honeycombio; co-wrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤Black Lives Matter🖤 Jun. 22, 2020 2 min read + Your Authors

your reminder that if you are eagerly eating up everything we say about observability-driven development, but then go out & buy a monitoring or APM tool*, you are not going to reap the desired results. 😕

*even if it says 'observability' on the box. sorry, i wish they wouldn't.

i have spent a lot of energy describing the technical details of observability and why they matter.  https://www.honeycomb.io/blog/so-you-want-to-build-an-observability-tool/ 

without them you *cannot* ask any new arbitrary question of your systems, even if you couldn't anticipate wanting to ask it, and without shipping new code.

got a monitoring tool? you can only ask the questions and get the answers if you thought to define and gather a custom metric for it in advance.

(and cost scales up linearly with each metric, which means devs are incentivized to prune away critical information all too often)

got a logging tool? better hope you thought to emit the detail you want to search for, and on the same line as any other detail you want to correlate with, and that both are indexed fields.

(and cost model of logs is simply too wacky and wildly variable to even generalize about)

got APM? great, if your app fails in the most boring and predictable and well-trodden ways, and the query/app/user/cart/whatever you care about is always helpfully in the displayed top-10 list.

(but APM is probably the closest you're gonna get, esp if tracing is well integrated)

i don't say this much, because i don't like mentioning competitors, and it sounds like shitposting.

but it's frustrating having people excitedly engage with the idea of observability-driven development but then run into a wall with their time series aggregate metrics bullshit

you cannot use a monitoring tool to practice ODD. or anything based on metrics+tags.

you HAVE to be able to run novel computations on arbitrarily-wide structured data blobs, one per req per service, to perform super basic ODD actions like diffing the errors between app versions.

it's not your fault; several companies have invested serious marketing dollars into muddying the waters and making you think they do observability too.

and i know they are investing loads of engineering effort turning their monitoring/logging products into observability tools

so *someday* you'll be able to use your datadawg/newrelic/splunk/whatevs for the kind of observability i am discussing. but as of today you cannot. please don't let that sour you on trying, it will change your life. p.s. try our free tier. <3

(more here:  https://thenewstack.io/observability-a-3-year-retrospective )


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.


Follow Threader