Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Jun. 19, 2019 2 min read

finally got around to reading this piece that was going around last week. it's a great piece! although the title is actively misleading. i guess it's cool that "observability" is now considered title clickbait? 🤷‍♀️  https://medium.com/windmill-engineering/observability-doesnt-work-in-dev-c214a9fb3e2d 

the throbbing heart of this piece is all about why testing and developing distributed systems is hard, in full painful glory. and ho boy is it hard.

he complains about the awkwardness of developing using prod tools. which .. sure, but that's not the key point here.

the critical point here is not primarily that o11y is cumbersome for development, it's that o11y and a prod-like dev environment is now the ONLY OPTION that delivers reasonably trustworthy results. in other words,

~ make dev like prod, then
~ test in prod.

this is the direction that all of software development is trending in. once upon a time you could just stuff your monolith and db in a container each and call it a day. now you have too many distributed components to trust those results.

and yes! this is a PITA to manage!

the whole point of the piece is to serve up this demand for a "specialized local development tool", like the one the author coincidentally is working on.

which i was all prepared to argue with, because ✨no!✨, we need more tools that ✨SPAN DEV AND PROD✨

but liz assures me that is in fact what Tilt is designed to do. so i will hold my fire and express my delight if so. ☺️

it is definitely time for more local development tools to span and smooth out the boundaries between pre/post prod.

ah -- one last thing, the piece details at length the pain of jumping from datadog to honeycomb to lightstep.

i agree, but this is self-inflicted. you can do all three using just honeycomb. why pay to store three times, why copy paste between tools? makes no sense.

you can set up a high level trigger to alert you if a threshold is reached using honeycomb; click the graph to dive deep using honeycomb; click any event to trace using honeycomb; click the span to zoom back out to events using honeycomb. #thereifixedit #notobservabilitysfault

quibbles around naming and tooling choices aside, this was a great piece about the problems with development and testing in the age of distributed systems. (just leave poor o11y out of it)

the solution, we all agree, is to: ✨Make Prod Boring✨  https://medium.com/windmill-engineering/observability-doesnt-work-in-dev-c214a9fb3e2d  ☺️


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.