Charity Majors+ Your Authors @mipsytipsy CTO @honeycombio; co-wrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤Black Lives Matter🖤 Jul. 08, 2020 3 min read + Your Authors

i love getting tagged in these ongoing conversations about complex adaptive systems. 😍 jon's excellent article here -- 

think about your infra diagram. it makes *sense* that LAMP stack-era tooling would struggle to make sense of this hairball, doesn't it?

there is a story that i fear is getting sidelined as the big company marketing departments and their megabucks swing into action, with everybody jumping on the buzzword bingo bus for observability.

these are two powerfully different modalities:

monitoring, LAMP stack, monolith, The Database, devs/ops, scary deploys, pager thresholds, dashboards, staging vs prod


observability, microservices, ephemeral/dynamic infra, serverless, polyglot persistence, ship continuously, software ownership, SLO budgets, test in prod

it isn't that one is "good" or "bad", or even "old" and "new", so much as... the complexity of our systems continues to skyrocket, forcing us to evolve new roles, new ways of organizing and managing our code, new tooling that gives us finer visibility and better control.

is observability "about" three pillars vs the arbitrary-wide structured data blob? hell no.

it is *about* understanding your systems. which is why when we talk about o11y, we are often highlighting aspects of complex systems that earlier tooling struggled to handle

handling lots of types of things -> ergo, high cardinality
stringing together many questions -> high dimensionality
unpredictability -> read time aggregation

honeycomb was built to the spec of 'what is needed in order to understand modern, complex systems'

people often accuse me of defining observability as "what honeycomb does", but they get the causation order backwards. 🙃 the definition came first, honeycomb was built to deliver it. 

however, i believe there is an immense amount of discovery left to do.

esp around UX. the fixed dashboard style of interacting with production is a legacy hangover. we need to make o11y tooling fluid, interactive, intuitive, a natural extension of developing code.

there should be ambient modes, exploratory modes, a shared team experience...

bubbleup was a massive innovative step forward for observability, but we need more. uniting the tracing workflow and the event exploration is a frontier that nobody has gotten right yet.

...i don't remember what this thread was supposed to be about. but hi, we're hiring. :P

... oh right, i was just trying to say: if you think you need observability, you're probably right.

but it's not as simple as "buy tool, achieve o11y". and you can't get observability with anyone who has rebranded their monitoring tool and duct taped on some logs and traces.

open plea to monitoring/logging vendors to stop confusing us with your rebranding efforts.

words matter. technical definitions matter. we need ways of talking about this modality shift and how our tools need to change. if you don't honor that reality, you do more harm than good.

oh right, and THIS -> 

is RIGHT-ON in its analysis of the problem, right up until they earnestly throw in "and therefore AIOps!" and ruin it all.

AIOps is the underpants gnomes of the VC/tech media moment, groan.

You do not need fucking AIOps. Most of the shill they're selling isn't even AI, or ops, especially the shit that kinda sorta works. Pure grade snake oil. 

Yes, many of your tools were built for a previous generation or two of complexity. Yes, you need to change them. Your tools must be less prescriptive, more exploratory and open-ended, better at helping you sift signal from noise.

Training your corpus on last week's deploy tho...

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.

Follow Threader