Charity Majors+ Your Authors @mipsytipsy cofounder/CTO @honeycombio, co-wrote Database Reliability Engineering, loves whiskey, rainbows, and Friday deploys. I test in production and so do you. 🌈 Jun. 09, 2019 1 min read + Your Authors

i keep forgetting to link to this longer piece. it tells the tale of how we tested a script to *terminate our entire infra and replace it with a newer OS* in prod.

"spend less time on what doesn't matter, more on what does. Test In Prod." 

but hey: the biggest reason "Don't Test in Prod" was a cardinal principle for years was because all you had was monitoring tools and aggregates, and you legit couldn't tell at a hi-res granularity what the subtle effects of your tests were.

if you're flying blind, *don't do it.*

observability -- the way i define it, which is NOT how every other vendor out there defines it -- requires the ability to slice and dice down to raw events, and group by any combination of multiple high cardinality dimensions.

so you can see what the loving hell you're doing.

time series intervals, averages of averages, fixed schemas, indexes, this shit is anathema to peering closely at the ripple effects of tiny changes to tiny parts of your systems.

DO NOT be all "oh charity says test in prod is fine" if you don't have the o11y to check yourself.

observability is the missing link to most recent trends in systems: chaos engineering, testing in production ... everything post microservices demands it.

call it observability or something else, i dont care, but get clear on the definition or you WILL hurt yourself.

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.

Follow Threader