Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Jun. 09, 2019 1 min read

i keep forgetting to link to this longer piece. it tells the tale of how we tested a script to *terminate our entire infra and replace it with a newer OS* in prod.

"spend less time on what doesn't matter, more on what does. Test In Prod." 

but hey: the biggest reason "Don't Test in Prod" was a cardinal principle for years was because all you had was monitoring tools and aggregates, and you legit couldn't tell at a hi-res granularity what the subtle effects of your tests were.

if you're flying blind, *don't do it.*

observability -- the way i define it, which is NOT how every other vendor out there defines it -- requires the ability to slice and dice down to raw events, and group by any combination of multiple high cardinality dimensions.

so you can see what the loving hell you're doing.

time series intervals, averages of averages, fixed schemas, indexes, this shit is anathema to peering closely at the ripple effects of tiny changes to tiny parts of your systems.

DO NOT be all "oh charity says test in prod is fine" if you don't have the o11y to check yourself.

observability is the missing link to most recent trends in systems: chaos engineering, testing in production ... everything post microservices demands it.

call it observability or something else, i dont care, but get clear on the definition or you WILL hurt yourself.

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.