Charity Majors
+ Your AuthorsArchive @mipsytipsy cofounder/CTO @honeycombio, co-wrote Database Reliability Engineering, loves whiskey, rainbows, and Friday deploys. I test in production and so do you. 🌈 Apr. 17, 2019 2 min read

Policies (and enumerated exceptions to policies, and exceptions to exceptions) are a piss-poor substitute for judgment. Rules are blunt instruments that stunt your engineers' development and critical thinking skills.

Allow me to illustrate.

Team A thinks deploys are scary. They try to ease the load on humans by freezing deploys on Fridays, before holidays, xmas-NYE, etc. Change will randomly upset god-knows-who (VPs, users, etc), and it can take multiple engineers multiple days to untangle an upset stakeholder.

Team B thinks deploys are ordinary. They happen constantly, every day. When one fails, it's in a fine state until someone decides to investigate. All stakeholders have bought in to the same SLO conditions, which are reviewed regularly, and nobody bugs an engineer out of hours

unless it exceeds an SLI. Any failed build or deploy will page the engineer who committed that diff, and they are expected to drop everything and fix it.

Every engineer knows that they own their code, and their diff isn't done until they watch users using their code in prod.

So they typically watch as it rolls out, breaking down by build id, comparing old vs new, and checking their instrumentation to validate that it is doing what they expected it to, and nothing else looks "weird". They let it bake for an hour or two and check again before leaving.

When problems are found in a deploy, most SLI/SLO impacting events show up quickly and take just a couple min to resolve and roll forward.

Subtler problems may take longer, but these tend to be judgment issues for PMs and leads to debate to resolution on Monday.

Would you rather work on team A, with rules to protect people from deploys, or team B, with a culture of ownership and good judgment (and feedback loops designed to align actions with consequences)?

I can tell you which set are going to grow into more powerful engineers.

Y'all, I am not dumb, I know most folks don't work and live in team B land. I know it takes management support. I don't begrudge you whatever you have to do to protect yourself and your team. I have done those things too.

But *aspire* to such a low bar? Glorify it? Hard no.

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.

Follow Threader