Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Apr. 17, 2019 2 min read

Policies (and enumerated exceptions to policies, and exceptions to exceptions) are a piss-poor substitute for judgment. Rules are blunt instruments that stunt your engineers' development and critical thinking skills.

Allow me to illustrate.

Team A thinks deploys are scary. They try to ease the load on humans by freezing deploys on Fridays, before holidays, xmas-NYE, etc. Change will randomly upset god-knows-who (VPs, users, etc), and it can take multiple engineers multiple days to untangle an upset stakeholder.

Team B thinks deploys are ordinary. They happen constantly, every day. When one fails, it's in a fine state until someone decides to investigate. All stakeholders have bought in to the same SLO conditions, which are reviewed regularly, and nobody bugs an engineer out of hours

unless it exceeds an SLI. Any failed build or deploy will page the engineer who committed that diff, and they are expected to drop everything and fix it.

Every engineer knows that they own their code, and their diff isn't done until they watch users using their code in prod.

So they typically watch as it rolls out, breaking down by build id, comparing old vs new, and checking their instrumentation to validate that it is doing what they expected it to, and nothing else looks "weird". They let it bake for an hour or two and check again before leaving.

When problems are found in a deploy, most SLI/SLO impacting events show up quickly and take just a couple min to resolve and roll forward.

Subtler problems may take longer, but these tend to be judgment issues for PMs and leads to debate to resolution on Monday.

Would you rather work on team A, with rules to protect people from deploys, or team B, with a culture of ownership and good judgment (and feedback loops designed to align actions with consequences)?

I can tell you which set are going to grow into more powerful engineers.

Y'all, I am not dumb, I know most folks don't work and live in team B land. I know it takes management support. I don't begrudge you whatever you have to do to protect yourself and your team. I have done those things too.

But *aspire* to such a low bar? Glorify it? Hard no.


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.