Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Sep. 27, 2019 1 min read

We have ~200 customers, some high volume enterprises, unpredictable traffic in a multitenant distributed system, and a single on call rotation...and our devs get alerted outside business hours around ✨once per week✨

On call does not have to be hell.

Oh, and we deploy code from cron. 🤘

It is absolutely harder to dig yourself out of a hole than to avoid falling into one in the first place. I don't want to minimize the magnitude of the work.

It's entirely probable that if Ben and I hadn't had the experience running ops at Parse, we wouldn't have known exactly what to invest in early on.

And most startups don't have the benefit of an ops co-founder from day one.

And holding the line here isn't free. Emily has had to inconveniently prioritize reliability work at times when we would have preferred to ship features.

We still spend about 30% of our eng time on cleanup and refactoring and things that don't directly move the product forward,

but we do it during normal working hours, and nobody is getting woken up or canceling weekend plans.

You have to treat it like a heart attack. So it doesn't become a chronic state. But this is the way software should be made and ✨owned✨

p.s. There is still plenty of work for opsfolks!!! We have three, counting me. They build out infra projects and ship some features too, and help support the rest of the team at on call.

Ops is never, ever going away, my friends, but we can drag it out of the gutter. 🌈

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.