Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Sep. 27, 2019 1 min read

We have ~200 customers, some high volume enterprises, unpredictable traffic in a multitenant distributed system, and a single on call rotation...and our devs get alerted outside business hours around ✨once per week✨

On call does not have to be hell.

Oh, and we deploy code from cron. 🤘

It is absolutely harder to dig yourself out of a hole than to avoid falling into one in the first place. I don't want to minimize the magnitude of the work.

It's entirely probable that if Ben and I hadn't had the experience running ops at Parse, we wouldn't have known exactly what to invest in early on.

And most startups don't have the benefit of an ops co-founder from day one.

And holding the line here isn't free. Emily has had to inconveniently prioritize reliability work at times when we would have preferred to ship features.

We still spend about 30% of our eng time on cleanup and refactoring and things that don't directly move the product forward,

but we do it during normal working hours, and nobody is getting woken up or canceling weekend plans.

You have to treat it like a heart attack. So it doesn't become a chronic state. But this is the way software should be made and ✨owned✨

p.s. There is still plenty of work for opsfolks!!! We have three, counting me. They build out infra projects and ship some features too, and help support the rest of the team at on call.

Ops is never, ever going away, my friends, but we can drag it out of the gutter. 🌈

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.