Charity Majors
+ Your AuthorsArchive @mipsytipsy cofounder/CTO @honeycombio, co-wrote Database Reliability Engineering, loves whiskey, rainbows, and Friday deploys. I test in production and so do you. 🌈 Jul. 11, 2019 6 min read

I am late sharing this link, but I needed time to do it justice. 🌷

The piece is called "How Dark Deploys In 50ms", by my dear, brilliant friend @paulbiggar, the cofounder/CTO of @darklang who previously (this is important) cofounded @CircleCI.

You might already be side-eyeing the title, wondering why I am telling you to read such obvious product marketing clickbait for a product youve never heard of, complete with implausibly grandiose numbers that bring to mind the bad old days of "MongoDB is Web Scale!"

Or as @markcallaghan calls it, "benchmarketing". 🎀🎀🎀

(Oops lol the link is here: )

But get past the subject line, because the very first sentence should fucking *slay you*.

"Speed of developer iteration is the single most important factor in how quickly a technology company can move."

This sentence is a weapon. This sentence should be on your wall.

This sentence should bracket every single convo about developer tooling.

It doesn't mention the term "deploy", yet is all the first principles you need to solve yours. Canaries, progressive deploys, Friday locks, per-commit builds and all the rest are just weeds we get lost in.

Now read the section on the CI/CD pipeline and all the steps involved in shipping a single change safely to prod.

... I'll wait.

(Are you crying yet?)

This part hit me in the gut the hardest. ✨Most of these steps now add no value, only risk✨. 😢💧💦🌧🌨⛈🌩⚡️⚡️⚡️

Oh it gets worse. Remember this is the BEST CASE SCENARIO. Most teams are nowhere near this level of elaborately pointless spaghetti-gineering!

Go on, read the rest. Watch how exactly Dark lives up to its hypey subject line, in particular its INTENSE usage of feature flags like whoaaaa-a-a-a and its tweakered kinda composability thru batshit pseudo immutability.

This crazy shit is the future of software. Bank on it.

(I didn't say /soon/, but someday!)

Are you seeing what is happening here? The founder of the leading CI/CD provider has just washed his hands of the whole shebang and is saying

🌟ENOUGH already🌟

Fuck all these pipelines and falsified envs. Test in prod. Hell DEV in prod.

He is not saying you should ssh in and vim the file, for heaven sakes, and nor am I.

He is saying that we need to tighten that feedback loop between writing code and understanding it running in prod, by orders of magnitude. And building guard rails around prod so it's safe.

Now, Paul is a fucking visionary, which to me means we may be a decade or more away from widespread adoption of even his tamer ideas. If you feel like the day calls for an operatic spit take, for example, feast your eyes on this one throwaway line:

I'm just... I can't even be mad about it, out of my soupy stew of reactions/emotions I think I've got to go with "charmed".

Yes. My editor is now my staging env that tests in prod and eliminated deploys and CI/CD and it also runs my infra and somehow I am fully charmed by it.

(Yes, emacs was my first text editor, M-x why-do-you-ask)

Also, just in case you missed it there, yes of *course* Dark ships with its own artfully alluded-to "data store". Much like honeycomb's own "storage engine".

Because nobody should ever write a database. So fortunate that both of us have clearly found a way to Not Do That.

Jokes aside, I hope that everyone who participated in the Friday deploys debate will read this piece.

Take a breath. Look up. Look out a year, two years, five. Where are you going? Where do you *want* to be going? How can you work backwards from that?

I fucking get what it's like to be down in the weeds and tired. I regret the times I have gotten overly excited and started sounding judgmental and prescriptive. There is no one path, I cannot walk yours.

But -- man, the contours of tomorrow are starting to take shape.

When we started honeycomb, we knew none of this. We had a stubborn conviction that an unmet need existed, but no words to describe it.

Figuring out the language of observability and cardinality and unknown-unknowns was astronomically harder than building the actual product.

But as we slowly picked and sifted and sprayed and stumbled our way forward to a grammar of distributed systems observability, a whole fabric of philosophies emerged with it.

Like what? Well, for example:

📍 developers must own their code
📍 operational skills are no longer nice-to-have
📍 the ops profession now consists of expert consulting for devs owning their code, plus building infra services and tooling
📍 the distributed-ness of systems is increasingly what matters most

📍 treating failure as "interesting, but not important" (~ @this_hits_home)
📍 embracing the fuuuuuck outta failures
📍 devs must be on call, but *on call must not be shitty*. No more gauntlets
📍 reliable, resilient teams of well rested people
📍 observability, not monitoring

📍 feature flags
📍 deploy per commit
📍 faster velocity; deploys should not be scary!
📍 microservices, polyglot persistence, "many" clients
📍 the inevitability of "test in prod", and what to do about it

... etc.

At some point I realized that we weren't the only ones.

Folks like Dark, Launch Darkly, Intercom, ... There are a slew of companies out here independently coming to very similar conclusions about the future of building and shipping software. (And no they aren't all dev tool vendors, thx smartass ☺️)

Anyway, back to the original point of this thread: deploys. If you're trying to make change happen in your org, deploys are usually the best place to start, with oodles of low hanging fruit and pain you can solve.

But keep fixed in your mind what Paul said:

The speed of 🌟dev iteration🌟 is everything. Deploys themselves are not the point. Only a means to an end; an unfortunate, fragile means.

Lastly, I just want to point out how right and beautiful it is for a CI/CD pioneer to light the path towards their destruction.

... I take that back. One more point to make!!

Now that you've savored the piece in its entirety, let's talk about what this means for #AIOps and how machine learning will save us... Oh, what?

AI isn't mentioned /at all/?!

Of course it fucking isn't. Because it's ~irrelevant.

If there's one thing these new sets of tools and philosophies unite around, it's that they are vigiriudkyhuman-centered, and their utility is designed for enhancing, suggesting, nudging, amplifying the human in the driver's seat.

Not replacing them. Never replacing them.

I don't see this changing for the next decade or two, if ever. I'm not smart enough to forecast the future.

But I'm plenty smart enough to take a word people like and slap it all over my marketing pages, like the monitoring kids have been doing with "observability". lol

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.

Follow Threader