Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Feb. 13, 2019 3 min read

I get this a lot: "Do you support on prem? I'd really like to use you guys for observability, but I can't convince the security team to let me." (forlorn look)

So listen up, because I'm about to give you my answer as well as something much better: answers from *actual experts*.

"No, we don't support on prem. Sorry, we have a very small team and more importantly

👋it's 2019👋

Also, observability is something you should affirmatively WANT to use a third-party service in the cloud for. Let's think this through...

"What happens if you run your systems and your o11y tooling on the same hardware facilities, network, etc as your production services? What happens to your o11y when prod goes down?

Now you have n^2 problems. *And* you're stumbling around in a closet blind drunk."

"In theory you could sequester your observability away from prod wholly & completely. I've never seen anyone successfully do that, but you /could/. But you cannot sequester the humans, so they will break things.

And who monitors your monitoring systems?

... omg wtf just happened, where did that chain of tweets come from and it became untweetable until I hit "tweet all" ... 😱

wait, so now nobody will know how long it takes me to compose shit? i can edit? THIS FEELS LIKE CHEATING

(anyhow ... back to the plot now that i've found the tweet button :P)

"in order to protect against human error, infra bugs, and availability zone outages, in order to preserve your ability to debug during outages; the best solution is to outsource your o11y. Use a service."

I consider all the other benefits of outsourcing to be nice-to-haves compared to this one. But they include things like nice features, cutting-edge visualizations, ease of use, oh and NOT HAVING TO HIRE A TEAM TO DO IT

whatever the vendor may cost, it won't compare with that.

"But what about my security?"

Well first of all, it's important to distinguish operational data from auditable data. I wrote a whole long-ass thread about this a few months ago, so read that.

The two types of data have radically different characteristics, footprints, use cases, etc. You should separate the streams as early as possible. No PII/PHI should make it into your operational data. Practice good data hygiene, and it should be easy to stream offsite.

More on operational data and what to gather, how to structure it, etc here:

or you can just use the honeycomb beelines, they do everything automagically. For extra scrubbage, one-way hash column values before you flush to disk: retains analytic value.

Ask your vendor if they have the ability to scan periodically from cron for PII/PHI, and if they can automatically shut down your ingestion upon detection until someone has okayed it. This isn't hard.

Good security teams are like good lawyers. They don't exist to tell you no; they exist to tell you *how to get to yes*. Any modern security team should be capable of telling you how to outsource your metrics and observability.

But now we've reached the edge of my knowledge. Which is why I reached out to some actual experts for help.

I just posted a three-piece series of guest posts on my blog to answer this question: "how can I get security to let me use a vendor for my observability?"

First: how to choose a third-party vendor and successfully champion them to your security team, by @georgechamales.  https://charity.wtf/2019/02/13/5247/ 

Has a super handy worksheet for you to fill out before approaching your security team. (Useful for vendors to fill out beforehand, too!)

Second: how to practice seeing security's paranoid point of view, and why it matters; and how to maintain your relationship with security over the long run. *great* stuff on empathy from @attacus_au.

 https://charity.wtf/2019/02/13/outsource-your-o11y-get-aligned-with-security-part-2-3/ 


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.