Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Apr. 09, 2019 1 min read

It's not often I realize I may have *under*-ranted about a particular topic, esp in the o11y space, but i just realized I've been letting pre-aggregated data off the hook in one very critical way: cost.

I have talked at length about how metrics strip context, how aggregation defines (and locks you into) only the questions you could predict having to ask, which is why metrics can't deliver observability (which encompasses unknown-unknowns).

But I haven't talked about cost. And it's ironic, because the metrics kids like to rag on the logging/events providers for being. And it's true: storing rich, wide events does look costly next to "cheap" metrics.

But you know what's *really* costly?

Having to store another copy of everything that happens for every. single. fucking. use case.

Having to shell out for yet another tool every time you want to interrogate your systems in some marginally different way, or by a different team.

A request enters your system, pings around a bit and exits -- once.

And you store the information about it in how many different places?

* monitoring/metrics
* logs
* traces
* APM
* exception tracker
* B.I.
* front end metrics
*...

And you pay for it again, every time.

I'm not one to skimp on o11y spend; I think you should spend 10-30% of your infra budget on tools to understand it and answer questions about it.

But that's 10-30% *total* spend. Most of these tools are priced to cost you 10-30% *each*.

Those tools would be reasonably priced if they were the only tool you need.

But they aren't.

Because they pre-aggregate.

You can only achieve observability if you have access to raw events, so you can ask new questions at will without shipping new code.

And while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events.

The reverse is impossible.


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.