Charity Majors @mipsytipsy CTO @honeycombio, ex-Parse, Facebook, Linden Lab; cowrote Database Reliability Engineering; loves whiskey, rainbows. I test in production and so do you. 🌈🖤 Feb. 26, 2019 1 min read

1. The only lens you should need for debugging your code is that of your own instrumentation.

2. Never let code into production unless you can answer the question: how will I know if this breaks or doesn't work?

3. Instrument your code while pretending you're a serverless dev, and you'll do all the right things.

4. Instrumentation isn't just for "at scale", it is the only way to inspect what your code *actually does* when it meets infra and users. It will improve your life today.

5. Magic auto-instrumentation will only ever catch the dumbest, most generic problems. It doesn't have to be *hard* or tedious to instrument, but someone, somewhere needs to roughly understand your code when instrumenting.

6. Arbitrarily wide events are the foundational data type for debugging, metrics and aggregates and traces can all be derived from them. Context is everything. Flexible schemas a must.

7. Structure. Your. Fucking. Data.

8. You must be able to run read-time aggregation and computation across raw events in order to ask new questions without shipping new code.

9. Sample -- don't pre-aggregate -- to manage costs. Practice dynamic sampling, retain fewer common events and more of uncommon.

10. Delete all your paging alerts. Rely on the big three and a few end-to-end checks or other high level proxies for user pain. Don't rely on page bombs for debugging. Only page if users are impacted, and work towards a world where everything can fail without impacting users.

There you go, ten commandments of instrumentation sprung forth ad hoc-edly on my walk home. Who wants to argue with me, or replace or amend any of them? The floor is open... 🌈💖🤗


You can follow @mipsytipsy.



Bookmark

____
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Threader is an independent project created by only two developers. The site gets 500,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Running this space is expensive and time consuming. If you find Threader useful, please consider supporting us to make it a sustainable project.