Charity Majors+ Your Authors @mipsytipsy cofounder/CTO @honeycombio, co-wrote Database Reliability Engineering, loves whiskey, rainbows, and Friday deploys. I test in production and so do you. 🌈 Feb. 26, 2019 1 min read + Your Authors

1. The only lens you should need for debugging your code is that of your own instrumentation.

2. Never let code into production unless you can answer the question: how will I know if this breaks or doesn't work?

3. Instrument your code while pretending you're a serverless dev, and you'll do all the right things.

4. Instrumentation isn't just for "at scale", it is the only way to inspect what your code *actually does* when it meets infra and users. It will improve your life today.

5. Magic auto-instrumentation will only ever catch the dumbest, most generic problems. It doesn't have to be *hard* or tedious to instrument, but someone, somewhere needs to roughly understand your code when instrumenting.

6. Arbitrarily wide events are the foundational data type for debugging, metrics and aggregates and traces can all be derived from them. Context is everything. Flexible schemas a must.

7. Structure. Your. Fucking. Data.

8. You must be able to run read-time aggregation and computation across raw events in order to ask new questions without shipping new code.

9. Sample -- don't pre-aggregate -- to manage costs. Practice dynamic sampling, retain fewer common events and more of uncommon.

10. Delete all your paging alerts. Rely on the big three and a few end-to-end checks or other high level proxies for user pain. Don't rely on page bombs for debugging. Only page if users are impacted, and work towards a world where everything can fail without impacting users.

There you go, ten commandments of instrumentation sprung forth ad hoc-edly on my walk home. Who wants to argue with me, or replace or amend any of them? The floor is open... 🌈💖🤗

You can follow @mipsytipsy.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.

Follow Threader