This is a great question, and I have made a note to write an article and chapter about it.
The tldr: I think of observability as being a tight feedback loop around building, fixing, changing code in systems, which relies on your having *some* ability to instrument.
But maybe this a tension between the technical definition I espouse (can you ask+answer novel questions to understand any system state), and what constitutional lawyers would call its penumbra.
There's no reason you *must* be able to instrument, just in practice you usually do.
There are certainly ways of assembling and approximating this data without new instrumentation, *if* the software is emitting it in the first place.
I am thinking of what I've done with databases in the past, like MongoDB and Mysql.
For Mysql, we approximated o11y through an unholy mix of sampled full query logging, streaming over tcpdump and reconstructing transactions on a separate host, plus a cron job that connected to the terminal and dumped stats every could seconds.
Gross, but works pretty well.
You can follow @mipsytipsy.
Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.
Enjoy Threader? Sign up.
Since you’re here...
... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.