Observability Engineering: the book that helps FDEs answer “why is the system slow?” without guessing

Picture a customer reporting that “the system is far too slow”. The dashboards are green and CPU looks normal, yet end users are still waiting. At that point many teams go looking for their longest-serving engineer, because only that person “knows the system”.

Observability Engineering: Achieving Production Excellence was written to break that habit. Its three authors, Charity Majors, Liz Fong-Jones and George Miranda, all come from Honeycomb; O’Reilly Media published the first edition on 6 May 2022. If your job is to walk into an unfamiliar system and find out within a few days why it is slow, this is a book worth reading.

What does the book cover, and which edition should you read?

Goodreads sums it up briefly: observability is a prerequisite for building, running and improving complex, business-critical systems. According to the authors’ own series of summaries, the groups of chapters centre on structured events, the analysis loop for debugging, SLO-based alerting and how the whole team works.

One thing to know before buying: Honeycomb has announced a second edition, almost entirely rewritten and twice as long, with 27 completely new chapters.

That edition adds a section on observability governance for technical decision-makers, and the author list changes too, adding Austin Parker, co-founder of OpenTelemetry. The announcement did not come with a release date for the second edition, so check which edition is on sale before you order.

The chapter numbers below refer to the first edition, because the authors’ own summary series is based on it. If you buy the second edition, the numbering will differ.

If you are not ready to buy yet, there is a shortcut. The “Authors’ Cut” series on the Honeycomb blog takes the main concepts from each group of chapters and rewrites them in a more readable form. The posts do not follow chapter order, so you can pick exactly the part you need.

Idea one: every request should tell its own story

The foundation of the whole book is the structured event. In a talk about the book, Honeycomb described linking structured events together to follow a request through a distributed trace, which shows how the components depend on one another.

Take the checkout flow of a retail customer. Each request emits an event carrying the endpoint, customer ID, region, build version, the name of the database called and the processing time. A log line such as “checkout ok, 2300ms” only tells you it was slow. An event with enough fields tells you where it was slow and for whom.

The most common mistake at this stage is instrumenting too few fields. If the checkout event lacks the build version or customer ID, you will never be able to split slow requests along those two dimensions later, however much data you have. The second mistake is recording only errors and ignoring successful requests, when the “successful but slow” requests are exactly what the customer is complaining about.

So the first thing to do on a customer site is to reread the logs of one slow request and ask: is this data enough to separate slow requests from normal ones? If it is not, fix the instrumentation before thinking about optimisation.

Idea two: debug with hypotheses, not memory

Chapters 8, 15 and 16 cover the core analysis loop and whether to build or buy tooling. For the question “why is it slow?”, this is the most valuable part. The authors describe the loop as a powerful tool that is accessible to many people.

Applied to the checkout example, it works like this. You isolate the slow requests, place them next to the normal ones, and look for fields that differ sharply.

Perhaps every slow request is running on a new build, or they all come from one customer with a very large basket. The hypothesis now comes from the data, and you can test it with the next query, instead of relying on someone’s memory of the last incident.

An engineer at Classmethod in Japan, reviewing the book, argued that its biggest benefit is that a team can debug without depending on engineers likened to “living dictionaries”.

The same reviewer made clear that the book does not reject traditional monitoring. It offers a new approach for the point at which a system’s complexity and scale start to exceed what monitoring can handle.

Idea three: alert on what customers actually care about

Chapters 12 and 13 cover SLO-based alerting. The authors’ summary says this approach helps teams respond to what matters most to customers and the business. For FDEs the point is very practical: what the customer cares about is whether orders go through, not whether CPU is at 80%.

That is also the third common mistake at handover: setting alerts on CPU or memory. In the checkout example, CPU can stay low while requests wait on a slow database query, so the on-call engineer gets no alert even though customers already see the slowdown.

The book also points out that adopting observability is not only about instrumenting the application but about how the whole team works together. So at handover you leave the customer a habit as well: SLOs written down, alerts tied to those SLOs, and everyone on call knowing how to question the data.

Who should read it, and in what order?

If you are an FDE, or a backend engineer hoping to move into FDE work, read the sections on structured events and traces first, then the core analysis loop, and only then SLOs. The governance section in the second edition suits people who have to persuade the customer’s leadership to invest in tooling.

On the career side, if the job descriptions you are targeting mention OpenTelemetry, distributed tracing or SLOs, those chapters are the ones to revise first.

On your CV, rather than just writing “used Grafana”, describe a time you instrumented a flow with structured events, isolated the slow requests and found the cause: a concrete story shows how you reason far better than a list of tools.

Next time a customer says “it’s too slow”, you do not need to be the person who has known the system longest in the room, but the one who knows what to ask the data.