# Handing over on-call: health endpoints, metrics and alerts the client can run alone

> A badly written health check can restart an entire cluster because the database flickered for a few seconds, and the person woken at midnight will be on the client's operations team, not you.

Original: https://fdetimes.net/en/guides/health-endpoints-metrics-alerts-handover/

The client's database flickers for about thirty seconds. The health check in the service you handed over calls the database, so it fails too, and the orchestrator restarts every instance in turn. One misbehaving dependency has become a cluster-wide incident.

Microsoft's Health Endpoint Monitoring guidance on the Azure Architecture Center warns about exactly this scenario: liveness checks must be "shallow" so that dependency failures do not cause unnecessary restarts. This kind of fault rarely shows up in a demo. It tends to surface after the FDE has left, when the client's operations team is alone in front of the dashboard.

For an FDE, then, handover does not end when the code runs. The client's team has to be able to run on-call for the service themselves. What follows builds three layers in order: health endpoints, operational metrics, then alerts.

## What will you build, and what do you need?

Picture an order-lookup service. It has one required dependency, a database, and one optional dependency, a product-recommendation cache. Without the cache the service is slower or lacks recommendations, but it can still return order information.

You need a small service you wrote yourself, in any framework, and a dev environment where you can switch dependencies on and off. The code below is **simplified sketches** written in a Python style. Port them to the framework and health-check library you actually use.

## Step 1: liveness answers only "is the process alive?"

Microsoft's example separates the two endpoints entirely. `/healthz/live` returns 200 as long as the process responds. Kubernetes uses the liveness probe to detect a container that has entered a broken state and restart it, so this endpoint should answer exactly one question: does it need restarting?

```python
# Simplified sketch
def live():
    # Do NOT call the database, cache or external APIs here
    return 200, {"status": "alive"}
```

**Check:** shut down the database in the dev environment and call `/healthz/live`. The result must still be 200. If it returns an error, you have just reproduced the mass-restart scenario from the top of this article.

## Step 2: readiness checks dependencies, but knows which ones matter more

`/healthz/ready` is the "deep" check. According to Microsoft, it returns 200 when the application can serve requests and 503 when a required check fails, and each dependency check must have its own timeout. The guidance is also explicit that optional dependencies must not take part in the decision to pull an instance out of rotation.

```python
# Simplified sketch
REQUIRED = {"db": check_db}
OPTIONAL = {"recommend_cache": check_cache}

def ready():
    for name, check in REQUIRED.items():
        if not run_with_timeout(check, seconds=2):
            return 503, {"status": "unhealthy"}
    for name, check in OPTIONAL.items():
        if not run_with_timeout(check, seconds=2):
            return 200, {"status": "degraded"}
    return 200, {"status": "healthy"}
```

The `degraded` state is the most valuable idea in this step. Microsoft notes that deep probes aimed at a slow dependency can cause cascading failures. Its mitigations are timeouts, circuit breakers and graded status reporting: when a non-critical dependency is lost, report degraded, not unhealthy.

The 2-second figure above is only an example. Tune it to the real latency of the client's database.

**Check:** with the cache off, `/healthz/ready` must return 200 with `degraded`. With the database off, it must return 503 within the timeout window, not hang indefinitely.

## Step 3: lock the endpoints down, don't just hide them

Microsoft's guidance is blunt: putting an endpoint on an unusual port or a hard-to-guess path is not access control. Public endpoints should return minimal information. Detailed diagnostics, such as which dependency failed or how many milliseconds it took, must require authentication.

On a client site this usually has to be done with their security team. Ask early which network internal probes travel over and who is allowed to see the diagnostics.

## Step 4: operational metrics means four golden signals, not forty charts

A health endpoint only gives you a snapshot of the present moment. Microsoft distinguishes health monitoring from availability monitoring: availability tracks a service over time to produce uptime statistics. The client's team should therefore have both, one to tell them whether the service is fine right now, and one to tell them how long it has been running well.

On which metrics to track, Google's SRE book sets out four golden signals for any user-facing service: latency, traffic, errors and saturation. For the order-lookup service, that means response time, requests per second, error rate and how full resources are, such as the connection pool.

Latency should be broken down by step. Microsoft recommends measuring the latency of each step in a request and attaching the measurements to that specific request, so bottlenecks can be found quickly. When total latency rises, the operations team must be able to see immediately whether the cause lies in the database query or the cache call.

There are also synthetic users, scripts that simulate a user walking through a predefined sequence of steps, alongside request tracing and logging of exceptions and warnings. For the example service, a synthetic user might log in, look up a sample order number and check the result.

## Step 5: an alert that wakes someone must come with something to do

This is where many handovers fail. Microsoft advises alerting only when the health of critical flows changes significantly, not on every failed probe, and tuning timeouts and thresholds so the system does not react to transient faults.

For example, a rule saying "readiness fails continuously for more than a few minutes" is better than "readiness fails once". Agree the exact numbers with the client.

Google's SRE book sets two requirements: every page must lead to an action, and alert rules meant for humans must be easy to understand and represent a clear failure. One way to apply this is to write each rule as data, with a field already set aside for the first thing to do:

```python
# Simplified sketch: not the syntax of any particular alerting tool.
# Port it to the syntax of the monitoring system the client uses.
ALERTS = [
    {
        "name": "ready_unhealthy",
        "level": "red",
        "condition": "readiness returns 503 continuously for 5 minutes",
        "page": True,
        "first_action": "Check database connectivity and status",
    },
    {
        "name": "ready_degraded",
        "level": "yellow",
        "condition": "readiness reports degraded continuously for 30 minutes",
        "page": False,
        "first_action": "Log it, fix the cache during business hours",
    },
]
```

The 5-minute and 30-minute thresholds are only examples. The more important principle: if you cannot fill in `first_action` for a rule, that rule should not yet have `page: True`.

Microsoft adds that the system should report an unhealthy component within seconds. It also describes a red, amber, green model, in which amber means only part of the functionality is working. That model maps onto the three states from Step 2: unhealthy, degraded and healthy.

**Key point:** If an alert does not come with a concrete action, do not wake anyone up for it.

Don't forget the security layer. According to Microsoft, repeated failed logins can signal a brute-force attack, and an abnormal surge in requests can signal DDoS. Both signals deserve their own alerts, routed to the client's security team.

The final handover deliverable is a table like the one below, placed directly in the runbook:

| Alert | Meaning | On-call engineer's first action |
|---|---|---|
| Red: sustained readiness 503 | Required database not responding | Check database connectivity and status |
| Amber: sustained degraded | Recommendation cache lost, orders still retrievable | Log it, fix the cache during business hours |
| Query-step latency spikes | Bottleneck is in the database | Review slow queries and the connection pool |
| Spike in failed logins | Possible brute-force attack | Hand over to the security team |

**Check:** give the table to someone on the client side who has never read the code. Any row that makes them ask "so what do I do?" should not yet be used to wake the on-call engineer.

## Common mistakes

The most common mistake is using one endpoint for both liveness and readiness. That single endpoint is then forced to touch dependencies, and any dependency incident can lead to a restart, precisely what Microsoft's shallow-check recommendation is meant to prevent.

The next is treating all dependencies alike, to the point where a secondary cache can pull an instance out of rotation. Then there are checks written without timeouts, so a slow database drags the probe into hanging with it.

On the alerting side, the common mistake is paging on every probe. The operations team receives a few false alarms, grows used to them, mutes the notifications, and when a real incident arrives nobody is paying attention.

## How does this skill show up on a client site and on your CV?

On a client site, the first thing to do is sit down with the operations team and ask which monitoring tools they use and who covers the night shift. Only then write the endpoints and alert rules to fit their system. However good a set of health checks is, it helps nobody if it is not wired into the client's on-call channel.

On your CV, instead of a generic "experience with monitoring", be specific: you separated liveness from readiness, added a degraded state and wrote a runbook so the client's team could run on-call themselves.

The measure of a good handover is simple: after you leave, on the first night with an incident, the client's team resolves it without calling you.

**Try this week:**

- Open a service you are working on, find its existing health check and see whether it calls a database or an external API. If it does, split it into live and ready.
- Switch off an optional dependency in the dev environment and confirm that /healthz/ready returns degraded, not 503.
- Write a list of alert rules for that service, each with a 'first action' field, then have a colleague who has never touched the code read it.

## Sources

- [Health Endpoint Monitoring Pattern - Azure Architecture Center | Microsoft Learn](https://learn.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring)

- [Best Practices for Monitoring and Diagnostics - Azure Architecture Center | Microsoft Learn](https://learn.microsoft.com/en-us/azure/architecture/best-practices/monitoring)

- [Monitoring Distributed Systems - Google SRE Book](https://sre.google/sre-book/monitoring-distributed-systems/)

- [Configure Liveness, Readiness and Startup Probes | Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)