Healthcheck Duration Can Warn Before Healthcheck Failure

A probe may keep returning success while taking progressively longer, showing a dependency slowdown before Docker marks the container unhealthy. On the finished hserver stack, healthcheck duration alongside health state is the signal that makes the difference visible. Latency degradation is often an earlier symptom than binary failure, especially for database or HTTP readiness checks.

The engineering pattern here is leading-indicator monitoring. Good monitoring should shorten diagnosis, so I prefer a small number of signals with clear semantics over a larger collection whose meaning is unclear during a failure.

Operationally I keep this constraint: Trend probe duration and investigate rising baselines before increasing timeouts, because a larger timeout can hide the real regression. It gives the dashboard, alert, and runbook the same interpretation instead of letting each layer invent its own definition of healthy.

The implementation can be traced to hserver commit b65d5d4. That provenance is part of the article because these notes document an actual production observability system, not a hypothetical monitoring design.