Hserver Monitoring: Observability Architecture · advanced
Freshness Guards Prevent Stale Metrics from Firing Misleading Alerts
A target that disappeared can leave its last sample available long enough for threshold expressions to evaluate against stale data.
A target that disappeared can leave its last sample available long enough for threshold expressions to evaluate against stale data. On hserver the first signal I use for this question is time() - timestamp(metric) freshness conditions in alert rules. Adding freshness constraints distinguishes a current bad value from an old value whose exporter or target is no longer updating.
The important part is interpretation rather than collecting another graph. staleness-aware alerting. That gives the metric a specific operational job instead of making it another number on a dashboard.
The practical control is straightforward: Use explicit freshness checks where stale samples could create false positives and pair them with target-down alerts for the missing telemetry path. This also gives me a repeatable check after deployments, exporter changes, or capacity tuning.
Repository evidence for this monitoring behavior is commit b65d5d4. I keep that reference with the note because a monitoring conclusion is stronger when the configuration and runtime decision that produced it can be inspected later.