Warning and Critical Are Different Response Contracts
A production-engineering deep dive into warning and critical are different response contracts, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Alertmanager
Self-Hosted ntfy Removed an External Delivery Dependency
The alert sink originally targeted a public ntfy service, which made production notification depend on infrastructure outside the hserver control boundary.
Grouping and Inhibition Stop One Failure Becoming Twenty Pages
A production-engineering deep dive into grouping and inhibition stop one failure becoming twenty pages, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Multi-Window Burn-Rate Alerts on a Tiny Self-Hosted Stack
A production-engineering deep dive into multi-window burn-rate alerts on a tiny self-hosted stack, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
What Prometheus `for:` Really Buys You—and What It Cannot
A production-engineering deep dive into what prometheus `for:` really buys you—and what it cannot, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Fast Burn and Slow Burn Are Different Incidents
A production-engineering deep dive into fast burn and slow burn are different incidents, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Notification Authentication Belongs Outside Git
Moving alert delivery to authenticated self-hosted ntfy introduced a publisher token that the alert-sink needs at runtime.
Alert Fatigue Is an Architecture Bug
A production-engineering deep dive into alert fatigue is an architecture bug, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Every Alert Should Suggest the Next Investigation
A production-engineering deep dive into every alert should suggest the next investigation, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitor Notification Success, Failure and Resolution Separately
An alert can fire correctly and still never reach the operator if Alertmanager or the external delivery path fails.
Threshold Alerts vs Symptom Alerts
A production-engineering deep dive into threshold alerts vs symptom alerts, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Severity Mapping Should Reflect Operator Action
If every alert is critical, operators lose the distinction between conditions that require immediate intervention and those that need scheduled review.
Alert Delivery Failure Needs an Alert of Its Own
A monitoring system that detects failures but cannot notify anyone is partially failed even if every Prometheus target remains green.
Why 99.9% Uptime Does Not Tell Me When to Wake Someone Up
A production-engineering deep dive into why 99.9% uptime does not tell me when to wake someone up, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Resolved Notifications Are Part of the Incident Lifecycle
A production-engineering deep dive into resolved notifications are part of the incident lifecycle, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.