Disk Usage Alerts Need Warning and Critical Bands
A single disk threshold gives operators no distinction between early cleanup work and a filesystem that is close to stopping writes.
Prometheus
Measure What the Operator or User Actually Cares About
CPU and container counts are supporting signals; service reachability, backup validity and call-path health are closer to the real objective.
Thermals on a 2014 Mac mini Are a Production Signal
A production-engineering deep dive into thermals on a 2014 mac mini are a production signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Critical systemd Units Need Their Own Health Signal
Container monitoring does not cover host services such as Docker, networking, tunnels, backup timers or other systemd-managed dependencies.
Linux PSI Changed How I Think About Saturation
A production-engineering deep dive into linux psi changed how i think about saturation, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Green Dashboard Can Hide a Dead Data Pipeline
A panel can render old data successfully after the collector behind it has already stopped.
Prometheus Labels Are Part of the API
Label names and cardinality are query contracts for dashboards, recording rules and alerts.
Clock Synchronization Is a Monitoring Dependency
Logs, TLS checks, backup ages and distributed event ordering all become harder to trust when the host clock drifts.
cAdvisor Used 428 MiB Until We Asked What We Actually Needed
Observability can become the workload if collection is broader than the questions operators actually need to answer.
Disk Throughput Is Not Disk Latency
A production-engineering deep dive into disk throughput is not disk latency, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Recording Rules Are a CPU Trade: Compute Once or Query Repeatedly
A production-engineering deep dive into recording rules are a cpu trade: compute once or query repeatedly, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why Every Target Should Not Be Scraped Every 15 Seconds
A production-engineering deep dive into why every target should not be scraped every 15 seconds, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Observability Became Useful When Every Graph Had an Operational Question
Dashboards improved once I stopped collecting attractive metrics and started collecting evidence for specific failure modes.
Swap Usage Needs Duration and Context
A few megabytes of swap on an old Linux host did not automatically mean an incident, especially after long uptime.
Freshness Guards Prevent Stale Metrics from Firing Misleading Alerts
A target that disappeared can leave its last sample available long enough for threshold expressions to evaluate against stale data.
Why I Monitor MemAvailable Instead of “Free RAM”
A production-engineering deep dive into why i monitor memavailable instead of “free ram”, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Five-Minute Cache Was the Right Place for Expensive Docker Storage Inventory
Not every metric needs to be collected at the same cadence; expensive inventory can be cached without weakening real-time health signals.
Metric Cardinality Is a Capacity Problem Disguised as a Naming Problem
Labels make dashboards flexible, but uncontrolled label values multiply time series and memory cost quickly.
30-Day Retention and a 15 GB Cap Were Capacity Decisions
A production-engineering deep dive into 30-day retention and a 15 gb cap were capacity decisions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Scrape Interval Should Match Signal Speed and Collection Cost
CPU can change meaningfully in seconds while Docker image storage or Grafana process memory does not need the same fifteen-second collection cadence.
Zero Firing Alerts Does Not Mean Monitoring Is Healthy
A broken rule evaluator or missing target can produce a beautifully quiet alert dashboard while the system is blind.
TCP Retransmit Ratio Is Better Than Counting Retransmits Alone
A few retransmissions during heavy traffic may be normal, while the same count during low traffic can represent a serious quality problem.
Pending Alerts Are Useful During Change Windows
A deployment may push a metric over threshold without immediately firing because the configured `for` period has not elapsed.
Normalize Load Average by CPU Count
A load average of four means something very different on a two-core system than on an eight-core system.
Memory Usage Is Not the Same as Memory Pressure
The host looked busy enough that a simple percentage could easily become the whole diagnosis, but Linux memory reclaim makes that misleading.
Drachtio Connectivity Is a Control-Plane Health Signal
The SIP proxy can be running as a process while its control connection to drachtio is down, leaving signaling logic unable to operate correctly.
Backup Age Is a Freshness Signal, Not a Recovery Guarantee
A recent backup timestamp can look reassuring even when the archive is incomplete, corrupt or impossible to restore.
Container Restart Counts Are Incident Breadcrumbs
A service can look healthy now and still have restarted repeatedly overnight, erasing the evidence from a simple current-state view.
SIP Proxy Failure Rate Needs a Window
A single SIP proxy error may be harmless noise, but repeated failures over a short interval can indicate backend, routing or dependency trouble.
Load Average Without CPU Count Is Almost Meaningless
A production-engineering deep dive into load average without cpu count is almost meaningless, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Storage Cache Needs Its Own Freshness Metric
Caching Docker storage inventory reduced overhead, but a silent refresh failure could otherwise leave old values looking current indefinitely.
Prometheus Series Count Is an Observability Capacity Metric
The hserver stack carried roughly twenty-seven thousand active Prometheus series, enough that label growth and exporter changes could materially change memory and storage cost.
Backup Job Exit Status Belongs in Prometheus
A systemd timer can fire on schedule while the backup service itself fails, times out or exits before producing a valid set.
How I Know Caddy Is Healthy but One Upstream Is Not
The edge process can be green while one proxied service is failing behind it.
Temperature Monitoring Belongs Beside Load
A small Mac mini running many containers can hit thermal constraints before ordinary CPU graphs explain why performance changed.
Notification Endpoint Health Needs Direct Monitoring
Delivery counters only change when an alert is sent, so a quiet system could leave a dead notification service unnoticed for hours.
Clock Synchronization Is an Availability Dependency
A production-engineering deep dive into clock synchronization is an availability dependency, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring TCP Retransmission on a Server That Also Runs VoIP
A production-engineering deep dive into monitoring tcp retransmission on a server that also runs voip, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Swap Usage Alone Is a Bad Memory Alert
A production-engineering deep dive into swap usage alone is a bad memory alert, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Rule Evaluation Failures Mean Monitoring Logic Is Broken
A production-engineering deep dive into rule evaluation failures mean monitoring logic is broken, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Prometheus Is My Metrics Control Plane, Not Just a Scraper
A production-engineering deep dive into prometheus is my metrics control plane, not just a scraper, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
PostgreSQL Connection Utilization Needs a Denominator
A count of sixty active PostgreSQL connections is meaningless until it is compared with the configured maximum for that server.
27.5k Active Series on a Small Server: What That Number Actually Means
A production-engineering deep dive into 27.5k active series on a small server: what that number actually means, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Disk Full and Inodes Full Are Two Different Outages
A production-engineering deep dive into disk full and inodes full are two different outages, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
WAL, Head Block and Why Prometheus Memory Does Not Equal Stored Data
A production-engineering deep dive into wal, head block and why prometheus memory does not equal stored data, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The `for` Clause Is an Anti-Flap Tool, Not Decoration
CPU, packet loss and endpoint probes can cross thresholds for a few seconds during harmless transitions or deployment activity.
Cardinality Is Usually a Schema Problem
A production-engineering deep dive into cardinality is usually a schema problem, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Prometheus Metrics Changed How I Looked at a Mobile Core Lab
Logs explain individual failures; metrics show whether the system is drifting before the failures become obvious.
Metric Relabeling Is Resource Control
A production-engineering deep dive into metric relabeling is resource control, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Packet Drops Are More Actionable Than Raw Bandwidth
High interface traffic can be completely healthy, while a small but sustained drop rate can damage voice, APIs and tunnel reliability.
Metric Relabeling Can Be a Memory Optimization
Grafana, Loki and Alloy expose many internal metrics that are useful for development but unnecessary on a small production Prometheus.
One Deadlock Is Worth Recording
A PostgreSQL deadlock can resolve automatically by aborting one transaction, leaving the service apparently healthy after the incident.
How Much Prometheus Is Too Much for an 8 GB Machine?
A production-engineering deep dive into how much prometheus is too much for an 8 gb machine?, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Page Cache Is Not a Memory Leak
A production-engineering deep dive into page cache is not a memory leak, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Container Network Errors Matter More Than Throughput Alone
RX and TX graphs looked busy enough, but throughput by itself could not tell whether traffic was healthy.
Retention Needs Both Time and Size Limits
Keeping thirty days of Prometheus data is useful until series growth causes the TSDB to consume more disk than the host can safely spare.