Disk Usage Alerts Need Warning and Critical Bands
A single disk threshold gives operators no distinction between early cleanup work and a filesystem that is close to stopping writes.
TOPIC
Observability & Monitoring
Lessons, explainers, experiments, and implementation notes.
Dead Synthetic Targets Should Be Removed, Not Hidden
Placeholder or obsolete probe targets can stay in configuration after architecture changes and permanently pollute availability dashboards.
Why “Up” Is One of the Weakest Signals in Production
A production-engineering deep dive into why “up” is one of the weakest signals in production, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
IP Addresses Belong in Log Content More Often Than Labels
A production-engineering deep dive into ip addresses belong in log content more often than labels, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Block I/O per Container Exposes Noisy Neighbors
Host disk latency can rise because one container is performing heavy reads or writes while every other service only sees the consequence.
Label Freedom Comes With a Time-Series Cost
Metric labels let us divide one measurement in many useful ways, but every new combination of label values can create another time series. Greater detail also carries collection, storage, and query cost.
Monitoring Secrets Without Monitoring Secret Values
A production-engineering deep dive into monitoring secrets without monitoring secret values, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Thermals on a 2014 Mac mini Are a Production Signal
A production-engineering deep dive into thermals on a 2014 mac mini are a production signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Warning and Critical Are Different Response Contracts
A production-engineering deep dive into warning and critical are different response contracts, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Fresh Backup Can Still Be Corrupt
A production-engineering deep dive into a fresh backup can still be corrupt, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Self-Hosted ntfy Removed an External Delivery Dependency
The alert sink originally targeted a public ntfy service, which made production notification depend on infrastructure outside the hserver control boundary.
Waiting Sessions Tell Me More Than CPU During Database Contention
A production-engineering deep dive into waiting sessions tell me more than cpu during database contention, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Firing, Pending and Rule Count Tell Different Alerting Stories
A dashboard that only shows firing alerts hides whether conditions are approaching thresholds or whether the expected rule set is even loaded.
Critical systemd Units Need Their Own Health Signal
Container monitoring does not cover host services such as Docker, networking, tunnels, backup timers or other systemd-managed dependencies.
Ingress Telemetry Is a Boundary You Must Test
A topology UI can pass synthetic tests and still misread production traffic at the first adapter.
Container Running, Healthy and Useful Are Three Different States
A production-engineering deep dive into container running, healthy and useful are three different states, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Linux PSI Changed How I Think About Saturation
A production-engineering deep dive into linux psi changed how i think about saturation, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How cAdvisor Became One of My Largest Workloads
A production-engineering deep dive into how cadvisor became one of my largest workloads, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Green Dashboard Can Hide a Dead Data Pipeline
A panel can render old data successfully after the collector behind it has already stopped.
Monitoring Redirects Can Create False Green Probes
A health probe that automatically follows redirects may end on an authentication page and report successful HTTP even though the original service route is wrong.
RTPEngine Failure Reasons Are Better Than One Failure Total
Media sessions can close for rejection, timeout, silent timeout, final timeout or offer timeout, and those reasons point to different failure paths.
A Sealed Secret Server Can Be Secure and Still Be Down
A production-engineering deep dive into a sealed secret server can be secure and still be down, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why Green Dashboards Can Still Lie
A production-engineering deep dive into why green dashboards can still lie, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Dashboards Should Follow Troubleshooting Workflows
A wall of attractive graphs is slow during an incident if related signals are scattered by exporter rather than by the question an operator is trying to answer.
Inode Exhaustion Can Break a Filesystem with Free Gigabytes
Free-space graphs can remain green while a workload creates huge numbers of tiny files and consumes the available inode table.
Prometheus Labels Are Part of the API
Label names and cardinality are query contracts for dashboards, recording rules and alerts.
Calibration Defines Zero; Detection Measures Deviation
The useful baseline is the room's normal operating condition, not an imaginary perfectly quiet environment.
Grouping and Inhibition Stop One Failure Becoming Twenty Pages
A production-engineering deep dive into grouping and inhibition stop one failure becoming twenty pages, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Separate Raw Sensor Data from Interpreted State
Presence, motion and occupancy are conclusions; retain enough underlying evidence to debug those conclusions.
Clock Synchronization Is a Monitoring Dependency
Logs, TLS checks, backup ages and distributed event ordering all become harder to trust when the host clock drifts.
Restore Verification Is the Most Important Backup Metric I Have
A production-engineering deep dive into restore verification is the most important backup metric i have, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Multi-Window Burn-Rate Alerts on a Tiny Self-Hosted Stack
A production-engineering deep dive into multi-window burn-rate alerts on a tiny self-hosted stack, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Kernel OOM Logs Are the Ground Truth for Host Memory Kills
A process disappearing can look like an application crash unless the kernel journal is checked for OOM-killer activity.
Volume Reference Counts Help Detect Orphaned Docker State
Old Docker volumes can consume storage after services are removed, and their names alone do not always reveal whether anything still depends on them.
The Observability Dashboard That Watches the Observability Stack
A production-engineering deep dive into the observability dashboard that watches the observability stack, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
DNS Is a Production Dependency, Not Plumbing
A production-engineering deep dive into dns is a production dependency, not plumbing, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Certificate Expiry Is an Availability Signal
TLS can work perfectly today and still have a known future outage date embedded in the certificate.
Monitoring SIP Is Not Monitoring RTP
A production-engineering deep dive into monitoring sip is not monitoring rtp, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
PSI Made Short Resource Spikes Easier to Interpret
Some periods looked acceptable in average CPU and RAM graphs while interactive services still felt slow.
Turning SSH Failures into a Production Security Signal
A production-engineering deep dive into turning ssh failures into a production security signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
OpenSIPS 5xx Spikes Show Signaling Quality
Registration counts and dialog counts can look stable while transaction failures increase for a subset of calls.
Who Does What When an Alert Fires
The value of an alert is not limited to detecting failure. It also includes what someone is expected to do after the signal arrives. If ownership is vague, even an accurate alert becomes just another piece of noise.
OpenSIPS 5xx Spikes Are More Useful Than Total SIP Traffic
A production-engineering deep dive into opensips 5xx spikes are more useful than total sip traffic, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How Logging Itself Can Become an I/O Workload
A production-engineering deep dive into how logging itself can become an i/o workload, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
CPU by Container Is Useful Only Beside Limits
A container using one full CPU may be expected on an unrestricted worker and catastrophic for a service capped at a fraction of a core.
HTTPS Latency Needs More Than One Number
A single total probe duration hides whether slowness came from name resolution, TCP connection, TLS negotiation or server response.
A Healthy FreeSWITCH Process Can Still Mean Broken Calls
A production-engineering deep dive into a healthy freeswitch process can still mean broken calls, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
When Clocks Drift, the Order of Events Can Drift Too
When logs from several servers are compared side by side, timestamps are often used to reconstruct the incident. If the clocks differ, however, a later event can appear to have happened first. Sorting timestamps is not enough to prove causal order.
TLS Certificate Expiration Is an Availability Incident Waiting to Happen
A production-engineering deep dive into tls certificate expiration is an availability incident waiting to happen, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Disk Throughput Is Not Disk Latency
A production-engineering deep dive into disk throughput is not disk latency, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
RPO, RTO and Evidence: When a Home Server Becomes a Production System
A production-engineering deep dive into rpo, rto and evidence: when a home server becomes a production system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Sudo Authentication Failures Belong in the Security Timeline
Repeated sudo failures may be operator error, expired credentials or suspicious privilege-escalation attempts, and they often happen outside application logs.
Docker Security Posture Belongs in Monitoring
Privileged containers, Docker socket mounts, root users and published ports are configuration facts that can silently drift after deployment.
Alert Panels and Investigation Panels Serve Different Humans
A production-engineering deep dive into alert panels and investigation panels serve different humans, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Recording Rules Are a CPU Trade: Compute Once or Query Repeatedly
A production-engineering deep dive into recording rules are a cpu trade: compute once or query repeatedly, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Backup Monitoring Needs UNKNOWN as a Real State
Least-privilege monitoring sometimes cannot read protected backup evidence, and treating that access failure as healthy would be dangerous.
Healthcheck Duration Can Warn Before Healthcheck Failure
A probe may keep returning success while taking progressively longer, showing a dependency slowdown before Docker marks the container unhealthy.
Backup Success Is Not Recoverability
A production-engineering deep dive into backup success is not recoverability, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why My Monitoring System Needed Monitoring Too
A production-engineering deep dive into why my monitoring system needed monitoring too, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Do Not Let Log History Block Current Work
Logs are useful for future investigations, but unlimited retention can let the evidence system consume the resources required by the service itself. Observability needs its own storage budget.
Redis Evictions Mean Memory Policy Is Affecting Data
Redis can remain responsive while silently evicting keys because the configured memory limit has been reached.
Why Every Target Should Not Be Scraped Every 15 Seconds
A production-engineering deep dive into why every target should not be scraped every 15 seconds, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitor the Log Pipeline for Dropped Entries
A logging stack can look healthy at the process level while silently discarding entries because of relabeling, backpressure or write failures.
The Stale Docker Container Problem in Alloy
A production-engineering deep dive into the stale docker container problem in alloy, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Redis Eviction Means Policy Is Already Affecting Data
A production-engineering deep dive into redis eviction means policy is already affecting data, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Monitoring Stack Needs a Memory Budget Too
Observability was becoming one of the larger workloads on a small production server, which is dangerous when monitoring competes with the services it protects.
Swap Usage Needs Duration and Context
A few megabytes of swap on an old Linux host did not automatically mean an incident, especially after long uptime.
Freshness Guards Prevent Stale Metrics from Firing Misleading Alerts
A target that disappeared can leave its last sample available long enough for threshold expressions to evaluate against stale data.
Why I Monitor MemAvailable Instead of “Free RAM”
A production-engineering deep dive into why i monitor memavailable instead of “free ram”, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
SMART Health Is Different from Filesystem Free Space
A disk can have plenty of free capacity while the underlying SSD is reporting temperature or device-health problems.
GitHub Actions as Five-Minute External Infrastructure
A production-engineering deep dive into github actions as five-minute external infrastructure, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Running Containers Are Not Healthy Containers
The Docker daemon can report a container as running even when the application inside it has stopped serving useful traffic.
Commit and Rollback Rates Reveal Database Behavior
Database traffic volume looked normal even when applications were rolling back more transactions than usual.
One PostgreSQL Deadlock Is Worth Recording
A production-engineering deep dive into one postgresql deadlock is worth recording, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Dead-Man Monitoring Should Not Need a Credential from the Dead System
A production-engineering deep dive into dead-man monitoring should not need a credential from the dead system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How I Took cAdvisor from ~428 MiB to ~20–28 MiB
A production-engineering deep dive into how i took cadvisor from ~428 mib to ~20–28 mib, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
30-Day Retention and a 15 GB Cap Were Capacity Decisions
A production-engineering deep dive into 30-day retention and a 15 gb cap were capacity decisions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
FreeSWITCH Session Counts Need Per-Node Breakdown
A healthy total session count can hide a load-balancing problem if one FreeSWITCH node carries nearly all calls while another remains idle.
Redis Fragmentation Can Look Like a Memory Leak
A production-engineering deep dive into redis fragmentation can look like a memory leak, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Database Size Growth Is a Capacity Trend, Not an Emergency Metric
A database normally grows, so alerting on size alone would create noise while ignoring the important question of growth rate and disk headroom.
What Do We Lose When We Do Not Keep Every Trace?
Keeping a complete trace for every request can be expensive. Sampling controls that cost, but it also defines what an investigation can and cannot see. A missing trace is not evidence that the event never happened.
Writable Layer Size Should Not Be Polled Expensively Every Scrape
Docker storage size is useful, but asking the daemon for deep filesystem inventory on every fast scrape consumed too much monitoring overhead.
No Healthy Worker Is a User-Facing Signal
A production-engineering deep dive into no healthy worker is a user-facing signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Restore Alert Should Distinguish Failed from Stale
No recent restore verification and a recent restore verification that actively failed are both bad, but they communicate different operational urgency.
Container I/O Pressure Shows Waiting, Not Just Bytes
A workload can perform modest disk throughput and still suffer because requests are waiting behind slow storage or competing I/O.
A Docker Storage Cache Needs Its Own Freshness Metric
A production-engineering deep dive into a docker storage cache needs its own freshness metric, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Which Identity Should Follow a Call Through the System?
A phone call that crosses several servers leaves pieces of its story in several places. Endpoint logs, PBX logs, and trunk logs may all describe it differently. Time proximity alone is not enough to prove that two log entries belong to the same call.
Scrape Interval Should Match Signal Speed and Collection Cost
CPU can change meaningfully in seconds while Docker image storage or Grafana process memory does not need the same fifteen-second collection cadence.
Ask the Operational Question Before Building the Dashboard
Adding a graph is easy; deciding what action that graph should support is harder. A dashboard can contain every metric and still leave an operator unsure where to begin during an incident.
Zero Firing Alerts Does Not Mean Monitoring Is Healthy
A broken rule evaluator or missing target can produce a beautifully quiet alert dashboard while the system is blind.
Config Drift Is a Monitoring Signal
A production-engineering deep dive into config drift is a monitoring signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
TCP Retransmit Ratio Is Better Than Counting Retransmits Alone
A few retransmissions during heavy traffic may be normal, while the same count during low traffic can represent a serious quality problem.
Seven Days of Logs Was an Engineering Choice, Not a Default
A production-engineering deep dive into seven days of logs was an engineering choice, not a default, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
PostgreSQL Connection Utilization Needs a Denominator
A production-engineering deep dive into postgresql connection utilization needs a denominator, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Pending Alerts Are Useful During Change Windows
A deployment may push a metric over threshold without immediately firing because the configured `for` period has not elapsed.
Normalize Load Average by CPU Count
A load average of four means something very different on a two-core system than on an eight-core system.
No Data Is Not Zero
Replacing missing data with zero can make a graph look cleaner while changing its meaning. Zero says a measurement was made and the result was zero. No data says the result is unknown.
Memory Usage Is Not the Same as Memory Pressure
The host looked busy enough that a simple percentage could easily become the whole diagnosis, but Linux memory reclaim makes that misleading.
SSH Failure Alerts Need Warning and Critical Bursts
A few failed SSH logins are normal on an administered server, while a rapid burst can indicate brute-force activity or a broken automation credential.
OOM Events Need Container and Host Context
An application disappearing under load can be either a container memory-limit event or a host-wide memory emergency.
Drachtio Connectivity Is a Control-Plane Health Signal
The SIP proxy can be running as a process while its control connection to drachtio is down, leaving signaling logic unable to operate correctly.
What Production-Grade Means on Decade-Old Hardware
A production-engineering deep dive into what production-grade means on decade-old hardware, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Metrics, Logs, Probes and State Checks Answer Different Questions
A production-engineering deep dive into metrics, logs, probes and state checks answer different questions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The VoIP Dashboard Must Follow a Call Across Layers
A production-engineering deep dive into the voip dashboard must follow a call across layers, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Backup Age Is a Freshness Signal, Not a Recovery Guarantee
A recent backup timestamp can look reassuring even when the archive is incomplete, corrupt or impossible to restore.
Container Restart Counts Are Incident Breadcrumbs
A service can look healthy now and still have restarted repeatedly overnight, erasing the evidence from a simple current-state view.
Restart Counts Without Context Create Noise
A production-engineering deep dive into restart counts without context create noise, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The cAdvisor Disk Scanner Was More Expensive Than the Data Was Worth
The production monitoring stack measured roughly 428 MiB of cAdvisor memory with filesystem disk collection enabled on a small host.
What Prometheus `for:` Really Buys You—and What It Cannot
A production-engineering deep dive into what prometheus `for:` really buys you—and what it cannot, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
SIP Proxy Failure Rate Needs a Window
A single SIP proxy error may be harmless noise, but repeated failures over a short interval can indicate backend, routing or dependency trouble.
Database Container Metrics Need Engine Metrics Beside Them
High database container CPU or memory can be an important symptom, but it cannot identify whether the engine is busy with useful work, blocked transactions or internal maintenance.
Fast Burn and Slow Burn Are Different Incidents
A production-engineering deep dive into fast burn and slow burn are different incidents, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Slow Users Disappear Inside the Average
A low average response time can make a system look fast even when a smaller group of users waits much longer. Their experience disappears into one number. To understand latency, we also need to understand its distribution.
Load Average Without CPU Count Is Almost Meaningless
A production-engineering deep dive into load average without cpu count is almost meaningless, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Storage Cache Needs Its Own Freshness Metric
Caching Docker storage inventory reduced overhead, but a silent refresh failure could otherwise leave old values looking current indefinitely.
Cloudflare Tunnel Disruptions Need Log-Derived Monitoring
An external endpoint can flap because the tunnel reconnects even when the local service and LAN probe remain healthy.
Restore Verification Needs Its Own Age
A restore drill that passed months ago does not prove that today's schema, credentials and backup format can still be recovered.
I Deliberately Broke the Probe to Test the Monitoring System
A production-engineering deep dive into i deliberately broke the probe to test the monitoring system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Notification Authentication Belongs Outside Git
Moving alert delivery to authenticated self-hosted ntfy introduced a publisher token that the alert-sink needs at runtime.
Journal Cursor Persistence Prevented Duplicate Log Ingestion
Restarting a log collector can replay old journal entries or skip new ones if it does not persist its position correctly.
Prometheus Series Count Is an Observability Capacity Metric
The hserver stack carried roughly twenty-seven thousand active Prometheus series, enough that label growth and exporter changes could materially change memory and storage cost.
Why Every Monitoring Component Had to Justify Its RAM
A production-engineering deep dive into why every monitoring component had to justify its ram, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
OpenSIPS Metrics Availability Is Separate from OpenSIPS Availability
OpenSIPS can continue processing calls while the MI metrics collector fails, leaving the service healthy but observability blind.
Backup Job Exit Status Belongs in Prometheus
A systemd timer can fire on schedule while the backup service itself fails, times out or exits before producing a valid set.
Public and Internal Probes Answer Different Availability Questions
A service can be healthy on the LAN and unreachable through Cloudflare, TLS, DNS or authentication at the public edge.
Temperature Monitoring Belongs Beside Load
A small Mac mini running many containers can hit thermal constraints before ordinary CPU graphs explain why performance changed.
Notification Endpoint Health Needs Direct Monitoring
Delivery counters only change when an alert is sent, so a quiet system could leave a dead notification service unnoticed for hours.
The Server Cannot Report Its Own Death
A production-engineering deep dive into the server cannot report its own death, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Observability Tax on a 7.1 GiB Linux Server
A production-engineering deep dive into the observability tax on a 7.1 gib linux server, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Critical Container Log Spikes Are a Triage Signal
A container can remain healthy by its HTTP probe while its logs suddenly fill with exceptions, retries or failed dependency calls.
Clock Synchronization Is an Availability Dependency
A production-engineering deep dive into clock synchronization is an availability dependency, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
RTPengine Timeout Reasons Tell Different Media Stories
A production-engineering deep dive into rtpengine timeout reasons tell different media stories, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Alert Fatigue Is an Architecture Bug
A production-engineering deep dive into alert fatigue is an architecture bug, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Building a NOC Dashboard for a Phone Screen
A production-engineering deep dive into building a noc dashboard for a phone screen, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Every Alert Should Suggest the Next Investigation
A production-engineering deep dive into every alert should suggest the next investigation, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring TCP Retransmission on a Server That Also Runs VoIP
A production-engineering deep dive into monitoring tcp retransmission on a server that also runs voip, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Swap Usage Alone Is a Bad Memory Alert
A production-engineering deep dive into swap usage alone is a bad memory alert, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Who Monitors the Monitoring System?
If monitoring stops, evidence of failures elsewhere can disappear too. Silence can then look like health. The monitoring system therefore needs evidence that its own collection and notification paths are still working.
Journald Cursor Persistence Prevents Duplicate or Missing Logs
A production-engineering deep dive into journald cursor persistence prevents duplicate or missing logs, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Information Density Without Dashboard Wall Art
A production-engineering deep dive into information density without dashboard wall art, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why a 302 Redirect Can Mean the Service Is Healthy
A production-engineering deep dive into why a 302 redirect can mean the service is healthy, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Rule Evaluation Failures Mean Monitoring Logic Is Broken
A production-engineering deep dive into rule evaluation failures mean monitoring logic is broken, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Counter Reset Changes the Story in the Graph
A counter accumulates events over time, but some counters start over when a process restarts. Subtracting the next value from the previous one can then produce an impossible negative event. The events did not run backward; the measurement history was reset.
A TCP Database Probe Is Not a Database Health Check
A listening PostgreSQL or Redis port proves that something accepted a TCP connection, not that queries, authentication or storage are working correctly.
Building a Dead-Man Monitor Outside the Machine It Watches
A production-engineering deep dive into building a dead-man monitor outside the machine it watches, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
SSH, sudo, Authelia and Docker Errors as One Security Telemetry Plane
A production-engineering deep dive into ssh, sudo, authelia and docker errors as one security telemetry plane, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Buffer Hits, Disk Reads and the Shape of PostgreSQL Cache Pressure
A production-engineering deep dive into buffer hits, disk reads and the shape of postgresql cache pressure, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Observability Needs a Source of Truth and an Acceptance Artifact
A monitoring stack can drift through dashboard edits, local files and runtime tuning until nobody knows whether Git can reproduce what is currently trusted in production.
Prometheus Is My Metrics Control Plane, Not Just a Scraper
A production-engineering deep dive into prometheus is my metrics control plane, not just a scraper, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
PostgreSQL Connection Utilization Needs a Denominator
A count of sixty active PostgreSQL connections is meaningless until it is compared with the configured maximum for that server.
27.5k Active Series on a Small Server: What That Number Actually Means
A production-engineering deep dive into 27.5k active series on a small server: what that number actually means, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Deep Database Collection Did Not Need Copied Passwords
Copying every application database password into the observability stack would have expanded the secret blast radius just to collect metrics.
Memory Limit Utilization vs Working Set: Which One Should Page Me?
A production-engineering deep dive into memory limit utilization vs working set: which one should page me?, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Docker Daemon Error Bursts Need a Separate Alert
Individual container logs can all look normal while the Docker daemon is failing image, network, storage or runtime operations underneath them.
Monitor Notification Success, Failure and Resolution Separately
An alert can fire correctly and still never reach the operator if Alertmanager or the external delivery path fails.
Do Not Turn Source IP into a High-Cardinality Metric Label
Security logs contain useful source addresses, but promoting every IP to a Prometheus or Loki index label would create unbounded cardinality on an Internet-facing service.
OOM Events Are Better Evidence Than “High Memory” Alone
A production-engineering deep dive into oom events are better evidence than “high memory” alone, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
What We Measure Is Not the Whole System
A dashboard can create the feeling that we are looking at the entire system. In reality, we are looking at selected measurements. What we measure, where we measure it, and how often already forms a model.
Checksum Monitoring Detects Silent Backup Corruption
A backup directory can exist with the expected filenames while one archive is truncated or modified after creation.
Backup Journals Make Failed Automation Debuggable
A red backup alert identifies the outcome but usually does not explain which command, mount or permission caused the failure.
Disk Full and Inodes Full Are Two Different Outages
A production-engineering deep dive into disk full and inodes full are two different outages, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
No Healthy SIP Worker Is a User-Facing Failure Counter
The most important load-balancer failure is not that a backend probe failed; it is that an incoming SIP request could not be assigned to any healthy worker.
FreeSWITCH Health Is a Fleet Count, Not One Boolean
A multi-worker voice stack can keep serving calls after one FreeSWITCH node fails, so one global up/down flag hides degraded capacity.
Internal Probe and Public Probe Answer Different Questions
A production-engineering deep dive into internal probe and public probe answer different questions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Threshold Alerts vs Symptom Alerts
A production-engineering deep dive into threshold alerts vs symptom alerts, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Disk Busy Time Needs Latency Beside It
A device at high utilization can be handling work efficiently, while a lower-utilization device can still be returning slow requests.
Authelia Authentication Failures Need Context, Not Panic
A two-factor login system naturally records failed credentials, expired sessions and rejected access, so any single failure is not automatically an attack.
Volume Size Monitoring Needs Ownership Context
A large Docker volume is not actionable if the dashboard cannot tell which service owns it or whether that growth is expected.
Disk Latency Explains More Than Disk Throughput
The host can show modest megabytes per second while applications still wait because each storage request takes too long.
CPU Utilization Without CPU Modes Hides the Bottleneck
The host can report high CPU usage even when the useful question is whether time is going to user work, system work, steal, or I/O wait.
One Machine, Many Failure Domains: Mapping the 2014 Mac mini
A production-engineering deep dive into one machine, many failure domains: mapping the 2014 mac mini, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Severity Mapping Should Reflect Operator Action
If every alert is critical, operators lose the distinction between conditions that require immediate intervention and those that need scheduled review.
Why Loki Labels Need to Stay Boring
A production-engineering deep dive into why loki labels need to stay boring, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
DNS Lookup Time Belongs in an Edge Dashboard
A slow public request is not always slow application code; DNS resolution can consume a meaningful part of the user-visible path.
Monitoring the System That Is Supposed to Save the System
A production-engineering deep dive into monitoring the system that is supposed to save the system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
WAL, Head Block and Why Prometheus Memory Does Not Equal Stored Data
A production-engineering deep dive into wal, head block and why prometheus memory does not equal stored data, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Wi-Fi Signal Needed a Fallback Path
The host collector originally relied on `iw`, but driver and interface behavior did not always expose the active connection signal consistently.
A Successful TCP Connection Does Not Mean the Database Is Healthy
A production-engineering deep dive into a successful tcp connection does not mean the database is healthy, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Registered Endpoints Are Capacity and Behavior Signals
A sudden drop in registered SIP endpoints can indicate network reachability, credential, expiry or registrar problems even when call processing components are up.
Disk Headroom Is Part of Backup Reliability
A backup process can be perfectly configured and still fail when the destination filesystem no longer has enough capacity for the next archive.
Temporary I/O Is a Query-Behavior Signal
A production-engineering deep dive into temporary i/o is a query-behavior signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Cloudflare and Authelia Logs Explain Edge Failures Differently
A public request can fail before reaching Authelia, inside the authentication flow, or after authentication while the upstream application is unavailable.
Mongo Connections and Query Rate Tell Different Stories
A MongoDB service can accumulate client connections without a matching rise in useful query work, which can point to pooling or application lifecycle problems.
IOPS and Throughput Answer Different Storage Questions
A database doing many tiny synchronous operations and a backup streaming large files can show similar disk utilization with very different access patterns.
Socket Counts Are Capacity Clues, Not Health by Themselves
A rising established-connection count can represent normal load, a leak, slow clients or a downstream dependency holding sockets open.
Dashboards Should Answer Questions, Not Display Metrics
A production-engineering deep dive into dashboards should answer questions, not display metrics, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Docker Filesystem Scanning Was More Expensive Than the Metric Was Worth
A production-engineering deep dive into docker filesystem scanning was more expensive than the metric was worth, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Desired State, Observed State and User-Visible State Are Three Different Things
A production-engineering deep dive into desired state, observed state and user-visible state are three different things, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Metrics Tell Me That Something Happened; Logs Tell Me What Happened
A production-engineering deep dive into metrics tell me that something happened; logs tell me what happened, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Redis Memory Needs Role Context
A Redis instance serving disposable cache and one serving authentication sessions can show the same memory growth with very different operational risk.
The `for` Clause Is an Anti-Flap Tool, Not Decoration
CPU, packet loss and endpoint probes can cross thresholds for a few seconds during harmless transitions or deployment activity.
Monitoring Is a Failure-Modeling Problem, Not a Dashboard Problem
A production-engineering deep dive into monitoring is a failure-modeling problem, not a dashboard problem, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Four Database Engines on One Small Server: Observability Without Exporter Sprawl
A production-engineering deep dive into four database engines on one small server: observability without exporter sprawl, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
MySQL Slow Queries Belong in Infrastructure Monitoring
A production-engineering deep dive into mysql slow queries belong in infrastructure monitoring, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Alert Delivery Failure Needs an Alert of Its Own
A monitoring system that detects failures but cannot notify anyone is partially failed even if every Prometheus target remains green.
Cardinality Is Usually a Schema Problem
A production-engineering deep dive into cardinality is usually a schema problem, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Container Block I/O and I/O Pressure Tell Different Stories
A production-engineering deep dive into container block i/o and i/o pressure tell different stories, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How I Exposed OpenBao Metrics Without Exposing OpenBao
A production-engineering deep dive into how i exposed openbao metrics without exposing openbao, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Metric Relabeling Is Resource Control
A production-engineering deep dive into metric relabeling is resource control, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Backup Size History Can Reveal Missing Data
A backup that suddenly becomes much smaller may have completed successfully while silently omitting a database, artifact directory or other expected state.
Packet Drops Are More Actionable Than Raw Bandwidth
High interface traffic can be completely healthy, while a small but sustained drop rate can damage voice, APIs and tunnel reliability.
Image Storage Growth Is Different from Runtime Data Growth
Docker images can quietly accumulate through repeated deployments even when application volumes and databases remain stable.
Metric Relabeling Can Be a Memory Optimization
Grafana, Loki and Alloy expose many internal metrics that are useful for development but unnecessary on a small production Prometheus.
One Deadlock Is Worth Recording
A PostgreSQL deadlock can resolve automatically by aborting one transaction, leaving the service apparently healthy after the incident.
Monitoring Tax: Why Exporters Need Resource Budgets
A production-engineering deep dive into monitoring tax: why exporters need resource budgets, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How Much Prometheus Is Too Much for an 8 GB Machine?
A production-engineering deep dive into how much prometheus is too much for an 8 gb machine?, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Page Cache Is Not a Memory Leak
A production-engineering deep dive into page cache is not a memory leak, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why 99.9% Uptime Does Not Tell Me When to Wake Someone Up
A production-engineering deep dive into why 99.9% uptime does not tell me when to wake someone up, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Mobile NOC Dashboard Should Show Decisions, Not Every Metric
The phone-sized operations view cannot carry hundreds of panels without turning urgent information into scrolling noise.
Overview vs Drill-Down: One Dashboard Cannot Do Both Well
A production-engineering deep dive into overview vs drill-down: one dashboard cannot do both well, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Heartbeat Freshness Is More Useful Than “Container Running” for Voice Workers
A production-engineering deep dive into heartbeat freshness is more useful than “container running” for voice workers, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Container Network Errors Matter More Than Throughput Alone
RX and TX graphs looked busy enough, but throughput by itself could not tell whether traffic was healthy.
Working Set Is More Useful Than a Single Memory Number
Container memory graphs become noisy when cache and reclaimable pages are treated exactly like unreclaimable application working memory.
Worker Heartbeat Age Catches Silent FreeSWITCH Failure
A FreeSWITCH process may remain alive while its worker integration stops reporting useful state to the signaling layer.
Retention Needs Both Time and Size Limits
Keeping thirty days of Prometheus data is useful until series growth causes the TSDB to consume more disk than the host can safely spare.
Resolved Notifications Are Part of the Incident Lifecycle
A production-engineering deep dive into resolved notifications are part of the incident lifecycle, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Context Switches and Interrupts Explain Some Invisible CPU Cost
CPU percentage alone did not reveal when the kernel was spending more work scheduling tasks or servicing device activity.