Label Freedom Comes With a Time-Series Cost
Metric labels let us divide one measurement in many useful ways, but every new combination of label values can create another time series. Greater detail also carries collection, storage, and query cost.
Metric labels let us divide one measurement in many useful ways, but every new combination of label values can create another time series. Greater detail also carries collection, storage, and query cost.
Label names and cardinality are query contracts for dashboards, recording rules and alerts.
Labels make dashboards flexible, but uncontrolled label values multiply time series and memory cost quickly.
The fastest observability optimization came from identifying one costly collector instead of globally lowering fidelity.
Replacing missing data with zero can make a graph look cleaner while changing its meaning. Zero says a measurement was made and the result was zero. No data says the result is unknown.
A low average response time can make a system look fast even when a smaller group of users waits much longer. Their experience disappears into one number. To understand latency, we also need to understand its distribution.
The edge process can be green while one proxied service is failing behind it.
A counter accumulates events over time, but some counters start over when a process restarts. Subtracting the next value from the previous one can then produce an impossible negative event. The events did not run backward; the measurement history was reset.
Hardware and drivers expose radio state differently, so a production metric may need a fallback without hiding uncertainty.
Logs explain individual failures; metrics show whether the system is drifting before the failures become obvious.