The Monitoring Stack Needs a Memory Budget Too

Observability was becoming one of the larger workloads on a small production server, which is dangerous when monitoring competes with the services it protects. What made the issue measurable was process memory for Prometheus, Grafana, Loki, Alloy, cAdvisor and exporters. Monitoring overhead is part of production capacity and should be measured like any other service rather than treated as free infrastructure.

I classify this as observability resource budgeting. The useful debugging sequence is to confirm the signal, compare it with the neighboring subsystem, then look at logs or detailed metrics only after the failure domain is smaller.

The production rule that came out of it is: Track the stack's aggregate memory, optimize expensive collectors first, and keep enough headroom that an incident does not cause the monitor itself to amplify pressure. This is deliberately more specific than adding another broad alert with no response procedure.

Commit 218300b is the repository evidence behind the note. It provides the concrete configuration or fix that turned the observation into a repeatable monitoring control.