Infrastructure & Operations / Built / deployed
Production Observability & Alert Validation
Metrics, logs, probes and alerting validated by controlled failures instead of dashboard appearance alone.
OVERVIEW
What I built.
Metrics, logs, probes and alerting validated by controlled failures instead of dashboard appearance alone.
Built host, container, endpoint, database and log observability using Prometheus/Grafana/Alertmanager and related exporters.
Validated alerts by deliberately creating controlled failures and then checking the entire detection path.
Fixed rule/label issues where metrics existed but alerts did not actually evaluate as intended.
SYSTEM ARCHITECTURE
How the system is divided.
Exporters/probes -> Prometheus -> rule evaluation -> Alertmanager -> private alert delivery.
Grafana provides investigation views while Loki/Alloy/OpenObserve provide log context.
Internal process state and external reachability are treated as different signals.
NETWORK / DATA FLOW
What talks to what.
TECHNOLOGY STACK
The technical surface.
IMPLEMENTATION
How it works in practice.
- 01
Exporters/probes -> Prometheus -> rule evaluation -> Alertmanager -> private alert delivery.
- 02
Grafana provides investigation views while Loki/Alloy/OpenObserve provide log context.
- 03
Internal process state and external reachability are treated as different signals.
TESTING & VALIDATION
How I know it works.
Controlled high-CPU and service-failure experiments
PromQL label-matching diagnosis
Alert firing after rule correction
Internal vs public blackbox probes
Monitoring-stack self-observation
Restore/backup and SIP-specific alert checks
EXTERNAL SERVICES & DEPENDENCIES
What sits outside the core.
ENGINEERING EVIDENCE
Artifacts behind the claims.
BOUNDARIES / CONSTRAINTS
What the project does not pretend to solve.
—A green dashboard is not proof of user-visible health
—Metrics collection and alert evaluation are separate failure domains
—Monitoring itself consumes bounded resources
RESULT / CURRENT STATE
What this produced.
Monitoring that has been failure-tested rather than assumed correct because dashboards are green.
DEEP CASE STUDY AVAILABLE