# Monitoring And Alerting ## Overview SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting. Current state: ```text Production ready ``` The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications. --- ## Stack Components ### Prometheus Purpose: * Metrics collection * Alert rule evaluation * Monitoring data storage Status: ```text Running ``` --- ### Grafana Purpose: * Monitoring dashboards * Prometheus datasource visualization * Backup, system and Docker visibility Dashboards: * Custom backup dashboard * Custom system dashboard * Docker monitoring dashboard Compatibility note: The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem. Status: ```text Running ``` --- ### Node Exporter Purpose: * Host metrics * CPU, memory and disk metrics * Textfile Collector for custom backup metrics Status: ```text Running ``` --- ### cAdvisor Purpose: * Docker container metrics * Container CPU and memory visibility * Container health and restart visibility Status: ```text Running ``` --- ### Alertmanager Purpose: * Alert routing * Alert grouping * Alert deduplication * Email notifications * Incident intelligence Recipients: * `oclick021@gmail.com` * `martin@silversolutions.nl` Status: ```text Running ``` --- ## Infrastructure Location: ```text /srv/docker/monitoring ``` Architecture: * Docker-based stack * Persistent Prometheus volume * Persistent Grafana volume * Dedicated monitoring network --- ## Metrics ### System Metrics Examples: * `node_load1` * `node_memory_*` * `node_filesystem_*` * Container metrics through cAdvisor --- ### Backup Metrics Export method: ```text Node Exporter Textfile Collector ``` Metrics: * `silverlinux_backup_success` * `silverlinux_backup_duration_seconds` * `silverlinux_backup_size_bytes` * `silverlinux_backup_timestamp_seconds` --- ## Prometheus Scrape Targets Prometheus successfully scrapes: * Prometheus * Node Exporter * cAdvisor * SilverLinux backup metrics Status: ```text Validated ``` --- ## Alerting Evolution ### Stage v1 Implemented: * Basic Prometheus rules * Simple email alerts --- ### Stage v2 Implemented: * Alertmanager * Email routing * Multi-recipient alerts --- ### Stage v3-v4 Implemented: * `incident_group` concept * Alert correlation by service * Severity-based routing * Alert deduplication * Reduced notification noise --- ## Incident Groups Alertmanager groups alerts by incident category. Incident groups: * `backup_failure` * `system_pressure` * `storage_pressure` * `database_outage` * `monitoring_failure` * `container_health` Outcome: * Multiple related alerts merge into single incidents. * Alerts are root-cause oriented. * Alert fatigue is reduced. --- ## Alert Rules ### Backup * `BackupFailed` * `BackupStale` ### System * `HighCPU` * `HighMemoryUsage` * `DiskAlmostFull` ### Monitoring * `PrometheusDown` * `GrafanaDown` ### Databases * `PostgresDown` * `MSSQLDown` ### Containers * `ContainerDown` * `TooManyRestarts` --- ## Implementation Fixes Resolved during implementation: * Fixed Docker mount errors caused by file versus directory conflicts * Standardized alert rule file naming as `alerts.yml` * Fixed Prometheus rule loading configuration * Eliminated duplicate backup success metric sources * Unified backup metric naming to `silverlinux_backup_*` --- ## Data Flow ```text Servers -> Node Exporter / cAdvisor -> Prometheus -> Alertmanager -> Email notifications -> Grafana dashboards ``` --- ## Validation Validated: * Prometheus UI * Prometheus targets * Node Exporter * cAdvisor * Backup metrics * Grafana datasource * Alertmanager email routing * Backup success and failure visibility --- ## Related Documentation * docs/services.md * docs/backups.md * docs/security.md * docs/network.md * docs/email.md * docs/decisions.md