silverlinux-infra/docs/monitoring.md

Monitoring And Alerting

Overview

SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.

Current state:

Production ready

The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.

---

Stack Components

Prometheus

Purpose:

• Metrics collection
• Alert rule evaluation
• Monitoring data storage

Status:

Running

---

Grafana

Purpose:

• Monitoring dashboards
• Prometheus datasource visualization
• Backup, system and Docker visibility

Dashboards:

• Custom backup dashboard
• Custom system dashboard
• Docker monitoring dashboard

Compatibility note:

The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus 3.5.4. This is a third-party dashboard compatibility issue, not an infrastructure problem.

Status:

Running

---

Node Exporter

Purpose:

• Host metrics
• CPU, memory and disk metrics
• Textfile Collector for custom backup metrics

Status:

Running

---

cAdvisor

Purpose:

• Docker container metrics
• Container CPU and memory visibility
• Container health and restart visibility

Status:

Running

---

Alertmanager

Purpose:

• Alert routing
• Alert grouping
• Alert deduplication
• Email notifications
• Incident intelligence

Recipients:

• oclick021@gmail.com
• martin@silversolutions.nl

Status:

Running

---

Infrastructure

Location:

/srv/docker/monitoring

Architecture:

• Docker-based stack
• Persistent Prometheus volume
• Persistent Grafana volume
• Dedicated monitoring network

---

Metrics

System Metrics

Examples:

• node_load1
• node_memory_*
• node_filesystem_*
• Container metrics through cAdvisor

---

Backup Metrics

Export method:

Node Exporter Textfile Collector

Metrics:

• silverlinux_backup_success
• silverlinux_backup_duration_seconds
• silverlinux_backup_size_bytes
• silverlinux_backup_timestamp_seconds

---

Prometheus Scrape Targets

Prometheus successfully scrapes:

• Prometheus
• Node Exporter
• cAdvisor
• SilverLinux backup metrics

Status:

Validated

---

Alerting Evolution

Stage v1

Implemented:

• Basic Prometheus rules
• Simple email alerts

---

Stage v2

Implemented:

• Alertmanager
• Email routing
• Multi-recipient alerts

---

Stage v3-v4

Implemented:

• incident_group concept
• Alert correlation by service
• Severity-based routing
• Alert deduplication
• Reduced notification noise

---

Incident Groups

Alertmanager groups alerts by incident category.

Incident groups:

• backup_failure
• system_pressure
• storage_pressure
• database_outage
• monitoring_failure
• container_health

Outcome:

• Multiple related alerts merge into single incidents.
• Alerts are root-cause oriented.
• Alert fatigue is reduced.

---

Alert Rules

Backup

• BackupFailed
• BackupStale

System

• HighCPU
• HighMemoryUsage
• DiskAlmostFull

Monitoring

• PrometheusDown
• GrafanaDown

Databases

• PostgresDown
• MSSQLDown

Containers

• ContainerDown
• TooManyRestarts

---

Implementation Fixes

Resolved during implementation:

• Fixed Docker mount errors caused by file versus directory conflicts
• Standardized alert rule file naming as alerts.yml
• Fixed Prometheus rule loading configuration
• Eliminated duplicate backup success metric sources
• Unified backup metric naming to silverlinux_backup_*

---

Data Flow

Servers
  -> Node Exporter / cAdvisor
  -> Prometheus
  -> Alertmanager
  -> Email notifications
  -> Grafana dashboards

---

Validation

Validated:

• Prometheus UI
• Prometheus targets
• Node Exporter
• cAdvisor
• Backup metrics
• Grafana datasource
• Alertmanager email routing
• Backup success and failure visibility

---

Related Documentation

• docs/services.md
• docs/backups.md
• docs/security.md
• docs/network.md
• docs/email.md
• docs/decisions.md