docs: add monitoring alerting updates

docs/monitoring.md

@@ -0,0 +1,311 @@
+# Monitoring And Alerting
+
+## Overview
+
+SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.
+
+Current state:
+
+```text
+Production ready
+```
+
+The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.
+
+---
+
+## Stack Components
+
+### Prometheus
+
+Purpose:
+
+* Metrics collection
+* Alert rule evaluation
+* Monitoring data storage
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Grafana
+
+Purpose:
+
+* Monitoring dashboards
+* Prometheus datasource visualization
+* Backup, system and Docker visibility
+
+Dashboards:
+
+* Custom backup dashboard
+* Custom system dashboard
+* Docker monitoring dashboard
+
+Compatibility note:
+
+The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Node Exporter
+
+Purpose:
+
+* Host metrics
+* CPU, memory and disk metrics
+* Textfile Collector for custom backup metrics
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### cAdvisor
+
+Purpose:
+
+* Docker container metrics
+* Container CPU and memory visibility
+* Container health and restart visibility
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Alertmanager
+
+Purpose:
+
+* Alert routing
+* Alert grouping
+* Alert deduplication
+* Email notifications
+* Incident intelligence
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Status:
+
+```text
+Running
+```
+
+---
+
+## Infrastructure
+
+Location:
+
+```text
+/srv/docker/monitoring
+```
+
+Architecture:
+
+* Docker-based stack
+* Persistent Prometheus volume
+* Persistent Grafana volume
+* Dedicated monitoring network
+
+---
+
+## Metrics
+
+### System Metrics
+
+Examples:
+
+* `node_load1`
+* `node_memory_*`
+* `node_filesystem_*`
+* Container metrics through cAdvisor
+
+---
+
+### Backup Metrics
+
+Export method:
+
+```text
+Node Exporter Textfile Collector
+```
+
+Metrics:
+
+* `silverlinux_backup_success`
+* `silverlinux_backup_duration_seconds`
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
+
+---
+
+## Prometheus Scrape Targets
+
+Prometheus successfully scrapes:
+
+* Prometheus
+* Node Exporter
+* cAdvisor
+* SilverLinux backup metrics
+
+Status:
+
+```text
+Validated
+```
+
+---
+
+## Alerting Evolution
+
+### Stage v1
+
+Implemented:
+
+* Basic Prometheus rules
+* Simple email alerts
+
+---
+
+### Stage v2
+
+Implemented:
+
+* Alertmanager
+* Email routing
+* Multi-recipient alerts
+
+---
+
+### Stage v3-v4
+
+Implemented:
+
+* `incident_group` concept
+* Alert correlation by service
+* Severity-based routing
+* Alert deduplication
+* Reduced notification noise
+
+---
+
+## Incident Groups
+
+Alertmanager groups alerts by incident category.
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Outcome:
+
+* Multiple related alerts merge into single incidents.
+* Alerts are root-cause oriented.
+* Alert fatigue is reduced.
+
+---
+
+## Alert Rules
+
+### Backup
+
+* `BackupFailed`
+* `BackupStale`
+
+### System
+
+* `HighCPU`
+* `HighMemoryUsage`
+* `DiskAlmostFull`
+
+### Monitoring
+
+* `PrometheusDown`
+* `GrafanaDown`
+
+### Databases
+
+* `PostgresDown`
+* `MSSQLDown`
+
+### Containers
+
+* `ContainerDown`
+* `TooManyRestarts`
+
+---
+
+## Implementation Fixes
+
+Resolved during implementation:
+
+* Fixed Docker mount errors caused by file versus directory conflicts
+* Standardized alert rule file naming as `alerts.yml`
+* Fixed Prometheus rule loading configuration
+* Eliminated duplicate backup success metric sources
+* Unified backup metric naming to `silverlinux_backup_*`
+
+---
+
+## Data Flow
+
+```text
+Servers
+  -> Node Exporter / cAdvisor
+  -> Prometheus
+  -> Alertmanager
+  -> Email notifications
+  -> Grafana dashboards
+```
+
+---
+
+## Validation
+
+Validated:
+
+* Prometheus UI
+* Prometheus targets
+* Node Exporter
+* cAdvisor
+* Backup metrics
+* Grafana datasource
+* Alertmanager email routing
+* Backup success and failure visibility
+
+---
+
+## Related Documentation
+
+* docs/services.md
+* docs/backups.md
+* docs/security.md
+* docs/network.md
+* docs/email.md
+* docs/decisions.md