docs: add monitoring alerting updates

README.md

@@ -39,6 +39,7 @@ SilverLinux is the primary self-hosted platform for Silver Solutions and hosts s
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 ### Planned
 

SUMMARY.md

@@ -8,7 +8,7 @@ SilverLinux is the primary infrastructure server for Silver Solutions.
 
 The platform hosts source control, project management, video conferencing, container management and supporting infrastructure services.
 
-Monitoring, automated backups and restore validation are now operational.
+Monitoring, alerting, automated backups and restore validation are now production-ready.
 
 ---
 
@@ -148,6 +148,7 @@ Components:
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 Validated:
 
@@ -157,6 +158,7 @@ Validated:
 * cAdvisor
 * Backup metrics
 * Grafana datasource
+* Alertmanager email routing
 
 Dashboards:
 
@@ -164,9 +166,16 @@ Dashboards:
 * Custom system dashboard
 * Docker monitoring dashboard
 
+Alerting:
+
+* Incident grouping
+* Severity-based routing
+* Alert deduplication
+* Multi-recipient email notifications
+
 Status:
 
-* Operational
+* Production ready
 
 ---
 
@@ -183,7 +192,7 @@ Purpose:
 
 Current version:
 
-* Backup v2
+* Backup v5.3
 
 Validated:
 
@@ -193,10 +202,14 @@ Validated:
 * PostgreSQL backup
 * Email notifications
 * Cron automation
+* Backup metrics export
+* Lock file protection
+* Disk space validation
+* Structured failure handling
 
 Status:
 
-* Operational
+* Production ready
 
 ---
 
@@ -360,7 +373,9 @@ Completed:
 * Grafana + Prometheus
 * Node Exporter
 * cAdvisor
+* Alertmanager
 * Backup metrics integration
+* Incident intelligence
 
 ### Phase 2 - Security And Stability
 
@@ -371,7 +386,7 @@ Completed:
 
 Completed:
 
-* Backup v2 automation
+* Backup v5.3 automation
 * Restore validation
 
 ### Phase 3 - Scale And Enterprise Services

docs/AI_CONTEXT.md

@@ -33,6 +33,7 @@ Public IP: 51.255.83.140
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 ## Domains
 
@@ -115,6 +116,7 @@ Components:
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 Validated:
 
@@ -124,12 +126,22 @@ Validated:
 * cAdvisor
 * Backup metrics
 * Grafana datasource
+* Alertmanager email routing
 
 Grafana has custom backup, system and Docker monitoring dashboards.
 
+Alertmanager implements incident intelligence with incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
 ## Backup
 
-Backup v2 is operational.
+Backup v5.3 is production.
 
 Implemented:
 
@@ -140,6 +152,10 @@ Implemented:
 * Automatic cleanup
 * Prometheus metrics export
 * Restore validation
+* Lock file protection
+* Disk space validation
+* Central logging
+* Structured failure handling
 
 Verified:
 
@@ -160,9 +176,9 @@ and the README.md of each service folder before making infrastructure recommenda
 
 Roadmap:
 
-* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, backup metrics integration
+* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, Alertmanager, backup metrics integration, incident intelligence
 * Phase 1 remaining: Loki, Redis, Authelia
-* Phase 2 completed: Backup v2 automation, restore validation
+* Phase 2 completed: Backup v5.3 automation, restore validation
 * Phase 2 remaining: CrowdSec, Sentry, Fail2Ban upgrade, Cloudflare rate limiting and WAF
 * Phase 3: SonarQube, Harbor, Backstage, Nextcloud
 

docs/backups.md

@@ -9,7 +9,7 @@ The goal is to ensure that all critical services can be restored after hardware
 Current state:
 
 ```text
-Backup v2 operational
+Backup v5.3 production
 ```
 
 ---
@@ -60,6 +60,34 @@ Operational
 
 ---
 
+### Backup v5.3
+
+Implemented:
+
+* PostgreSQL backup using `pg_dumpall`
+* MSSQL backup using `sqlcmd` and `docker cp`
+* Gitea backup
+* BaGet backup
+* OpenProject backup
+* Docker configuration backup
+* Final compressed archive
+* Lock file protection
+* Disk space validation
+* Central backup logging
+* Structured failure handling
+* Prometheus Textfile Collector metrics output
+* Backup success and failure tracking
+* Duration and size monitoring
+* Historical logging
+
+Status:
+
+```text
+Production
+```
+
+---
+
 ## Backup Priorities
 
 ### Critical
@@ -351,7 +379,7 @@ Suggested Structure:
 
 Current cleanup behavior:
 
-* Backup v2 monitors available disk space.
+* Backup v5.3 monitors available disk space.
 * If disk space is low, the oldest backups are deleted automatically.
 * Cleanup is logged.
 
@@ -431,13 +459,21 @@ Validation results:
 
 ## Monitoring Integration
 
-Backup v2 exports metrics for Prometheus.
+Backup v5.3 exports metrics for Prometheus.
 
 Current metrics:
 
 * Backup success
 * Backup duration
 * Backup size
+* Backup timestamp
+
+Metric names:
+
+* `silverlinux_backup_success`
+* `silverlinux_backup_duration_seconds`
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
 
 Export path:
 
@@ -546,6 +582,7 @@ Verify:
 * docs/server.md
 * docs/security.md
 * docs/secrets.md
+* docs/monitoring.md
 * postgres/README.md
 * gitea/README.md
 * openproject/README.md

docs/decisions.md

@@ -459,6 +459,7 @@ Deploy a dedicated monitoring Docker stack using:
 * Grafana for visualization
 * Node Exporter for host metrics
 * cAdvisor for Docker metrics
+* Alertmanager for alert routing and incident notifications
 
 Reason:
 
@@ -466,6 +467,7 @@ Reason:
 * Grafana provides flexible dashboards and datasource integration.
 * Node Exporter provides host CPU, memory and disk visibility.
 * cAdvisor provides Docker container visibility.
+* Alertmanager provides grouping, deduplication and multi-recipient email routing.
 
 Status:
 
@@ -479,6 +481,7 @@ Validation:
 * Prometheus targets validated
 * Grafana datasource connected to Prometheus
 * Initial monitoring dashboards created and validated
+* Alertmanager email routing validated
 
 ---
 
@@ -490,9 +493,10 @@ Export SilverLinux backup metrics through the Node Exporter Textfile Collector.
 
 Metrics:
 
-* Backup success
+* `silverlinux_backup_success`
-* Backup duration
+* `silverlinux_backup_duration_seconds`
-* Backup size
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
 
 Reason:
 
@@ -508,11 +512,11 @@ Implemented
 
 ---
 
-## 2026-06-25 - Validate Backup v2 And Restore Workflow
+## 2026-06-25 - Validate Backup v5.3 And Restore Workflow
 
 Decision:
 
-Treat Backup v2 as the operational SilverLinux backup system.
+Treat Backup v5.3 as the operational SilverLinux backup system.
 
 Implemented:
 
@@ -525,6 +529,10 @@ Implemented:
 * Backup metrics
 * Automatic low disk space cleanup
 * Daily cron execution
+* Lock file protection
+* Disk space validation
+* Central logging
+* Structured failure handling
 
 Validation:
 
@@ -545,6 +553,36 @@ Implemented
 
 ---
 
+## 2026-06-25 - Implement Incident Intelligence Alerting
+
+Decision:
+
+Use Alertmanager with incident grouping, severity-based routing and alert deduplication for SilverLinux monitoring.
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Reason:
+
+* Merge related alerts into single incidents.
+* Reduce notification noise.
+* Route alerts by severity.
+* Make alerts root-cause oriented.
+
+Status:
+
+```text
+Implemented
+```
+
+---
+
 ## Future Decisions
 
 Document future decisions using the following template:

docs/email.md

@@ -147,6 +147,25 @@ Configured and validated
 
 ---
 
+### Alertmanager
+
+Purpose:
+
+* Monitoring alert emails
+* Incident grouped notifications
+* Severity-based alert routing
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Status:
+
+Configured and validated
+
+---
+
 ## Future Applications
 
 Future services should use the centralized SMTP configuration whenever possible.
@@ -231,5 +250,6 @@ Verify:
 * docs/secrets.md
 * docs/security.md
 * docs/backups.md
+* docs/monitoring.md
 * openproject/README.md
 * gitea/README.md

docs/monitoring.md

@@ -0,0 +1,311 @@
+# Monitoring And Alerting
+
+## Overview
+
+SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.
+
+Current state:
+
+```text
+Production ready
+```
+
+The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.
+
+---
+
+## Stack Components
+
+### Prometheus
+
+Purpose:
+
+* Metrics collection
+* Alert rule evaluation
+* Monitoring data storage
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Grafana
+
+Purpose:
+
+* Monitoring dashboards
+* Prometheus datasource visualization
+* Backup, system and Docker visibility
+
+Dashboards:
+
+* Custom backup dashboard
+* Custom system dashboard
+* Docker monitoring dashboard
+
+Compatibility note:
+
+The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Node Exporter
+
+Purpose:
+
+* Host metrics
+* CPU, memory and disk metrics
+* Textfile Collector for custom backup metrics
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### cAdvisor
+
+Purpose:
+
+* Docker container metrics
+* Container CPU and memory visibility
+* Container health and restart visibility
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Alertmanager
+
+Purpose:
+
+* Alert routing
+* Alert grouping
+* Alert deduplication
+* Email notifications
+* Incident intelligence
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Status:
+
+```text
+Running
+```
+
+---
+
+## Infrastructure
+
+Location:
+
+```text
+/srv/docker/monitoring
+```
+
+Architecture:
+
+* Docker-based stack
+* Persistent Prometheus volume
+* Persistent Grafana volume
+* Dedicated monitoring network
+
+---
+
+## Metrics
+
+### System Metrics
+
+Examples:
+
+* `node_load1`
+* `node_memory_*`
+* `node_filesystem_*`
+* Container metrics through cAdvisor
+
+---
+
+### Backup Metrics
+
+Export method:
+
+```text
+Node Exporter Textfile Collector
+```
+
+Metrics:
+
+* `silverlinux_backup_success`
+* `silverlinux_backup_duration_seconds`
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
+
+---
+
+## Prometheus Scrape Targets
+
+Prometheus successfully scrapes:
+
+* Prometheus
+* Node Exporter
+* cAdvisor
+* SilverLinux backup metrics
+
+Status:
+
+```text
+Validated
+```
+
+---
+
+## Alerting Evolution
+
+### Stage v1
+
+Implemented:
+
+* Basic Prometheus rules
+* Simple email alerts
+
+---
+
+### Stage v2
+
+Implemented:
+
+* Alertmanager
+* Email routing
+* Multi-recipient alerts
+
+---
+
+### Stage v3-v4
+
+Implemented:
+
+* `incident_group` concept
+* Alert correlation by service
+* Severity-based routing
+* Alert deduplication
+* Reduced notification noise
+
+---
+
+## Incident Groups
+
+Alertmanager groups alerts by incident category.
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Outcome:
+
+* Multiple related alerts merge into single incidents.
+* Alerts are root-cause oriented.
+* Alert fatigue is reduced.
+
+---
+
+## Alert Rules
+
+### Backup
+
+* `BackupFailed`
+* `BackupStale`
+
+### System
+
+* `HighCPU`
+* `HighMemoryUsage`
+* `DiskAlmostFull`
+
+### Monitoring
+
+* `PrometheusDown`
+* `GrafanaDown`
+
+### Databases
+
+* `PostgresDown`
+* `MSSQLDown`
+
+### Containers
+
+* `ContainerDown`
+* `TooManyRestarts`
+
+---
+
+## Implementation Fixes
+
+Resolved during implementation:
+
+* Fixed Docker mount errors caused by file versus directory conflicts
+* Standardized alert rule file naming as `alerts.yml`
+* Fixed Prometheus rule loading configuration
+* Eliminated duplicate backup success metric sources
+* Unified backup metric naming to `silverlinux_backup_*`
+
+---
+
+## Data Flow
+
+```text
+Servers
+  -> Node Exporter / cAdvisor
+  -> Prometheus
+  -> Alertmanager
+  -> Email notifications
+  -> Grafana dashboards
+```
+
+---
+
+## Validation
+
+Validated:
+
+* Prometheus UI
+* Prometheus targets
+* Node Exporter
+* cAdvisor
+* Backup metrics
+* Grafana datasource
+* Alertmanager email routing
+* Backup success and failure visibility
+
+---
+
+## Related Documentation
+
+* docs/services.md
+* docs/backups.md
+* docs/security.md
+* docs/network.md
+* docs/email.md
+* docs/decisions.md

docs/network.md

@@ -80,6 +80,7 @@ Internal monitoring services:
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 Verified subnet:
 
@@ -188,6 +189,7 @@ Services:
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 Validated:
 
@@ -196,6 +198,7 @@ Validated:
 * Prometheus scrapes cAdvisor
 * Prometheus scrapes SilverLinux backup metrics
 * Grafana connects to Prometheus
+* Alertmanager routes monitoring alerts
 
 ---
 
@@ -292,5 +295,6 @@ Periodically verify:
 
 * docs/services.md
 * docs/security.md
+* docs/monitoring.md
 * docs/decisions.md
 * nginx-proxy-manager/README.md

docs/roadmap.md

@@ -51,6 +51,7 @@ Completed components:
 * Grafana
 * Node Exporter
 * cAdvisor
+* Alertmanager
 
 Validation:
 
@@ -58,6 +59,14 @@ Validation:
 * Prometheus targets validated
 * Grafana datasource connected to Prometheus
 * Initial dashboards created and validated
+* Alertmanager email routing validated
+
+Alerting:
+
+* Incident grouping implemented
+* Severity-based routing implemented
+* Alert deduplication implemented
+* Multi-recipient email alerts implemented
 
 ---
 
@@ -257,7 +266,7 @@ Planned
 Current backup status:
 
 ```text
-Backup v2 operational
+Backup v5.3 production
 ```
 
 Completed:
@@ -269,6 +278,10 @@ Completed:
 * Automatic cleanup
 * Backup metrics export
 * Restore validation
+* Lock file protection
+* Disk space validation
+* Central logging
+* Structured failure handling
 
 ---
 
@@ -288,7 +301,9 @@ Completed:
 * Prometheus
 * Node Exporter
 * cAdvisor
+* Alertmanager
 * Backup metrics integration
+* Incident intelligence
 
 ---
 
@@ -303,7 +318,7 @@ Items:
 
 Completed:
 
-* Backup v2 automation
+* Backup v5.3 automation
 * Restore validation
 
 ---
@@ -340,4 +355,5 @@ Each roadmap item should receive documentation before deployment:
 * docs/security.md
 * docs/network.md
 * docs/backups.md
+* docs/monitoring.md
 * docs/decisions.md

docs/services.md

@@ -180,6 +180,7 @@ Running
 Purpose:
 
 * Metrics collection
+* Alert rule engine
 * Monitoring data storage
 * Scrape targets for system, Docker and backup metrics
 
@@ -190,6 +191,14 @@ Scrapes:
 * cAdvisor
 * SilverLinux backup metrics
 
+Alert categories:
+
+* Backup
+* System pressure
+* Monitoring health
+* Database availability
+* Container health
+
 Status:
 
 ```text
@@ -224,6 +233,38 @@ Running
 
 ---
 
+### Alertmanager
+
+Purpose:
+
+* Alert routing
+* Email notifications
+* Incident grouping
+* Alert deduplication
+* Severity-based routing
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Status:
+
+```text
+Running
+```
+
+---
+
 ### Node Exporter
 
 Purpose:
@@ -552,6 +593,7 @@ Cleanup:
 * docs/server.md
 * docs/network.md
 * docs/roadmap.md
+* docs/monitoring.md
 * docs/domains.md
 * docs/security.md
 * docs/backups.md