From a74909be255412091dde1ee72b0f6c4f12096f56 Mon Sep 17 00:00:00 2001 From: Saeed Asgari Date: Thu, 25 Jun 2026 14:11:23 +0330 Subject: [PATCH] docs: add monitoring alerting updates --- README.md | 1 + SUMMARY.md | 25 +++- docs/AI_CONTEXT.md | 22 +++- docs/backups.md | 43 ++++++- docs/decisions.md | 48 ++++++- docs/email.md | 20 +++ docs/monitoring.md | 311 +++++++++++++++++++++++++++++++++++++++++++++ docs/network.md | 4 + docs/roadmap.md | 20 ++- docs/services.md | 42 ++++++ 10 files changed, 518 insertions(+), 18 deletions(-) create mode 100644 docs/monitoring.md diff --git a/README.md b/README.md index cda3bd7..98e0a20 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,7 @@ SilverLinux is the primary self-hosted platform for Silver Solutions and hosts s * Grafana * Node Exporter * cAdvisor +* Alertmanager ### Planned diff --git a/SUMMARY.md b/SUMMARY.md index 10f2772..54835d6 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -8,7 +8,7 @@ SilverLinux is the primary infrastructure server for Silver Solutions. The platform hosts source control, project management, video conferencing, container management and supporting infrastructure services. -Monitoring, automated backups and restore validation are now operational. +Monitoring, alerting, automated backups and restore validation are now production-ready. --- @@ -148,6 +148,7 @@ Components: * Grafana * Node Exporter * cAdvisor +* Alertmanager Validated: @@ -157,6 +158,7 @@ Validated: * cAdvisor * Backup metrics * Grafana datasource +* Alertmanager email routing Dashboards: @@ -164,9 +166,16 @@ Dashboards: * Custom system dashboard * Docker monitoring dashboard +Alerting: + +* Incident grouping +* Severity-based routing +* Alert deduplication +* Multi-recipient email notifications + Status: -* Operational +* Production ready --- @@ -183,7 +192,7 @@ Purpose: Current version: -* Backup v2 +* Backup v5.3 Validated: @@ -193,10 +202,14 @@ Validated: * PostgreSQL backup * Email notifications * Cron automation +* Backup metrics export +* Lock file protection +* Disk space validation +* Structured failure handling Status: -* Operational +* Production ready --- @@ -360,7 +373,9 @@ Completed: * Grafana + Prometheus * Node Exporter * cAdvisor +* Alertmanager * Backup metrics integration +* Incident intelligence ### Phase 2 - Security And Stability @@ -371,7 +386,7 @@ Completed: Completed: -* Backup v2 automation +* Backup v5.3 automation * Restore validation ### Phase 3 - Scale And Enterprise Services diff --git a/docs/AI_CONTEXT.md b/docs/AI_CONTEXT.md index 6af7861..b7be917 100644 --- a/docs/AI_CONTEXT.md +++ b/docs/AI_CONTEXT.md @@ -33,6 +33,7 @@ Public IP: 51.255.83.140 * Grafana * Node Exporter * cAdvisor +* Alertmanager ## Domains @@ -115,6 +116,7 @@ Components: * Grafana * Node Exporter * cAdvisor +* Alertmanager Validated: @@ -124,12 +126,22 @@ Validated: * cAdvisor * Backup metrics * Grafana datasource +* Alertmanager email routing Grafana has custom backup, system and Docker monitoring dashboards. +Alertmanager implements incident intelligence with incident groups: + +* `backup_failure` +* `system_pressure` +* `storage_pressure` +* `database_outage` +* `monitoring_failure` +* `container_health` + ## Backup -Backup v2 is operational. +Backup v5.3 is production. Implemented: @@ -140,6 +152,10 @@ Implemented: * Automatic cleanup * Prometheus metrics export * Restore validation +* Lock file protection +* Disk space validation +* Central logging +* Structured failure handling Verified: @@ -160,9 +176,9 @@ and the README.md of each service folder before making infrastructure recommenda Roadmap: -* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, backup metrics integration +* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, Alertmanager, backup metrics integration, incident intelligence * Phase 1 remaining: Loki, Redis, Authelia -* Phase 2 completed: Backup v2 automation, restore validation +* Phase 2 completed: Backup v5.3 automation, restore validation * Phase 2 remaining: CrowdSec, Sentry, Fail2Ban upgrade, Cloudflare rate limiting and WAF * Phase 3: SonarQube, Harbor, Backstage, Nextcloud diff --git a/docs/backups.md b/docs/backups.md index 224774b..8de6762 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -9,7 +9,7 @@ The goal is to ensure that all critical services can be restored after hardware Current state: ```text -Backup v2 operational +Backup v5.3 production ``` --- @@ -60,6 +60,34 @@ Operational --- +### Backup v5.3 + +Implemented: + +* PostgreSQL backup using `pg_dumpall` +* MSSQL backup using `sqlcmd` and `docker cp` +* Gitea backup +* BaGet backup +* OpenProject backup +* Docker configuration backup +* Final compressed archive +* Lock file protection +* Disk space validation +* Central backup logging +* Structured failure handling +* Prometheus Textfile Collector metrics output +* Backup success and failure tracking +* Duration and size monitoring +* Historical logging + +Status: + +```text +Production +``` + +--- + ## Backup Priorities ### Critical @@ -351,7 +379,7 @@ Suggested Structure: Current cleanup behavior: -* Backup v2 monitors available disk space. +* Backup v5.3 monitors available disk space. * If disk space is low, the oldest backups are deleted automatically. * Cleanup is logged. @@ -431,13 +459,21 @@ Validation results: ## Monitoring Integration -Backup v2 exports metrics for Prometheus. +Backup v5.3 exports metrics for Prometheus. Current metrics: * Backup success * Backup duration * Backup size +* Backup timestamp + +Metric names: + +* `silverlinux_backup_success` +* `silverlinux_backup_duration_seconds` +* `silverlinux_backup_size_bytes` +* `silverlinux_backup_timestamp_seconds` Export path: @@ -546,6 +582,7 @@ Verify: * docs/server.md * docs/security.md * docs/secrets.md +* docs/monitoring.md * postgres/README.md * gitea/README.md * openproject/README.md diff --git a/docs/decisions.md b/docs/decisions.md index 2eb7c5b..27bea8f 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -459,6 +459,7 @@ Deploy a dedicated monitoring Docker stack using: * Grafana for visualization * Node Exporter for host metrics * cAdvisor for Docker metrics +* Alertmanager for alert routing and incident notifications Reason: @@ -466,6 +467,7 @@ Reason: * Grafana provides flexible dashboards and datasource integration. * Node Exporter provides host CPU, memory and disk visibility. * cAdvisor provides Docker container visibility. +* Alertmanager provides grouping, deduplication and multi-recipient email routing. Status: @@ -479,6 +481,7 @@ Validation: * Prometheus targets validated * Grafana datasource connected to Prometheus * Initial monitoring dashboards created and validated +* Alertmanager email routing validated --- @@ -490,9 +493,10 @@ Export SilverLinux backup metrics through the Node Exporter Textfile Collector. Metrics: -* Backup success -* Backup duration -* Backup size +* `silverlinux_backup_success` +* `silverlinux_backup_duration_seconds` +* `silverlinux_backup_size_bytes` +* `silverlinux_backup_timestamp_seconds` Reason: @@ -508,11 +512,11 @@ Implemented --- -## 2026-06-25 - Validate Backup v2 And Restore Workflow +## 2026-06-25 - Validate Backup v5.3 And Restore Workflow Decision: -Treat Backup v2 as the operational SilverLinux backup system. +Treat Backup v5.3 as the operational SilverLinux backup system. Implemented: @@ -525,6 +529,10 @@ Implemented: * Backup metrics * Automatic low disk space cleanup * Daily cron execution +* Lock file protection +* Disk space validation +* Central logging +* Structured failure handling Validation: @@ -545,6 +553,36 @@ Implemented --- +## 2026-06-25 - Implement Incident Intelligence Alerting + +Decision: + +Use Alertmanager with incident grouping, severity-based routing and alert deduplication for SilverLinux monitoring. + +Incident groups: + +* `backup_failure` +* `system_pressure` +* `storage_pressure` +* `database_outage` +* `monitoring_failure` +* `container_health` + +Reason: + +* Merge related alerts into single incidents. +* Reduce notification noise. +* Route alerts by severity. +* Make alerts root-cause oriented. + +Status: + +```text +Implemented +``` + +--- + ## Future Decisions Document future decisions using the following template: diff --git a/docs/email.md b/docs/email.md index 2b82422..7978c76 100644 --- a/docs/email.md +++ b/docs/email.md @@ -147,6 +147,25 @@ Configured and validated --- +### Alertmanager + +Purpose: + +* Monitoring alert emails +* Incident grouped notifications +* Severity-based alert routing + +Recipients: + +* `oclick021@gmail.com` +* `martin@silversolutions.nl` + +Status: + +Configured and validated + +--- + ## Future Applications Future services should use the centralized SMTP configuration whenever possible. @@ -231,5 +250,6 @@ Verify: * docs/secrets.md * docs/security.md * docs/backups.md +* docs/monitoring.md * openproject/README.md * gitea/README.md diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000..a25eb86 --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,311 @@ +# Monitoring And Alerting + +## Overview + +SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting. + +Current state: + +```text +Production ready +``` + +The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications. + +--- + +## Stack Components + +### Prometheus + +Purpose: + +* Metrics collection +* Alert rule evaluation +* Monitoring data storage + +Status: + +```text +Running +``` + +--- + +### Grafana + +Purpose: + +* Monitoring dashboards +* Prometheus datasource visualization +* Backup, system and Docker visibility + +Dashboards: + +* Custom backup dashboard +* Custom system dashboard +* Docker monitoring dashboard + +Compatibility note: + +The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem. + +Status: + +```text +Running +``` + +--- + +### Node Exporter + +Purpose: + +* Host metrics +* CPU, memory and disk metrics +* Textfile Collector for custom backup metrics + +Status: + +```text +Running +``` + +--- + +### cAdvisor + +Purpose: + +* Docker container metrics +* Container CPU and memory visibility +* Container health and restart visibility + +Status: + +```text +Running +``` + +--- + +### Alertmanager + +Purpose: + +* Alert routing +* Alert grouping +* Alert deduplication +* Email notifications +* Incident intelligence + +Recipients: + +* `oclick021@gmail.com` +* `martin@silversolutions.nl` + +Status: + +```text +Running +``` + +--- + +## Infrastructure + +Location: + +```text +/srv/docker/monitoring +``` + +Architecture: + +* Docker-based stack +* Persistent Prometheus volume +* Persistent Grafana volume +* Dedicated monitoring network + +--- + +## Metrics + +### System Metrics + +Examples: + +* `node_load1` +* `node_memory_*` +* `node_filesystem_*` +* Container metrics through cAdvisor + +--- + +### Backup Metrics + +Export method: + +```text +Node Exporter Textfile Collector +``` + +Metrics: + +* `silverlinux_backup_success` +* `silverlinux_backup_duration_seconds` +* `silverlinux_backup_size_bytes` +* `silverlinux_backup_timestamp_seconds` + +--- + +## Prometheus Scrape Targets + +Prometheus successfully scrapes: + +* Prometheus +* Node Exporter +* cAdvisor +* SilverLinux backup metrics + +Status: + +```text +Validated +``` + +--- + +## Alerting Evolution + +### Stage v1 + +Implemented: + +* Basic Prometheus rules +* Simple email alerts + +--- + +### Stage v2 + +Implemented: + +* Alertmanager +* Email routing +* Multi-recipient alerts + +--- + +### Stage v3-v4 + +Implemented: + +* `incident_group` concept +* Alert correlation by service +* Severity-based routing +* Alert deduplication +* Reduced notification noise + +--- + +## Incident Groups + +Alertmanager groups alerts by incident category. + +Incident groups: + +* `backup_failure` +* `system_pressure` +* `storage_pressure` +* `database_outage` +* `monitoring_failure` +* `container_health` + +Outcome: + +* Multiple related alerts merge into single incidents. +* Alerts are root-cause oriented. +* Alert fatigue is reduced. + +--- + +## Alert Rules + +### Backup + +* `BackupFailed` +* `BackupStale` + +### System + +* `HighCPU` +* `HighMemoryUsage` +* `DiskAlmostFull` + +### Monitoring + +* `PrometheusDown` +* `GrafanaDown` + +### Databases + +* `PostgresDown` +* `MSSQLDown` + +### Containers + +* `ContainerDown` +* `TooManyRestarts` + +--- + +## Implementation Fixes + +Resolved during implementation: + +* Fixed Docker mount errors caused by file versus directory conflicts +* Standardized alert rule file naming as `alerts.yml` +* Fixed Prometheus rule loading configuration +* Eliminated duplicate backup success metric sources +* Unified backup metric naming to `silverlinux_backup_*` + +--- + +## Data Flow + +```text +Servers + -> Node Exporter / cAdvisor + -> Prometheus + -> Alertmanager + -> Email notifications + -> Grafana dashboards +``` + +--- + +## Validation + +Validated: + +* Prometheus UI +* Prometheus targets +* Node Exporter +* cAdvisor +* Backup metrics +* Grafana datasource +* Alertmanager email routing +* Backup success and failure visibility + +--- + +## Related Documentation + +* docs/services.md +* docs/backups.md +* docs/security.md +* docs/network.md +* docs/email.md +* docs/decisions.md diff --git a/docs/network.md b/docs/network.md index 3daab1e..e21d661 100644 --- a/docs/network.md +++ b/docs/network.md @@ -80,6 +80,7 @@ Internal monitoring services: * Grafana * Node Exporter * cAdvisor +* Alertmanager Verified subnet: @@ -188,6 +189,7 @@ Services: * Grafana * Node Exporter * cAdvisor +* Alertmanager Validated: @@ -196,6 +198,7 @@ Validated: * Prometheus scrapes cAdvisor * Prometheus scrapes SilverLinux backup metrics * Grafana connects to Prometheus +* Alertmanager routes monitoring alerts --- @@ -292,5 +295,6 @@ Periodically verify: * docs/services.md * docs/security.md +* docs/monitoring.md * docs/decisions.md * nginx-proxy-manager/README.md diff --git a/docs/roadmap.md b/docs/roadmap.md index 298e8a4..0b87dee 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -51,6 +51,7 @@ Completed components: * Grafana * Node Exporter * cAdvisor +* Alertmanager Validation: @@ -58,6 +59,14 @@ Validation: * Prometheus targets validated * Grafana datasource connected to Prometheus * Initial dashboards created and validated +* Alertmanager email routing validated + +Alerting: + +* Incident grouping implemented +* Severity-based routing implemented +* Alert deduplication implemented +* Multi-recipient email alerts implemented --- @@ -257,7 +266,7 @@ Planned Current backup status: ```text -Backup v2 operational +Backup v5.3 production ``` Completed: @@ -269,6 +278,10 @@ Completed: * Automatic cleanup * Backup metrics export * Restore validation +* Lock file protection +* Disk space validation +* Central logging +* Structured failure handling --- @@ -288,7 +301,9 @@ Completed: * Prometheus * Node Exporter * cAdvisor +* Alertmanager * Backup metrics integration +* Incident intelligence --- @@ -303,7 +318,7 @@ Items: Completed: -* Backup v2 automation +* Backup v5.3 automation * Restore validation --- @@ -340,4 +355,5 @@ Each roadmap item should receive documentation before deployment: * docs/security.md * docs/network.md * docs/backups.md +* docs/monitoring.md * docs/decisions.md diff --git a/docs/services.md b/docs/services.md index 4d9db61..beaab76 100644 --- a/docs/services.md +++ b/docs/services.md @@ -180,6 +180,7 @@ Running Purpose: * Metrics collection +* Alert rule engine * Monitoring data storage * Scrape targets for system, Docker and backup metrics @@ -190,6 +191,14 @@ Scrapes: * cAdvisor * SilverLinux backup metrics +Alert categories: + +* Backup +* System pressure +* Monitoring health +* Database availability +* Container health + Status: ```text @@ -224,6 +233,38 @@ Running --- +### Alertmanager + +Purpose: + +* Alert routing +* Email notifications +* Incident grouping +* Alert deduplication +* Severity-based routing + +Recipients: + +* `oclick021@gmail.com` +* `martin@silversolutions.nl` + +Incident groups: + +* `backup_failure` +* `system_pressure` +* `storage_pressure` +* `database_outage` +* `monitoring_failure` +* `container_health` + +Status: + +```text +Running +``` + +--- + ### Node Exporter Purpose: @@ -552,6 +593,7 @@ Cleanup: * docs/server.md * docs/network.md * docs/roadmap.md +* docs/monitoring.md * docs/domains.md * docs/security.md * docs/backups.md