docs: add monitoring alerting updates

This commit is contained in:
2026-06-25 14:11:23 +03:30
parent 4cf06da0a8
commit a74909be25
10 changed files with 518 additions and 18 deletions
+1
View File
@@ -39,6 +39,7 @@ SilverLinux is the primary self-hosted platform for Silver Solutions and hosts s
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
### Planned
+20 -5
View File
@@ -8,7 +8,7 @@ SilverLinux is the primary infrastructure server for Silver Solutions.
The platform hosts source control, project management, video conferencing, container management and supporting infrastructure services.
Monitoring, automated backups and restore validation are now operational.
Monitoring, alerting, automated backups and restore validation are now production-ready.
---
@@ -148,6 +148,7 @@ Components:
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
Validated:
@@ -157,6 +158,7 @@ Validated:
* cAdvisor
* Backup metrics
* Grafana datasource
* Alertmanager email routing
Dashboards:
@@ -164,9 +166,16 @@ Dashboards:
* Custom system dashboard
* Docker monitoring dashboard
Alerting:
* Incident grouping
* Severity-based routing
* Alert deduplication
* Multi-recipient email notifications
Status:
* Operational
* Production ready
---
@@ -183,7 +192,7 @@ Purpose:
Current version:
* Backup v2
* Backup v5.3
Validated:
@@ -193,10 +202,14 @@ Validated:
* PostgreSQL backup
* Email notifications
* Cron automation
* Backup metrics export
* Lock file protection
* Disk space validation
* Structured failure handling
Status:
* Operational
* Production ready
---
@@ -360,7 +373,9 @@ Completed:
* Grafana + Prometheus
* Node Exporter
* cAdvisor
* Alertmanager
* Backup metrics integration
* Incident intelligence
### Phase 2 - Security And Stability
@@ -371,7 +386,7 @@ Completed:
Completed:
* Backup v2 automation
* Backup v5.3 automation
* Restore validation
### Phase 3 - Scale And Enterprise Services
+19 -3
View File
@@ -33,6 +33,7 @@ Public IP: 51.255.83.140
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
## Domains
@@ -115,6 +116,7 @@ Components:
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
Validated:
@@ -124,12 +126,22 @@ Validated:
* cAdvisor
* Backup metrics
* Grafana datasource
* Alertmanager email routing
Grafana has custom backup, system and Docker monitoring dashboards.
Alertmanager implements incident intelligence with incident groups:
* `backup_failure`
* `system_pressure`
* `storage_pressure`
* `database_outage`
* `monitoring_failure`
* `container_health`
## Backup
Backup v2 is operational.
Backup v5.3 is production.
Implemented:
@@ -140,6 +152,10 @@ Implemented:
* Automatic cleanup
* Prometheus metrics export
* Restore validation
* Lock file protection
* Disk space validation
* Central logging
* Structured failure handling
Verified:
@@ -160,9 +176,9 @@ and the README.md of each service folder before making infrastructure recommenda
Roadmap:
* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, backup metrics integration
* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, Alertmanager, backup metrics integration, incident intelligence
* Phase 1 remaining: Loki, Redis, Authelia
* Phase 2 completed: Backup v2 automation, restore validation
* Phase 2 completed: Backup v5.3 automation, restore validation
* Phase 2 remaining: CrowdSec, Sentry, Fail2Ban upgrade, Cloudflare rate limiting and WAF
* Phase 3: SonarQube, Harbor, Backstage, Nextcloud
+40 -3
View File
@@ -9,7 +9,7 @@ The goal is to ensure that all critical services can be restored after hardware
Current state:
```text
Backup v2 operational
Backup v5.3 production
```
---
@@ -60,6 +60,34 @@ Operational
---
### Backup v5.3
Implemented:
* PostgreSQL backup using `pg_dumpall`
* MSSQL backup using `sqlcmd` and `docker cp`
* Gitea backup
* BaGet backup
* OpenProject backup
* Docker configuration backup
* Final compressed archive
* Lock file protection
* Disk space validation
* Central backup logging
* Structured failure handling
* Prometheus Textfile Collector metrics output
* Backup success and failure tracking
* Duration and size monitoring
* Historical logging
Status:
```text
Production
```
---
## Backup Priorities
### Critical
@@ -351,7 +379,7 @@ Suggested Structure:
Current cleanup behavior:
* Backup v2 monitors available disk space.
* Backup v5.3 monitors available disk space.
* If disk space is low, the oldest backups are deleted automatically.
* Cleanup is logged.
@@ -431,13 +459,21 @@ Validation results:
## Monitoring Integration
Backup v2 exports metrics for Prometheus.
Backup v5.3 exports metrics for Prometheus.
Current metrics:
* Backup success
* Backup duration
* Backup size
* Backup timestamp
Metric names:
* `silverlinux_backup_success`
* `silverlinux_backup_duration_seconds`
* `silverlinux_backup_size_bytes`
* `silverlinux_backup_timestamp_seconds`
Export path:
@@ -546,6 +582,7 @@ Verify:
* docs/server.md
* docs/security.md
* docs/secrets.md
* docs/monitoring.md
* postgres/README.md
* gitea/README.md
* openproject/README.md
+43 -5
View File
@@ -459,6 +459,7 @@ Deploy a dedicated monitoring Docker stack using:
* Grafana for visualization
* Node Exporter for host metrics
* cAdvisor for Docker metrics
* Alertmanager for alert routing and incident notifications
Reason:
@@ -466,6 +467,7 @@ Reason:
* Grafana provides flexible dashboards and datasource integration.
* Node Exporter provides host CPU, memory and disk visibility.
* cAdvisor provides Docker container visibility.
* Alertmanager provides grouping, deduplication and multi-recipient email routing.
Status:
@@ -479,6 +481,7 @@ Validation:
* Prometheus targets validated
* Grafana datasource connected to Prometheus
* Initial monitoring dashboards created and validated
* Alertmanager email routing validated
---
@@ -490,9 +493,10 @@ Export SilverLinux backup metrics through the Node Exporter Textfile Collector.
Metrics:
* Backup success
* Backup duration
* Backup size
* `silverlinux_backup_success`
* `silverlinux_backup_duration_seconds`
* `silverlinux_backup_size_bytes`
* `silverlinux_backup_timestamp_seconds`
Reason:
@@ -508,11 +512,11 @@ Implemented
---
## 2026-06-25 - Validate Backup v2 And Restore Workflow
## 2026-06-25 - Validate Backup v5.3 And Restore Workflow
Decision:
Treat Backup v2 as the operational SilverLinux backup system.
Treat Backup v5.3 as the operational SilverLinux backup system.
Implemented:
@@ -525,6 +529,10 @@ Implemented:
* Backup metrics
* Automatic low disk space cleanup
* Daily cron execution
* Lock file protection
* Disk space validation
* Central logging
* Structured failure handling
Validation:
@@ -545,6 +553,36 @@ Implemented
---
## 2026-06-25 - Implement Incident Intelligence Alerting
Decision:
Use Alertmanager with incident grouping, severity-based routing and alert deduplication for SilverLinux monitoring.
Incident groups:
* `backup_failure`
* `system_pressure`
* `storage_pressure`
* `database_outage`
* `monitoring_failure`
* `container_health`
Reason:
* Merge related alerts into single incidents.
* Reduce notification noise.
* Route alerts by severity.
* Make alerts root-cause oriented.
Status:
```text
Implemented
```
---
## Future Decisions
Document future decisions using the following template:
+20
View File
@@ -147,6 +147,25 @@ Configured and validated
---
### Alertmanager
Purpose:
* Monitoring alert emails
* Incident grouped notifications
* Severity-based alert routing
Recipients:
* `oclick021@gmail.com`
* `martin@silversolutions.nl`
Status:
Configured and validated
---
## Future Applications
Future services should use the centralized SMTP configuration whenever possible.
@@ -231,5 +250,6 @@ Verify:
* docs/secrets.md
* docs/security.md
* docs/backups.md
* docs/monitoring.md
* openproject/README.md
* gitea/README.md
+311
View File
@@ -0,0 +1,311 @@
# Monitoring And Alerting
## Overview
SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.
Current state:
```text
Production ready
```
The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.
---
## Stack Components
### Prometheus
Purpose:
* Metrics collection
* Alert rule evaluation
* Monitoring data storage
Status:
```text
Running
```
---
### Grafana
Purpose:
* Monitoring dashboards
* Prometheus datasource visualization
* Backup, system and Docker visibility
Dashboards:
* Custom backup dashboard
* Custom system dashboard
* Docker monitoring dashboard
Compatibility note:
The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
Status:
```text
Running
```
---
### Node Exporter
Purpose:
* Host metrics
* CPU, memory and disk metrics
* Textfile Collector for custom backup metrics
Status:
```text
Running
```
---
### cAdvisor
Purpose:
* Docker container metrics
* Container CPU and memory visibility
* Container health and restart visibility
Status:
```text
Running
```
---
### Alertmanager
Purpose:
* Alert routing
* Alert grouping
* Alert deduplication
* Email notifications
* Incident intelligence
Recipients:
* `oclick021@gmail.com`
* `martin@silversolutions.nl`
Status:
```text
Running
```
---
## Infrastructure
Location:
```text
/srv/docker/monitoring
```
Architecture:
* Docker-based stack
* Persistent Prometheus volume
* Persistent Grafana volume
* Dedicated monitoring network
---
## Metrics
### System Metrics
Examples:
* `node_load1`
* `node_memory_*`
* `node_filesystem_*`
* Container metrics through cAdvisor
---
### Backup Metrics
Export method:
```text
Node Exporter Textfile Collector
```
Metrics:
* `silverlinux_backup_success`
* `silverlinux_backup_duration_seconds`
* `silverlinux_backup_size_bytes`
* `silverlinux_backup_timestamp_seconds`
---
## Prometheus Scrape Targets
Prometheus successfully scrapes:
* Prometheus
* Node Exporter
* cAdvisor
* SilverLinux backup metrics
Status:
```text
Validated
```
---
## Alerting Evolution
### Stage v1
Implemented:
* Basic Prometheus rules
* Simple email alerts
---
### Stage v2
Implemented:
* Alertmanager
* Email routing
* Multi-recipient alerts
---
### Stage v3-v4
Implemented:
* `incident_group` concept
* Alert correlation by service
* Severity-based routing
* Alert deduplication
* Reduced notification noise
---
## Incident Groups
Alertmanager groups alerts by incident category.
Incident groups:
* `backup_failure`
* `system_pressure`
* `storage_pressure`
* `database_outage`
* `monitoring_failure`
* `container_health`
Outcome:
* Multiple related alerts merge into single incidents.
* Alerts are root-cause oriented.
* Alert fatigue is reduced.
---
## Alert Rules
### Backup
* `BackupFailed`
* `BackupStale`
### System
* `HighCPU`
* `HighMemoryUsage`
* `DiskAlmostFull`
### Monitoring
* `PrometheusDown`
* `GrafanaDown`
### Databases
* `PostgresDown`
* `MSSQLDown`
### Containers
* `ContainerDown`
* `TooManyRestarts`
---
## Implementation Fixes
Resolved during implementation:
* Fixed Docker mount errors caused by file versus directory conflicts
* Standardized alert rule file naming as `alerts.yml`
* Fixed Prometheus rule loading configuration
* Eliminated duplicate backup success metric sources
* Unified backup metric naming to `silverlinux_backup_*`
---
## Data Flow
```text
Servers
-> Node Exporter / cAdvisor
-> Prometheus
-> Alertmanager
-> Email notifications
-> Grafana dashboards
```
---
## Validation
Validated:
* Prometheus UI
* Prometheus targets
* Node Exporter
* cAdvisor
* Backup metrics
* Grafana datasource
* Alertmanager email routing
* Backup success and failure visibility
---
## Related Documentation
* docs/services.md
* docs/backups.md
* docs/security.md
* docs/network.md
* docs/email.md
* docs/decisions.md
+4
View File
@@ -80,6 +80,7 @@ Internal monitoring services:
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
Verified subnet:
@@ -188,6 +189,7 @@ Services:
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
Validated:
@@ -196,6 +198,7 @@ Validated:
* Prometheus scrapes cAdvisor
* Prometheus scrapes SilverLinux backup metrics
* Grafana connects to Prometheus
* Alertmanager routes monitoring alerts
---
@@ -292,5 +295,6 @@ Periodically verify:
* docs/services.md
* docs/security.md
* docs/monitoring.md
* docs/decisions.md
* nginx-proxy-manager/README.md
+18 -2
View File
@@ -51,6 +51,7 @@ Completed components:
* Grafana
* Node Exporter
* cAdvisor
* Alertmanager
Validation:
@@ -58,6 +59,14 @@ Validation:
* Prometheus targets validated
* Grafana datasource connected to Prometheus
* Initial dashboards created and validated
* Alertmanager email routing validated
Alerting:
* Incident grouping implemented
* Severity-based routing implemented
* Alert deduplication implemented
* Multi-recipient email alerts implemented
---
@@ -257,7 +266,7 @@ Planned
Current backup status:
```text
Backup v2 operational
Backup v5.3 production
```
Completed:
@@ -269,6 +278,10 @@ Completed:
* Automatic cleanup
* Backup metrics export
* Restore validation
* Lock file protection
* Disk space validation
* Central logging
* Structured failure handling
---
@@ -288,7 +301,9 @@ Completed:
* Prometheus
* Node Exporter
* cAdvisor
* Alertmanager
* Backup metrics integration
* Incident intelligence
---
@@ -303,7 +318,7 @@ Items:
Completed:
* Backup v2 automation
* Backup v5.3 automation
* Restore validation
---
@@ -340,4 +355,5 @@ Each roadmap item should receive documentation before deployment:
* docs/security.md
* docs/network.md
* docs/backups.md
* docs/monitoring.md
* docs/decisions.md
+42
View File
@@ -180,6 +180,7 @@ Running
Purpose:
* Metrics collection
* Alert rule engine
* Monitoring data storage
* Scrape targets for system, Docker and backup metrics
@@ -190,6 +191,14 @@ Scrapes:
* cAdvisor
* SilverLinux backup metrics
Alert categories:
* Backup
* System pressure
* Monitoring health
* Database availability
* Container health
Status:
```text
@@ -224,6 +233,38 @@ Running
---
### Alertmanager
Purpose:
* Alert routing
* Email notifications
* Incident grouping
* Alert deduplication
* Severity-based routing
Recipients:
* `oclick021@gmail.com`
* `martin@silversolutions.nl`
Incident groups:
* `backup_failure`
* `system_pressure`
* `storage_pressure`
* `database_outage`
* `monitoring_failure`
* `container_health`
Status:
```text
Running
```
---
### Node Exporter
Purpose:
@@ -552,6 +593,7 @@ Cleanup:
* docs/server.md
* docs/network.md
* docs/roadmap.md
* docs/monitoring.md
* docs/domains.md
* docs/security.md
* docs/backups.md