docs: add monitoring alerting updates
This commit is contained in:
+19
-3
@@ -33,6 +33,7 @@ Public IP: 51.255.83.140
|
||||
* Grafana
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Alertmanager
|
||||
|
||||
## Domains
|
||||
|
||||
@@ -115,6 +116,7 @@ Components:
|
||||
* Grafana
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Alertmanager
|
||||
|
||||
Validated:
|
||||
|
||||
@@ -124,12 +126,22 @@ Validated:
|
||||
* cAdvisor
|
||||
* Backup metrics
|
||||
* Grafana datasource
|
||||
* Alertmanager email routing
|
||||
|
||||
Grafana has custom backup, system and Docker monitoring dashboards.
|
||||
|
||||
Alertmanager implements incident intelligence with incident groups:
|
||||
|
||||
* `backup_failure`
|
||||
* `system_pressure`
|
||||
* `storage_pressure`
|
||||
* `database_outage`
|
||||
* `monitoring_failure`
|
||||
* `container_health`
|
||||
|
||||
## Backup
|
||||
|
||||
Backup v2 is operational.
|
||||
Backup v5.3 is production.
|
||||
|
||||
Implemented:
|
||||
|
||||
@@ -140,6 +152,10 @@ Implemented:
|
||||
* Automatic cleanup
|
||||
* Prometheus metrics export
|
||||
* Restore validation
|
||||
* Lock file protection
|
||||
* Disk space validation
|
||||
* Central logging
|
||||
* Structured failure handling
|
||||
|
||||
Verified:
|
||||
|
||||
@@ -160,9 +176,9 @@ and the README.md of each service folder before making infrastructure recommenda
|
||||
|
||||
Roadmap:
|
||||
|
||||
* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, backup metrics integration
|
||||
* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, Alertmanager, backup metrics integration, incident intelligence
|
||||
* Phase 1 remaining: Loki, Redis, Authelia
|
||||
* Phase 2 completed: Backup v2 automation, restore validation
|
||||
* Phase 2 completed: Backup v5.3 automation, restore validation
|
||||
* Phase 2 remaining: CrowdSec, Sentry, Fail2Ban upgrade, Cloudflare rate limiting and WAF
|
||||
* Phase 3: SonarQube, Harbor, Backstage, Nextcloud
|
||||
|
||||
|
||||
+40
-3
@@ -9,7 +9,7 @@ The goal is to ensure that all critical services can be restored after hardware
|
||||
Current state:
|
||||
|
||||
```text
|
||||
Backup v2 operational
|
||||
Backup v5.3 production
|
||||
```
|
||||
|
||||
---
|
||||
@@ -60,6 +60,34 @@ Operational
|
||||
|
||||
---
|
||||
|
||||
### Backup v5.3
|
||||
|
||||
Implemented:
|
||||
|
||||
* PostgreSQL backup using `pg_dumpall`
|
||||
* MSSQL backup using `sqlcmd` and `docker cp`
|
||||
* Gitea backup
|
||||
* BaGet backup
|
||||
* OpenProject backup
|
||||
* Docker configuration backup
|
||||
* Final compressed archive
|
||||
* Lock file protection
|
||||
* Disk space validation
|
||||
* Central backup logging
|
||||
* Structured failure handling
|
||||
* Prometheus Textfile Collector metrics output
|
||||
* Backup success and failure tracking
|
||||
* Duration and size monitoring
|
||||
* Historical logging
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Production
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup Priorities
|
||||
|
||||
### Critical
|
||||
@@ -351,7 +379,7 @@ Suggested Structure:
|
||||
|
||||
Current cleanup behavior:
|
||||
|
||||
* Backup v2 monitors available disk space.
|
||||
* Backup v5.3 monitors available disk space.
|
||||
* If disk space is low, the oldest backups are deleted automatically.
|
||||
* Cleanup is logged.
|
||||
|
||||
@@ -431,13 +459,21 @@ Validation results:
|
||||
|
||||
## Monitoring Integration
|
||||
|
||||
Backup v2 exports metrics for Prometheus.
|
||||
Backup v5.3 exports metrics for Prometheus.
|
||||
|
||||
Current metrics:
|
||||
|
||||
* Backup success
|
||||
* Backup duration
|
||||
* Backup size
|
||||
* Backup timestamp
|
||||
|
||||
Metric names:
|
||||
|
||||
* `silverlinux_backup_success`
|
||||
* `silverlinux_backup_duration_seconds`
|
||||
* `silverlinux_backup_size_bytes`
|
||||
* `silverlinux_backup_timestamp_seconds`
|
||||
|
||||
Export path:
|
||||
|
||||
@@ -546,6 +582,7 @@ Verify:
|
||||
* docs/server.md
|
||||
* docs/security.md
|
||||
* docs/secrets.md
|
||||
* docs/monitoring.md
|
||||
* postgres/README.md
|
||||
* gitea/README.md
|
||||
* openproject/README.md
|
||||
|
||||
+43
-5
@@ -459,6 +459,7 @@ Deploy a dedicated monitoring Docker stack using:
|
||||
* Grafana for visualization
|
||||
* Node Exporter for host metrics
|
||||
* cAdvisor for Docker metrics
|
||||
* Alertmanager for alert routing and incident notifications
|
||||
|
||||
Reason:
|
||||
|
||||
@@ -466,6 +467,7 @@ Reason:
|
||||
* Grafana provides flexible dashboards and datasource integration.
|
||||
* Node Exporter provides host CPU, memory and disk visibility.
|
||||
* cAdvisor provides Docker container visibility.
|
||||
* Alertmanager provides grouping, deduplication and multi-recipient email routing.
|
||||
|
||||
Status:
|
||||
|
||||
@@ -479,6 +481,7 @@ Validation:
|
||||
* Prometheus targets validated
|
||||
* Grafana datasource connected to Prometheus
|
||||
* Initial monitoring dashboards created and validated
|
||||
* Alertmanager email routing validated
|
||||
|
||||
---
|
||||
|
||||
@@ -490,9 +493,10 @@ Export SilverLinux backup metrics through the Node Exporter Textfile Collector.
|
||||
|
||||
Metrics:
|
||||
|
||||
* Backup success
|
||||
* Backup duration
|
||||
* Backup size
|
||||
* `silverlinux_backup_success`
|
||||
* `silverlinux_backup_duration_seconds`
|
||||
* `silverlinux_backup_size_bytes`
|
||||
* `silverlinux_backup_timestamp_seconds`
|
||||
|
||||
Reason:
|
||||
|
||||
@@ -508,11 +512,11 @@ Implemented
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-25 - Validate Backup v2 And Restore Workflow
|
||||
## 2026-06-25 - Validate Backup v5.3 And Restore Workflow
|
||||
|
||||
Decision:
|
||||
|
||||
Treat Backup v2 as the operational SilverLinux backup system.
|
||||
Treat Backup v5.3 as the operational SilverLinux backup system.
|
||||
|
||||
Implemented:
|
||||
|
||||
@@ -525,6 +529,10 @@ Implemented:
|
||||
* Backup metrics
|
||||
* Automatic low disk space cleanup
|
||||
* Daily cron execution
|
||||
* Lock file protection
|
||||
* Disk space validation
|
||||
* Central logging
|
||||
* Structured failure handling
|
||||
|
||||
Validation:
|
||||
|
||||
@@ -545,6 +553,36 @@ Implemented
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-25 - Implement Incident Intelligence Alerting
|
||||
|
||||
Decision:
|
||||
|
||||
Use Alertmanager with incident grouping, severity-based routing and alert deduplication for SilverLinux monitoring.
|
||||
|
||||
Incident groups:
|
||||
|
||||
* `backup_failure`
|
||||
* `system_pressure`
|
||||
* `storage_pressure`
|
||||
* `database_outage`
|
||||
* `monitoring_failure`
|
||||
* `container_health`
|
||||
|
||||
Reason:
|
||||
|
||||
* Merge related alerts into single incidents.
|
||||
* Reduce notification noise.
|
||||
* Route alerts by severity.
|
||||
* Make alerts root-cause oriented.
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Implemented
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future Decisions
|
||||
|
||||
Document future decisions using the following template:
|
||||
|
||||
@@ -147,6 +147,25 @@ Configured and validated
|
||||
|
||||
---
|
||||
|
||||
### Alertmanager
|
||||
|
||||
Purpose:
|
||||
|
||||
* Monitoring alert emails
|
||||
* Incident grouped notifications
|
||||
* Severity-based alert routing
|
||||
|
||||
Recipients:
|
||||
|
||||
* `oclick021@gmail.com`
|
||||
* `martin@silversolutions.nl`
|
||||
|
||||
Status:
|
||||
|
||||
Configured and validated
|
||||
|
||||
---
|
||||
|
||||
## Future Applications
|
||||
|
||||
Future services should use the centralized SMTP configuration whenever possible.
|
||||
@@ -231,5 +250,6 @@ Verify:
|
||||
* docs/secrets.md
|
||||
* docs/security.md
|
||||
* docs/backups.md
|
||||
* docs/monitoring.md
|
||||
* openproject/README.md
|
||||
* gitea/README.md
|
||||
|
||||
@@ -0,0 +1,311 @@
|
||||
# Monitoring And Alerting
|
||||
|
||||
## Overview
|
||||
|
||||
SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.
|
||||
|
||||
Current state:
|
||||
|
||||
```text
|
||||
Production ready
|
||||
```
|
||||
|
||||
The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.
|
||||
|
||||
---
|
||||
|
||||
## Stack Components
|
||||
|
||||
### Prometheus
|
||||
|
||||
Purpose:
|
||||
|
||||
* Metrics collection
|
||||
* Alert rule evaluation
|
||||
* Monitoring data storage
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Running
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Grafana
|
||||
|
||||
Purpose:
|
||||
|
||||
* Monitoring dashboards
|
||||
* Prometheus datasource visualization
|
||||
* Backup, system and Docker visibility
|
||||
|
||||
Dashboards:
|
||||
|
||||
* Custom backup dashboard
|
||||
* Custom system dashboard
|
||||
* Docker monitoring dashboard
|
||||
|
||||
Compatibility note:
|
||||
|
||||
The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Running
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Node Exporter
|
||||
|
||||
Purpose:
|
||||
|
||||
* Host metrics
|
||||
* CPU, memory and disk metrics
|
||||
* Textfile Collector for custom backup metrics
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Running
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### cAdvisor
|
||||
|
||||
Purpose:
|
||||
|
||||
* Docker container metrics
|
||||
* Container CPU and memory visibility
|
||||
* Container health and restart visibility
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Running
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Alertmanager
|
||||
|
||||
Purpose:
|
||||
|
||||
* Alert routing
|
||||
* Alert grouping
|
||||
* Alert deduplication
|
||||
* Email notifications
|
||||
* Incident intelligence
|
||||
|
||||
Recipients:
|
||||
|
||||
* `oclick021@gmail.com`
|
||||
* `martin@silversolutions.nl`
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Running
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure
|
||||
|
||||
Location:
|
||||
|
||||
```text
|
||||
/srv/docker/monitoring
|
||||
```
|
||||
|
||||
Architecture:
|
||||
|
||||
* Docker-based stack
|
||||
* Persistent Prometheus volume
|
||||
* Persistent Grafana volume
|
||||
* Dedicated monitoring network
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
### System Metrics
|
||||
|
||||
Examples:
|
||||
|
||||
* `node_load1`
|
||||
* `node_memory_*`
|
||||
* `node_filesystem_*`
|
||||
* Container metrics through cAdvisor
|
||||
|
||||
---
|
||||
|
||||
### Backup Metrics
|
||||
|
||||
Export method:
|
||||
|
||||
```text
|
||||
Node Exporter Textfile Collector
|
||||
```
|
||||
|
||||
Metrics:
|
||||
|
||||
* `silverlinux_backup_success`
|
||||
* `silverlinux_backup_duration_seconds`
|
||||
* `silverlinux_backup_size_bytes`
|
||||
* `silverlinux_backup_timestamp_seconds`
|
||||
|
||||
---
|
||||
|
||||
## Prometheus Scrape Targets
|
||||
|
||||
Prometheus successfully scrapes:
|
||||
|
||||
* Prometheus
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* SilverLinux backup metrics
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Validated
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alerting Evolution
|
||||
|
||||
### Stage v1
|
||||
|
||||
Implemented:
|
||||
|
||||
* Basic Prometheus rules
|
||||
* Simple email alerts
|
||||
|
||||
---
|
||||
|
||||
### Stage v2
|
||||
|
||||
Implemented:
|
||||
|
||||
* Alertmanager
|
||||
* Email routing
|
||||
* Multi-recipient alerts
|
||||
|
||||
---
|
||||
|
||||
### Stage v3-v4
|
||||
|
||||
Implemented:
|
||||
|
||||
* `incident_group` concept
|
||||
* Alert correlation by service
|
||||
* Severity-based routing
|
||||
* Alert deduplication
|
||||
* Reduced notification noise
|
||||
|
||||
---
|
||||
|
||||
## Incident Groups
|
||||
|
||||
Alertmanager groups alerts by incident category.
|
||||
|
||||
Incident groups:
|
||||
|
||||
* `backup_failure`
|
||||
* `system_pressure`
|
||||
* `storage_pressure`
|
||||
* `database_outage`
|
||||
* `monitoring_failure`
|
||||
* `container_health`
|
||||
|
||||
Outcome:
|
||||
|
||||
* Multiple related alerts merge into single incidents.
|
||||
* Alerts are root-cause oriented.
|
||||
* Alert fatigue is reduced.
|
||||
|
||||
---
|
||||
|
||||
## Alert Rules
|
||||
|
||||
### Backup
|
||||
|
||||
* `BackupFailed`
|
||||
* `BackupStale`
|
||||
|
||||
### System
|
||||
|
||||
* `HighCPU`
|
||||
* `HighMemoryUsage`
|
||||
* `DiskAlmostFull`
|
||||
|
||||
### Monitoring
|
||||
|
||||
* `PrometheusDown`
|
||||
* `GrafanaDown`
|
||||
|
||||
### Databases
|
||||
|
||||
* `PostgresDown`
|
||||
* `MSSQLDown`
|
||||
|
||||
### Containers
|
||||
|
||||
* `ContainerDown`
|
||||
* `TooManyRestarts`
|
||||
|
||||
---
|
||||
|
||||
## Implementation Fixes
|
||||
|
||||
Resolved during implementation:
|
||||
|
||||
* Fixed Docker mount errors caused by file versus directory conflicts
|
||||
* Standardized alert rule file naming as `alerts.yml`
|
||||
* Fixed Prometheus rule loading configuration
|
||||
* Eliminated duplicate backup success metric sources
|
||||
* Unified backup metric naming to `silverlinux_backup_*`
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```text
|
||||
Servers
|
||||
-> Node Exporter / cAdvisor
|
||||
-> Prometheus
|
||||
-> Alertmanager
|
||||
-> Email notifications
|
||||
-> Grafana dashboards
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
Validated:
|
||||
|
||||
* Prometheus UI
|
||||
* Prometheus targets
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Backup metrics
|
||||
* Grafana datasource
|
||||
* Alertmanager email routing
|
||||
* Backup success and failure visibility
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
* docs/services.md
|
||||
* docs/backups.md
|
||||
* docs/security.md
|
||||
* docs/network.md
|
||||
* docs/email.md
|
||||
* docs/decisions.md
|
||||
@@ -80,6 +80,7 @@ Internal monitoring services:
|
||||
* Grafana
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Alertmanager
|
||||
|
||||
Verified subnet:
|
||||
|
||||
@@ -188,6 +189,7 @@ Services:
|
||||
* Grafana
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Alertmanager
|
||||
|
||||
Validated:
|
||||
|
||||
@@ -196,6 +198,7 @@ Validated:
|
||||
* Prometheus scrapes cAdvisor
|
||||
* Prometheus scrapes SilverLinux backup metrics
|
||||
* Grafana connects to Prometheus
|
||||
* Alertmanager routes monitoring alerts
|
||||
|
||||
---
|
||||
|
||||
@@ -292,5 +295,6 @@ Periodically verify:
|
||||
|
||||
* docs/services.md
|
||||
* docs/security.md
|
||||
* docs/monitoring.md
|
||||
* docs/decisions.md
|
||||
* nginx-proxy-manager/README.md
|
||||
|
||||
+18
-2
@@ -51,6 +51,7 @@ Completed components:
|
||||
* Grafana
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Alertmanager
|
||||
|
||||
Validation:
|
||||
|
||||
@@ -58,6 +59,14 @@ Validation:
|
||||
* Prometheus targets validated
|
||||
* Grafana datasource connected to Prometheus
|
||||
* Initial dashboards created and validated
|
||||
* Alertmanager email routing validated
|
||||
|
||||
Alerting:
|
||||
|
||||
* Incident grouping implemented
|
||||
* Severity-based routing implemented
|
||||
* Alert deduplication implemented
|
||||
* Multi-recipient email alerts implemented
|
||||
|
||||
---
|
||||
|
||||
@@ -257,7 +266,7 @@ Planned
|
||||
Current backup status:
|
||||
|
||||
```text
|
||||
Backup v2 operational
|
||||
Backup v5.3 production
|
||||
```
|
||||
|
||||
Completed:
|
||||
@@ -269,6 +278,10 @@ Completed:
|
||||
* Automatic cleanup
|
||||
* Backup metrics export
|
||||
* Restore validation
|
||||
* Lock file protection
|
||||
* Disk space validation
|
||||
* Central logging
|
||||
* Structured failure handling
|
||||
|
||||
---
|
||||
|
||||
@@ -288,7 +301,9 @@ Completed:
|
||||
* Prometheus
|
||||
* Node Exporter
|
||||
* cAdvisor
|
||||
* Alertmanager
|
||||
* Backup metrics integration
|
||||
* Incident intelligence
|
||||
|
||||
---
|
||||
|
||||
@@ -303,7 +318,7 @@ Items:
|
||||
|
||||
Completed:
|
||||
|
||||
* Backup v2 automation
|
||||
* Backup v5.3 automation
|
||||
* Restore validation
|
||||
|
||||
---
|
||||
@@ -340,4 +355,5 @@ Each roadmap item should receive documentation before deployment:
|
||||
* docs/security.md
|
||||
* docs/network.md
|
||||
* docs/backups.md
|
||||
* docs/monitoring.md
|
||||
* docs/decisions.md
|
||||
|
||||
@@ -180,6 +180,7 @@ Running
|
||||
Purpose:
|
||||
|
||||
* Metrics collection
|
||||
* Alert rule engine
|
||||
* Monitoring data storage
|
||||
* Scrape targets for system, Docker and backup metrics
|
||||
|
||||
@@ -190,6 +191,14 @@ Scrapes:
|
||||
* cAdvisor
|
||||
* SilverLinux backup metrics
|
||||
|
||||
Alert categories:
|
||||
|
||||
* Backup
|
||||
* System pressure
|
||||
* Monitoring health
|
||||
* Database availability
|
||||
* Container health
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
@@ -224,6 +233,38 @@ Running
|
||||
|
||||
---
|
||||
|
||||
### Alertmanager
|
||||
|
||||
Purpose:
|
||||
|
||||
* Alert routing
|
||||
* Email notifications
|
||||
* Incident grouping
|
||||
* Alert deduplication
|
||||
* Severity-based routing
|
||||
|
||||
Recipients:
|
||||
|
||||
* `oclick021@gmail.com`
|
||||
* `martin@silversolutions.nl`
|
||||
|
||||
Incident groups:
|
||||
|
||||
* `backup_failure`
|
||||
* `system_pressure`
|
||||
* `storage_pressure`
|
||||
* `database_outage`
|
||||
* `monitoring_failure`
|
||||
* `container_health`
|
||||
|
||||
Status:
|
||||
|
||||
```text
|
||||
Running
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Node Exporter
|
||||
|
||||
Purpose:
|
||||
@@ -552,6 +593,7 @@ Cleanup:
|
||||
* docs/server.md
|
||||
* docs/network.md
|
||||
* docs/roadmap.md
|
||||
* docs/monitoring.md
|
||||
* docs/domains.md
|
||||
* docs/security.md
|
||||
* docs/backups.md
|
||||
|
||||
Reference in New Issue
Block a user