312 lines
4.0 KiB
Markdown
312 lines
4.0 KiB
Markdown
# Monitoring And Alerting
|
|
|
|
## Overview
|
|
|
|
SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.
|
|
|
|
Current state:
|
|
|
|
```text
|
|
Production ready
|
|
```
|
|
|
|
The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.
|
|
|
|
---
|
|
|
|
## Stack Components
|
|
|
|
### Prometheus
|
|
|
|
Purpose:
|
|
|
|
* Metrics collection
|
|
* Alert rule evaluation
|
|
* Monitoring data storage
|
|
|
|
Status:
|
|
|
|
```text
|
|
Running
|
|
```
|
|
|
|
---
|
|
|
|
### Grafana
|
|
|
|
Purpose:
|
|
|
|
* Monitoring dashboards
|
|
* Prometheus datasource visualization
|
|
* Backup, system and Docker visibility
|
|
|
|
Dashboards:
|
|
|
|
* Custom backup dashboard
|
|
* Custom system dashboard
|
|
* Docker monitoring dashboard
|
|
|
|
Compatibility note:
|
|
|
|
The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
|
|
|
|
Status:
|
|
|
|
```text
|
|
Running
|
|
```
|
|
|
|
---
|
|
|
|
### Node Exporter
|
|
|
|
Purpose:
|
|
|
|
* Host metrics
|
|
* CPU, memory and disk metrics
|
|
* Textfile Collector for custom backup metrics
|
|
|
|
Status:
|
|
|
|
```text
|
|
Running
|
|
```
|
|
|
|
---
|
|
|
|
### cAdvisor
|
|
|
|
Purpose:
|
|
|
|
* Docker container metrics
|
|
* Container CPU and memory visibility
|
|
* Container health and restart visibility
|
|
|
|
Status:
|
|
|
|
```text
|
|
Running
|
|
```
|
|
|
|
---
|
|
|
|
### Alertmanager
|
|
|
|
Purpose:
|
|
|
|
* Alert routing
|
|
* Alert grouping
|
|
* Alert deduplication
|
|
* Email notifications
|
|
* Incident intelligence
|
|
|
|
Recipients:
|
|
|
|
* `oclick021@gmail.com`
|
|
* `martin@silversolutions.nl`
|
|
|
|
Status:
|
|
|
|
```text
|
|
Running
|
|
```
|
|
|
|
---
|
|
|
|
## Infrastructure
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/monitoring
|
|
```
|
|
|
|
Architecture:
|
|
|
|
* Docker-based stack
|
|
* Persistent Prometheus volume
|
|
* Persistent Grafana volume
|
|
* Dedicated monitoring network
|
|
|
|
---
|
|
|
|
## Metrics
|
|
|
|
### System Metrics
|
|
|
|
Examples:
|
|
|
|
* `node_load1`
|
|
* `node_memory_*`
|
|
* `node_filesystem_*`
|
|
* Container metrics through cAdvisor
|
|
|
|
---
|
|
|
|
### Backup Metrics
|
|
|
|
Export method:
|
|
|
|
```text
|
|
Node Exporter Textfile Collector
|
|
```
|
|
|
|
Metrics:
|
|
|
|
* `silverlinux_backup_success`
|
|
* `silverlinux_backup_duration_seconds`
|
|
* `silverlinux_backup_size_bytes`
|
|
* `silverlinux_backup_timestamp_seconds`
|
|
|
|
---
|
|
|
|
## Prometheus Scrape Targets
|
|
|
|
Prometheus successfully scrapes:
|
|
|
|
* Prometheus
|
|
* Node Exporter
|
|
* cAdvisor
|
|
* SilverLinux backup metrics
|
|
|
|
Status:
|
|
|
|
```text
|
|
Validated
|
|
```
|
|
|
|
---
|
|
|
|
## Alerting Evolution
|
|
|
|
### Stage v1
|
|
|
|
Implemented:
|
|
|
|
* Basic Prometheus rules
|
|
* Simple email alerts
|
|
|
|
---
|
|
|
|
### Stage v2
|
|
|
|
Implemented:
|
|
|
|
* Alertmanager
|
|
* Email routing
|
|
* Multi-recipient alerts
|
|
|
|
---
|
|
|
|
### Stage v3-v4
|
|
|
|
Implemented:
|
|
|
|
* `incident_group` concept
|
|
* Alert correlation by service
|
|
* Severity-based routing
|
|
* Alert deduplication
|
|
* Reduced notification noise
|
|
|
|
---
|
|
|
|
## Incident Groups
|
|
|
|
Alertmanager groups alerts by incident category.
|
|
|
|
Incident groups:
|
|
|
|
* `backup_failure`
|
|
* `system_pressure`
|
|
* `storage_pressure`
|
|
* `database_outage`
|
|
* `monitoring_failure`
|
|
* `container_health`
|
|
|
|
Outcome:
|
|
|
|
* Multiple related alerts merge into single incidents.
|
|
* Alerts are root-cause oriented.
|
|
* Alert fatigue is reduced.
|
|
|
|
---
|
|
|
|
## Alert Rules
|
|
|
|
### Backup
|
|
|
|
* `BackupFailed`
|
|
* `BackupStale`
|
|
|
|
### System
|
|
|
|
* `HighCPU`
|
|
* `HighMemoryUsage`
|
|
* `DiskAlmostFull`
|
|
|
|
### Monitoring
|
|
|
|
* `PrometheusDown`
|
|
* `GrafanaDown`
|
|
|
|
### Databases
|
|
|
|
* `PostgresDown`
|
|
* `MSSQLDown`
|
|
|
|
### Containers
|
|
|
|
* `ContainerDown`
|
|
* `TooManyRestarts`
|
|
|
|
---
|
|
|
|
## Implementation Fixes
|
|
|
|
Resolved during implementation:
|
|
|
|
* Fixed Docker mount errors caused by file versus directory conflicts
|
|
* Standardized alert rule file naming as `alerts.yml`
|
|
* Fixed Prometheus rule loading configuration
|
|
* Eliminated duplicate backup success metric sources
|
|
* Unified backup metric naming to `silverlinux_backup_*`
|
|
|
|
---
|
|
|
|
## Data Flow
|
|
|
|
```text
|
|
Servers
|
|
-> Node Exporter / cAdvisor
|
|
-> Prometheus
|
|
-> Alertmanager
|
|
-> Email notifications
|
|
-> Grafana dashboards
|
|
```
|
|
|
|
---
|
|
|
|
## Validation
|
|
|
|
Validated:
|
|
|
|
* Prometheus UI
|
|
* Prometheus targets
|
|
* Node Exporter
|
|
* cAdvisor
|
|
* Backup metrics
|
|
* Grafana datasource
|
|
* Alertmanager email routing
|
|
* Backup success and failure visibility
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
* docs/services.md
|
|
* docs/backups.md
|
|
* docs/security.md
|
|
* docs/network.md
|
|
* docs/email.md
|
|
* docs/decisions.md
|