docs: add monitoring alerting updates

docs: update infrastructure state and roadmap
2026-06-25 14:11:23 +03:30 · 2026-06-25 12:10:56 +03:30
13 changed files with 2340 additions and 38 deletions
@@ -34,11 +34,27 @@ SilverLinux is the primary self-hosted platform for Silver Solutions and hosts s
 * BaGet
 * Gitea Actions Runner
 * DbGate
+* Xray
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager

 ### Planned

-* Sentry
 * Nextcloud
+* Loki
+* Redis
+* Authelia
+* CrowdSec
+* Sentry
+* Restic
+* Harbor
+* SonarQube
+* Backstage
+
+See [docs/roadmap.md](docs/roadmap.md) for the phased infrastructure roadmap.

 ### Removed

@@ -54,6 +70,9 @@ The following components are shared across multiple services:
 * Docker Compose
 * Internal Docker Network
 * Proxy Docker Network
+* Isolated Docker Networks
+* Monitoring Docker Stack
+* Automated Backup System
 * Shared Secrets Management

 Secrets are stored outside the repository:
@@ -130,6 +149,8 @@ This repository should contain enough information to rebuild the entire SilverLi
 See:

 * docs/server.md
+* docs/network.md
+* docs/roadmap.md
 * docs/backups.md
 * docs/security.md
 * docs/secrets.md
@@ -1,6 +1,6 @@
 # SilverLinux Infrastructure Summary

-Last Updated: 2026-06-22
+Last Updated: 2026-06-25

 ## Overview

@@ -8,6 +8,8 @@ SilverLinux is the primary infrastructure server for Silver Solutions.

 The platform hosts source control, project management, video conferencing, container management and supporting infrastructure services.

+Monitoring, alerting, automated backups and restore validation are now production-ready.
+
 ---

 ## Public Services
@@ -20,6 +22,7 @@ The platform hosts source control, project management, video conferencing, conta
 | Jitsi Meet  | https://meet.silveressence.net      |
 | BaGet       | https://nuget.silveressence.net     |
 | DbGate      | https://dbgate.silveressence.net    |
+| Xray        | Public tunneling endpoint           |

 ---

@@ -130,6 +133,86 @@ Purpose:

 ---

+### Monitoring Stack
+
+Purpose:
+
+* Metrics collection
+* Monitoring dashboards
+* Host and Docker visibility
+* Backup metrics visibility
+
+Components:
+
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager
+
+Validated:
+
+* Prometheus UI
+* Prometheus targets
+* Node Exporter
+* cAdvisor
+* Backup metrics
+* Grafana datasource
+* Alertmanager email routing
+
+Dashboards:
+
+* Custom backup dashboard
+* Custom system dashboard
+* Docker monitoring dashboard
+
+Alerting:
+
+* Incident grouping
+* Severity-based routing
+* Alert deduplication
+* Multi-recipient email notifications
+
+Status:
+
+* Production ready
+
+---
+
+### Automated Backup System
+
+Purpose:
+
+* Daily backup execution
+* Archive generation
+* HTML reporting
+* Email notification
+* Restore validation
+* Prometheus backup metrics
+
+Current version:
+
+* Backup v5.3
+
+Validated:
+
+* Backup execution
+* Restore archive extraction
+* MSSQL backup integrity
+* PostgreSQL backup
+* Email notifications
+* Cron automation
+* Backup metrics export
+* Lock file protection
+* Disk space validation
+* Structured failure handling
+
+Status:
+
+* Production ready
+
+---
+
 ### BaGet

 Purpose:
@@ -176,6 +259,25 @@ Status:

 ---

+### Xray
+
+Purpose:
+
+* Connectivity and tunneling layer
+* Improve stable access paths to SilverLinux
+
+Access:
+
+* Attached to `proxy`
+* Publishes `32443 -> 8080/tcp`
+* `xray_default` exists, but no running Xray container was attached to it in the 2026-06-24 runtime snapshot
+
+Status:
+
+* Operational
+
+---
+
 ## Shared Infrastructure

 ### Docker
@@ -191,6 +293,18 @@ Purpose:

 * Private communication between services

+### Proxy Network
+
+Purpose:
+
+* Public HTTP and HTTPS services through Nginx Proxy Manager
+
+### Isolated Networks
+
+Purpose:
+
+* Separate specialized stacks such as Jitsi, Xray and CI/CD runners where active runtime attachments support it
+
 ### Shared Secrets

 Location:
@@ -220,7 +334,7 @@ OpenProject is the primary project management platform.

 ### Plane.so Removed

-Plane.so was removed to reduce maintenance complexity and duplicate functionality.
+Plane.so was removed to reduce maintenance complexity and duplicate functionality. It is not used and will not be used. Its containers, Nginx Proxy Manager route and unused SSL certificate were removed.

 ### Centralized Secrets

@@ -230,27 +344,57 @@ All shared credentials are stored in:
 /srv/secrets/company.env
 ```

+### Xray Connectivity Layer
+
+Xray is classified as a tunneling and connectivity layer rather than an application backend.
+
+### Multi-Network Segmentation
+
+SilverLinux uses `internal`, `proxy` and isolated infrastructure networks to separate database, public and specialized stack traffic.
+
 ---

 ## Future Roadmap

-### Phase 1
+See:

-* Enable Google OAuth2
-* Enable GitHub OAuth2
-* Configure automated backups
+```text
+docs/roadmap.md
+```

-### Phase 2
+### Phase 1 - Immediate Impact

-* Expand package publishing to additional repositories
+* Loki logging
+* Redis
+* Authelia SSO

-### Phase 3
+Completed:

-* Deploy Sentry
+* Grafana + Prometheus
+* Node Exporter
+* cAdvisor
+* Alertmanager
+* Backup metrics integration
+* Incident intelligence

-### Phase 4
+### Phase 2 - Security And Stability

-* Automated deployment to Windows IIS servers
+* CrowdSec
+* Sentry
+* Fail2Ban upgrade
+* Cloudflare rate limiting and WAF
+
+Completed:
+
+* Backup v5.3 automation
+* Restore validation
+
+### Phase 3 - Scale And Enterprise Services
+
+* SonarQube
+* Harbor Docker registry
+* Backstage portal
+* Nextcloud

 ---

@@ -28,6 +28,12 @@ Public IP: 51.255.83.140
 * BaGet
 * Gitea Actions Runner (`silverlinux-runner`)
 * DbGate
+* Xray
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager

 ## Domains

@@ -43,6 +49,8 @@ nuget.silveressence.net -> BaGet (through Nginx Proxy Manager)

 dbgate.silveressence.net -> DbGate (through Nginx Proxy Manager)

+Xray -> Public tunneling endpoint through proxy network
+
 ## Secrets

 Secrets are stored in:
@@ -57,9 +65,14 @@ Always use env_file.

 proxy
 internal
+jitsi-docker-jitsi-meet-aa76415_meet.jitsi
+xray_default
+gitea-runner_default

 MSSQL is attached only to `internal`. DbGate is attached to both `internal` and `proxy`.

+Jitsi uses `jitsi-docker-jitsi-meet-aa76415_meet.jitsi` for its internal cluster and `proxy` for web access. Xray is currently attached to `proxy`; `xray_default` exists but had no running containers attached in the 2026-06-24 runtime snapshot.
+
 ## Authentication

 Gitea:
@@ -91,6 +104,68 @@ MSSQL uses hostname `mssql` on the `internal` Docker network. Port 1433 is not p

 DbGate provides authenticated HTTPS administration for MSSQL and PostgreSQL at `https://dbgate.silveressence.net`.

+Xray is a connectivity layer, not an application backend. It runs from `ghcr.io/xtls/xray-core:latest` and publishes `0.0.0.0:32443 -> 8080/tcp`. Its endpoint exposure and access policy should be reviewed during security checks.
+
+## Monitoring
+
+The monitoring stack is operational.
+
+Components:
+
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager
+
+Validated:
+
+* Prometheus UI
+* Prometheus targets
+* Node Exporter
+* cAdvisor
+* Backup metrics
+* Grafana datasource
+* Alertmanager email routing
+
+Grafana has custom backup, system and Docker monitoring dashboards.
+
+Alertmanager implements incident intelligence with incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+## Backup
+
+Backup v5.3 is production.
+
+Implemented:
+
+* Daily cron backup
+* HTML reports
+* Email reports through Postfix and Gmail SMTP
+* Automatic archive generation
+* Automatic cleanup
+* Prometheus metrics export
+* Restore validation
+* Lock file protection
+* Disk space validation
+* Central logging
+* Structured failure handling
+
+Verified:
+
+* PostgreSQL
+* MSSQL
+* Gitea
+* BaGet
+* OpenProject assets
+* Docker configuration
+
 ## Documentation

 Read all files in:
@@ -99,6 +174,20 @@ docs/

 and the README.md of each service folder before making infrastructure recommendations.

+Roadmap:
+
+* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, Alertmanager, backup metrics integration, incident intelligence
+* Phase 1 remaining: Loki, Redis, Authelia
+* Phase 2 completed: Backup v5.3 automation, restore validation
+* Phase 2 remaining: CrowdSec, Sentry, Fail2Ban upgrade, Cloudflare rate limiting and WAF
+* Phase 3: SonarQube, Harbor, Backstage, Nextcloud
+
+See:
+
+```text
+docs/roadmap.md
+```
+
 ## Important Decisions

 * Plane.so removed
@@ -110,3 +199,5 @@ and the README.md of each service folder before making infrastructure recommenda
 * Gitea Actions and a global self-hosted runner selected for package publishing
 * MSSQL Express deployed alongside PostgreSQL
 * DbGate selected as the database administration portal
+* Xray introduced as a tunneling and connectivity layer
+* Multi-network segmentation formalized: internal, proxy and isolated infrastructure networks
@@ -6,6 +6,86 @@ This document describes the backup strategy for SilverLinux.

 The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios.

+Current state:
+
+```text
+Backup v5.3 production
+```
+
+---
+
+## Backup System Status
+
+### Backup v1.4
+
+Implemented:
+
+* HTML backup reports
+* Email notification after backup
+* Automatic archive creation
+* Backup logging
+* Restore validation
+
+---
+
+### Backup v1.5
+
+Implemented:
+
+* MSSQL backup fix
+* Runtime MSSQL data included correctly
+* Restore tests verified MSSQL data is present in backups
+* Improved backup structure
+
+---
+
+### Backup v2
+
+Implemented:
+
+* Backup metrics generation
+* Prometheus metrics export
+* Backup duration metric
+* Backup size metric
+* Backup success metric
+* Automatic low disk space handling
+* Automatic deletion of oldest backups when required
+* Improved logging
+
+Status:
+
+```text
+Operational
+```
+
+---
+
+### Backup v5.3
+
+Implemented:
+
+* PostgreSQL backup using `pg_dumpall`
+* MSSQL backup using `sqlcmd` and `docker cp`
+* Gitea backup
+* BaGet backup
+* OpenProject backup
+* Docker configuration backup
+* Final compressed archive
+* Lock file protection
+* Disk space validation
+* Central backup logging
+* Structured failure handling
+* Prometheus Textfile Collector metrics output
+* Backup success and failure tracking
+* Duration and size monitoring
+* Historical logging
+
+Status:
+
+```text
+Production
+```
+
 ---

 ## Backup Priorities
@@ -41,12 +121,13 @@ Required:
 * Native SQL Server database backups
 * MSSQL Compose configuration
 * `MSSQL_SA_PASSWORD` from the shared secrets backup
+* Runtime MSSQL data included by the v1.5+ backup structure

 Importance:

 Critical

-Do not rely on copying live database files as the only backup method.
+Restore validation has verified that MSSQL data is present in current backup archives.

 ---

@@ -241,8 +322,42 @@ Medium

 ---

+#### Xray Configuration
+
+Location:
+
+```text
+/srv/docker/xray
+```
+
+Contains:
+
+* Xray Docker Compose configuration
+* Xray runtime configuration
+* Endpoint credentials, UUIDs or keys if stored with the service
+
+Importance:
+
+Medium
+
+Xray credentials and keys must be backed up securely and must never be committed to Git.
+
+---
+
 ## Backup Storage

+Current backup capabilities:
+
+* Automatic archive generation
+* HTML backup report generation
+* Backup log generation
+* Email report delivery
+* Prometheus metrics export
+* Automatic cleanup when disk space is low
+* Automatic deletion of oldest backups when required
+
+---
+
 Current Location:

 ```text
@@ -262,6 +377,14 @@ Suggested Structure:

 ## Retention Policy

+Current cleanup behavior:
+
+* Backup v5.3 monitors available disk space.
+* If disk space is low, the oldest backups are deleted automatically.
+* Cleanup is logged.
+
+---
+
 ### Daily

 Keep:
@@ -303,6 +426,107 @@ Restore order:
 9. BaGet
 10. Gitea Actions Runner
 11. DbGate
+12. Xray
+
+---
+
+## Restore Validation
+
+Restore procedure has been validated.
+
+Completed:
+
+* Dry-run restore
+* Full archive validation
+* Archive extraction
+
+Verified:
+
+* PostgreSQL
+* Microsoft SQL Server
+* Gitea
+* BaGet
+* OpenProject assets
+* Docker configuration
+
+Validation results:
+
+* MSSQL backup integrity verified
+* PostgreSQL backup verified
+* Restore archive extraction verified
+
+---
+
+## Monitoring Integration
+
+Backup v5.3 exports metrics for Prometheus.
+
+Current metrics:
+
+* Backup success
+* Backup duration
+* Backup size
+* Backup timestamp
+
+Metric names:
+
+* `silverlinux_backup_success`
+* `silverlinux_backup_duration_seconds`
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
+
+Export path:
+
+```text
+Node Exporter Textfile Collector
+```
+
+Prometheus target status:
+
+```text
+Validated
+```
+
+---
+
+## Automation
+
+Current automation:
+
+* Daily backup execution
+* Root cron configuration
+* Backup wrapper command
+* Email report
+* HTML report
+* Automatic cleanup
+* Automatic archive generation
+
+Status:
+
+```text
+Operational
+```
+
+---
+
+## Email Notifications
+
+Postfix is configured to relay through Gmail SMTP.
+
+Implemented:
+
+* Gmail relay
+* App Password authentication
+* TLS encryption
+* Successful email delivery
+* Backup success emails
+* Backup report emails
+
+Status:
+
+```text
+Operational
+```

 ---

@@ -358,8 +582,10 @@ Verify:
 * docs/server.md
 * docs/security.md
 * docs/secrets.md
+* docs/monitoring.md
 * postgres/README.md
 * gitea/README.md
 * openproject/README.md
 * mssql/README.md
 * dbgate/README.md
+* docs/roadmap.md
@@ -322,6 +322,267 @@ Implemented

 ---

+## 2026-06-24 - Formalize Multi-Network Segmentation
+
+Decision:
+
+* Treat SilverLinux as a three-layer Docker network architecture.
+* Use `internal` for private databases and backend service traffic.
+* Use `proxy` for public HTTP and HTTPS services routed through Nginx Proxy Manager.
+* Use isolated stack networks for Jitsi, Xray and CI/CD runner traffic where the active runtime attachments support it.
+
+Reason:
+
+* Keep databases private while allowing selected web interfaces to be public.
+* Reduce unnecessary cross-service connectivity.
+* Make specialized infrastructure stacks easier to reason about and audit.
+
+Status:
+
+```text
+Accepted
+```
+
+---
+
+## 2026-06-24 - Confirm Nginx Proxy Manager as the Single Reverse Proxy
+
+Decision:
+
+Use Nginx Proxy Manager as the sole reverse proxy and SSL termination layer for SilverLinux HTTP and HTTPS services.
+
+Reason:
+
+* The active platform uses Nginx Proxy Manager for domain routing.
+* No Traefik or alternate reverse proxy is active.
+* A single proxy layer keeps exposure and certificate management centralized.
+
+Status:
+
+```text
+Confirmed
+```
+
+---
+
+## 2026-06-24 - Add Xray Connectivity Layer
+
+Decision:
+
+* Classify Xray as a network tunneling and connectivity layer.
+* Attach Xray to the `proxy` network for public endpoint access.
+* Keep `xray_default` documented as the Compose-created Xray stack network.
+* Monitor the service as an exposed connectivity endpoint rather than as an application backend.
+
+Reason:
+
+* Improve connectivity stability to SilverLinux.
+* Support secure long-distance connection paths.
+* Keep tunneling concerns separate from application and database services.
+
+Status:
+
+```text
+Accepted
+```
+
+Runtime Note:
+
+The 2026-06-24 Docker snapshot showed the running `xray` container attached to `proxy` only. The `xray_default` network existed but had no running containers attached.
+
+---
+
+## 2026-06-24 - Confirm Jitsi Meet Is Fully Deployed
+
+Decision:
+
+Treat the Jitsi Meet stack as an operational isolated collaboration stack.
+
+Components:
+
+* jitsi-web
+* jitsi-jvb
+* jitsi-jicofo
+* jitsi-prosody
+
+Networks:
+
+* `proxy`
+* `jitsi-docker-jitsi-meet-aa76415_meet.jitsi`
+
+Reason:
+
+* Jitsi Web is expected to be public.
+* Internal Jitsi components should remain isolated inside the Jitsi stack network.
+
+Status:
+
+```text
+Confirmed
+```
+
+---
+
+## 2026-06-24 - Adopt Phased Infrastructure Roadmap
+
+Decision:
+
+Use a phased roadmap for planned SilverLinux platform improvements.
+
+Phases:
+
+* Phase 1: Grafana, Prometheus, Loki, Redis and Authelia.
+* Phase 2: CrowdSec, Sentry, Restic, Fail2Ban upgrade and Cloudflare rate limiting/WAF.
+* Phase 3: SonarQube, Harbor, Backstage and Nextcloud.
+
+Reason:
+
+* Prioritize observability, access control and shared runtime foundations first.
+* Add security and backup maturity before larger platform expansion.
+* Keep enterprise developer-platform tools grouped together.
+
+Status:
+
+```text
+Accepted
+```
+
+---
+
+## 2026-06-25 - Select Prometheus And Grafana Monitoring Stack
+
+Decision:
+
+Deploy a dedicated monitoring Docker stack using:
+
+* Prometheus for metrics collection
+* Grafana for visualization
+* Node Exporter for host metrics
+* cAdvisor for Docker metrics
+* Alertmanager for alert routing and incident notifications
+
+Reason:
+
+* Prometheus provides a standard pull-based metrics system.
+* Grafana provides flexible dashboards and datasource integration.
+* Node Exporter provides host CPU, memory and disk visibility.
+* cAdvisor provides Docker container visibility.
+* Alertmanager provides grouping, deduplication and multi-recipient email routing.
+
+Status:
+
+```text
+Implemented
+```
+
+Validation:
+
+* Prometheus UI validated
+* Prometheus targets validated
+* Grafana datasource connected to Prometheus
+* Initial monitoring dashboards created and validated
+* Alertmanager email routing validated
+
+---
+
+## 2026-06-25 - Export Backup Metrics Through Node Exporter
+
+Decision:
+
+Export SilverLinux backup metrics through the Node Exporter Textfile Collector.
+
+Metrics:
+
+* `silverlinux_backup_success`
+* `silverlinux_backup_duration_seconds`
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
+
+Reason:
+
+* Keep backup status visible in the same monitoring stack as host and container metrics.
+* Avoid a separate metrics service for backup scripts.
+* Allow Grafana dashboards to show backup health.
+
+Status:
+
+```text
+Implemented
+```
+
+---
+
+## 2026-06-25 - Validate Backup v5.3 And Restore Workflow
+
+Decision:
+
+Treat Backup v5.3 as the operational SilverLinux backup system.
+
+Implemented:
+
+* HTML backup reports
+* Email notifications
+* Automatic archive creation
+* Backup logging
+* Restore validation
+* MSSQL data inclusion fix
+* Backup metrics
+* Automatic low disk space cleanup
+* Daily cron execution
+* Lock file protection
+* Disk space validation
+* Central logging
+* Structured failure handling
+
+Validation:
+
+* Dry-run restore completed
+* Full archive validation completed
+* PostgreSQL restore coverage verified
+* MSSQL backup integrity verified
+* Gitea backup verified
+* BaGet backup verified
+* OpenProject assets verified
+* Docker configuration verified
+
+Status:
+
+```text
+Implemented
+```
+
+---
+
+## 2026-06-25 - Implement Incident Intelligence Alerting
+
+Decision:
+
+Use Alertmanager with incident grouping, severity-based routing and alert deduplication for SilverLinux monitoring.
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Reason:
+
+* Merge related alerts into single incidents.
+* Reduce notification noise.
+* Route alerts by severity.
+* Make alerts root-cause oriented.
+
+Status:
+
+```text
+Implemented
+```
+
+---
+
 ## Future Decisions

 Document future decisions using the following template:
@@ -6,6 +6,12 @@ This document contains the public domains used by SilverLinux and the services t

 All public services are routed through Nginx Proxy Manager.

+Verified proxy-host snapshot:
+
+```text
+2026-06-24
+```
+
 ---

 ## Primary Domain
@@ -22,7 +28,58 @@ Used for:

 ---

-## Active Domains
+## Nginx Proxy Manager Hosts
+
+| ID | Domain | Forward target | SSL Force | HTTP/2 | HSTS | WebSocket | Access List | Status |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 1 | `portainer.silveressence.net` | `http://portainer:9000` | enabled | enabled | enabled | enabled | none | enabled |
+| 2 | `git.silveressence.net` | `http://gitea:3000` | disabled | disabled | disabled | enabled | none | enabled |
+| 3 | `meet.silveressence.net` | `http://jitsi-docker-jitsi-meet-aa76415-web-1:80` | disabled | disabled | disabled | enabled | none | enabled |
+| 5 | `team.silveressence.net` | `http://openproject:80` | enabled | enabled | disabled | enabled | none | enabled |
+| 6 | `nuget.silveressence.net` | `http://baget:80` | disabled | disabled | disabled | disabled | none | enabled |
+| 8 | `dbgate.silveressence.net` | `http://dbgate:3000` | disabled | disabled | disabled | enabled | none | enabled |
+
+All listed active inventory hosts have Block Exploits enabled.
+
+---
+
+## Additional Active Proxy Hosts
+
+These proxy hosts were present and enabled in Nginx Proxy Manager during the 2026-06-24 snapshot, but are not part of the current core infrastructure service inventory.
+
+| ID | Domain | Forward target | Reason |
+| --- | --- | --- | --- |
+| 7 | `cicd.silveressence.net` | `http://bobsAutowas:8080` | Active prototype deployment and test endpoint; may change frequently |
+| 9 | `exactsyncer.silveressence.net` | `http://exact-syncer:8080` | Active client-facing demo for the Exact Online syncer application |
+
+Action:
+
+* Keep `cicd.silveressence.net` documented as a fast-changing prototype endpoint.
+* Keep `exactsyncer.silveressence.net` documented as an intentional client-facing demo endpoint.
+* Document owner, authentication model and exposure policy for active non-core endpoints.
+
+Plane.so cleanup:
+
+* Plane.so containers were removed.
+* `plane.silveressence.net` was removed from Nginx Proxy Manager.
+* The unused Plane.so SSL certificate was removed from the server.
+
+---
+
+## Proxy Hardening Review
+
+Nginx Proxy Manager findings from the 2026-06-24 snapshot:
+
+* `portainer.silveressence.net` has SSL Force, HTTP/2 and HSTS enabled.
+* `team.silveressence.net` has SSL Force and HTTP/2 enabled.
+* `git.silveressence.net`, `meet.silveressence.net`, `nuget.silveressence.net` and `dbgate.silveressence.net` do not have SSL Force enabled.
+* No listed proxy host uses an Nginx Proxy Manager access list.
+* DbGate is a high-sensitivity database administration interface and should be prioritized for SSL Force and an access restriction policy.
+* Plane.so has been removed from containers, Nginx Proxy Manager and server certificate storage.
+
+---
+
+## Domain Details

 ### Gitea

@@ -87,7 +144,7 @@ Purpose:
 Container Stack:

 ```text
-jitsi
+jitsi-docker-jitsi-meet-aa76415-web-1
 ```

 ---
@@ -243,4 +300,5 @@ portainer

 * docs/server.md
 * docs/security.md
+* docs/network.md
 * nginx-proxy-manager/README.md
@@ -10,6 +10,7 @@ Email is used for:
 * User invitations
 * Notifications
 * System alerts
+* Backup reports
 * Future CI/CD notifications

 ---
@@ -126,6 +127,45 @@ Configured

 ---

+### Backup System
+
+Purpose:
+
+* Backup success emails
+* HTML backup report emails
+* Operational backup notifications
+
+Delivery:
+
+* Postfix relay through Gmail SMTP
+* Gmail App Password authentication
+* TLS encryption
+
+Status:
+
+Configured and validated
+
+---
+
+### Alertmanager
+
+Purpose:
+
+* Monitoring alert emails
+* Incident grouped notifications
+* Severity-based alert routing
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Status:
+
+Configured and validated
+
+---
+
 ## Future Applications

 Future services should use the centralized SMTP configuration whenever possible.
@@ -134,7 +174,6 @@ Examples:

 * BaGet
 * CI/CD notifications
-* Monitoring systems
 * Alerting systems

 ---
@@ -186,6 +225,15 @@ Verify:
 * STARTTLS is enabled
 * Port 587 is used

+### Backup Reports
+
+Verify:
+
+* Postfix relay is active
+* Gmail App Password authentication succeeds
+* TLS delivery succeeds
+* Backup wrapper sends the HTML report after backup completion
+
 ---

 ## Security Notes
@@ -201,5 +249,7 @@ Verify:

 * docs/secrets.md
 * docs/security.md
+* docs/backups.md
+* docs/monitoring.md
 * openproject/README.md
 * gitea/README.md
@@ -0,0 +1,311 @@
+# Monitoring And Alerting
+
+## Overview
+
+SilverLinux runs a self-hosted observability stack for metrics, dashboards, backup visibility and incident alerting.
+
+Current state:
+
+```text
+Production ready
+```
+
+The stack has evolved from basic monitoring into an incident intelligence system with alert grouping, severity routing and email notifications.
+
+---
+
+## Stack Components
+
+### Prometheus
+
+Purpose:
+
+* Metrics collection
+* Alert rule evaluation
+* Monitoring data storage
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Grafana
+
+Purpose:
+
+* Monitoring dashboards
+* Prometheus datasource visualization
+* Backup, system and Docker visibility
+
+Dashboards:
+
+* Custom backup dashboard
+* Custom system dashboard
+* Docker monitoring dashboard
+
+Compatibility note:
+
+The imported Prometheus 2.0 Overview dashboard is not compatible with Prometheus `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Node Exporter
+
+Purpose:
+
+* Host metrics
+* CPU, memory and disk metrics
+* Textfile Collector for custom backup metrics
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### cAdvisor
+
+Purpose:
+
+* Docker container metrics
+* Container CPU and memory visibility
+* Container health and restart visibility
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Alertmanager
+
+Purpose:
+
+* Alert routing
+* Alert grouping
+* Alert deduplication
+* Email notifications
+* Incident intelligence
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Status:
+
+```text
+Running
+```
+
+---
+
+## Infrastructure
+
+Location:
+
+```text
+/srv/docker/monitoring
+```
+
+Architecture:
+
+* Docker-based stack
+* Persistent Prometheus volume
+* Persistent Grafana volume
+* Dedicated monitoring network
+
+---
+
+## Metrics
+
+### System Metrics
+
+Examples:
+
+* `node_load1`
+* `node_memory_*`
+* `node_filesystem_*`
+* Container metrics through cAdvisor
+
+---
+
+### Backup Metrics
+
+Export method:
+
+```text
+Node Exporter Textfile Collector
+```
+
+Metrics:
+
+* `silverlinux_backup_success`
+* `silverlinux_backup_duration_seconds`
+* `silverlinux_backup_size_bytes`
+* `silverlinux_backup_timestamp_seconds`
+
+---
+
+## Prometheus Scrape Targets
+
+Prometheus successfully scrapes:
+
+* Prometheus
+* Node Exporter
+* cAdvisor
+* SilverLinux backup metrics
+
+Status:
+
+```text
+Validated
+```
+
+---
+
+## Alerting Evolution
+
+### Stage v1
+
+Implemented:
+
+* Basic Prometheus rules
+* Simple email alerts
+
+---
+
+### Stage v2
+
+Implemented:
+
+* Alertmanager
+* Email routing
+* Multi-recipient alerts
+
+---
+
+### Stage v3-v4
+
+Implemented:
+
+* `incident_group` concept
+* Alert correlation by service
+* Severity-based routing
+* Alert deduplication
+* Reduced notification noise
+
+---
+
+## Incident Groups
+
+Alertmanager groups alerts by incident category.
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Outcome:
+
+* Multiple related alerts merge into single incidents.
+* Alerts are root-cause oriented.
+* Alert fatigue is reduced.
+
+---
+
+## Alert Rules
+
+### Backup
+
+* `BackupFailed`
+* `BackupStale`
+
+### System
+
+* `HighCPU`
+* `HighMemoryUsage`
+* `DiskAlmostFull`
+
+### Monitoring
+
+* `PrometheusDown`
+* `GrafanaDown`
+
+### Databases
+
+* `PostgresDown`
+* `MSSQLDown`
+
+### Containers
+
+* `ContainerDown`
+* `TooManyRestarts`
+
+---
+
+## Implementation Fixes
+
+Resolved during implementation:
+
+* Fixed Docker mount errors caused by file versus directory conflicts
+* Standardized alert rule file naming as `alerts.yml`
+* Fixed Prometheus rule loading configuration
+* Eliminated duplicate backup success metric sources
+* Unified backup metric naming to `silverlinux_backup_*`
+
+---
+
+## Data Flow
+
+```text
+Servers
+  -> Node Exporter / cAdvisor
+  -> Prometheus
+  -> Alertmanager
+  -> Email notifications
+  -> Grafana dashboards
+```
+
+---
+
+## Validation
+
+Validated:
+
+* Prometheus UI
+* Prometheus targets
+* Node Exporter
+* cAdvisor
+* Backup metrics
+* Grafana datasource
+* Alertmanager email routing
+* Backup success and failure visibility
+
+---
+
+## Related Documentation
+
+* docs/services.md
+* docs/backups.md
+* docs/security.md
+* docs/network.md
+* docs/email.md
+* docs/decisions.md
@@ -0,0 +1,300 @@
+# Network Architecture
+
+## Overview
+
+SilverLinux uses Docker networks to separate public entry points, private service communication and isolated infrastructure stacks.
+
+Current model:
+
+* `internal` for private databases and backend service traffic
+* `proxy` for HTTP and HTTPS services routed by Nginx Proxy Manager
+* Isolated stack networks for specialized infrastructure such as Jitsi, Xray, monitoring and CI/CD runners
+
+Verified runtime snapshot:
+
+```text
+2026-06-24 10:15 CEST
+```
+
+---
+
+## Network Layers
+
+### Internal Layer
+
+Purpose:
+
+* Private database access
+* Backend-to-database communication
+* Service traffic that should not be reachable from the internet
+
+Network:
+
+```text
+internal
+```
+
+Services:
+
+* PostgreSQL
+* Microsoft SQL Server Express
+* Gitea backend connectivity
+* OpenProject backend connectivity
+* DbGate database connectivity
+
+Policy:
+
+* Database ports must not be published on the host
+* Database engines must not be routed through Nginx Proxy Manager
+* Only containers that require private backend access should join this network
+
+---
+
+### Proxy Layer
+
+Purpose:
+
+* Public HTTP and HTTPS entry point
+* SSL termination
+* Domain routing through Nginx Proxy Manager
+
+Network:
+
+```text
+proxy
+```
+
+Public services:
+
+* Gitea
+* OpenProject
+* Portainer
+* DbGate
+* Xray
+* Jitsi Web
+* BaGet
+
+Internal monitoring services:
+
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager
+
+Verified subnet:
+
+```text
+172.18.0.0/16
+```
+
+Policy:
+
+* Nginx Proxy Manager is the single reverse proxy system
+* HTTP services should be exposed through Nginx Proxy Manager instead of direct host ports
+* Public management and administration services require strong authentication
+
+---
+
+### Isolated Infrastructure Layers
+
+Purpose:
+
+* Keep specialized stacks separate from the shared application and database networks
+* Limit cross-service traffic to only the attachments each stack requires
+
+Networks:
+
+```text
+jitsi-docker-jitsi-meet-aa76415_meet.jitsi
+xray_default
+gitea-runner_default
+monitoring stack network
+```
+
+#### jitsi-docker-jitsi-meet-aa76415_meet.jitsi
+
+Purpose:
+
+* Isolated Jitsi communication cluster
+
+Services:
+
+* jitsi-web
+* jitsi-jvb
+* jitsi-jicofo
+* jitsi-prosody
+
+Public attachment:
+
+* All currently running Jitsi containers also attach to `proxy`.
+* Public user access is expected through the Jitsi Web component.
+
+Verified subnet:
+
+```text
+172.20.0.0/16
+```
+
+#### xray_default
+
+Purpose:
+
+* Isolated tunneling and connectivity system
+* Compose-created network for the Xray stack
+
+Services:
+
+* No running containers were attached in the 2026-06-24 runtime snapshot
+
+Public attachment:
+
+* The running `xray` container is attached to `proxy`.
+* The running `xray` container was not attached to `xray_default` in the 2026-06-24 runtime snapshot.
+
+Verified subnet:
+
+```text
+172.22.0.0/16
+```
+
+#### gitea-runner_default
+
+Purpose:
+
+* Runner execution and workflow isolation
+* Keep build traffic separate from public services and private databases unless explicit access is required
+
+Runtime attachment:
+
+* `gitea-runner` is attached to both `gitea-runner_default` and `internal`.
+
+Verified subnet:
+
+```text
+172.21.0.0/16
+```
+
+#### Monitoring Stack Network
+
+Purpose:
+
+* Dedicated monitoring stack communication
+* Prometheus scraping of monitoring targets
+* Grafana datasource connectivity to Prometheus
+
+Services:
+
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager
+
+Validated:
+
+* Prometheus scrapes Prometheus
+* Prometheus scrapes Node Exporter
+* Prometheus scrapes cAdvisor
+* Prometheus scrapes SilverLinux backup metrics
+* Grafana connects to Prometheus
+* Alertmanager routes monitoring alerts
+
+---
+
+## Runtime-Published Ports
+
+The current Docker runtime includes these host-published ports.
+
+| Service | Published port | Container port | UFW state | Notes |
+| --- | --- | --- | --- | --- |
+| Nginx Proxy Manager | `80`, `81`, `443` | `80`, `81`, `443` | `80` and `443` allowed; `81` not listed | Public proxy and NPM admin interface |
+| Gitea | `2222` | `22` | Allowed | Git SSH access |
+| OpenProject | `8082` | `80` | Not listed as allowed | Direct HTTP port also published |
+| BaGet | `5555` | `80` | Not listed as allowed | Direct HTTP port also published |
+| Jitsi Web | `8000`, `8443` | `80`, `443` | `8443` allowed; `8000` not listed | Direct HTTP and HTTPS ports also published |
+| Jitsi JVB | `10000/udp` | `10000/udp` | Allowed | Required media bridge traffic |
+| Xray | `32443` | `8080` | Allowed | Public tunneling endpoint |
+
+Additional UFW allowed ports with no matching Docker-published service in the 2026-06-24 Docker snapshot:
+
+* `8080/tcp`
+* `8085/tcp`
+* `8444/tcp`
+
+Database ports:
+
+* PostgreSQL exposes `5432/tcp` only inside Docker.
+* MSSQL exposes `1433/tcp` only inside Docker.
+* DbGate exposes `3000/tcp` only inside Docker.
+
+---
+
+## Topology Summary
+
+```text
+Internet
+  -> Nginx Proxy Manager
+     -> proxy network
+        -> Gitea
+        -> OpenProject
+        -> Portainer
+        -> DbGate
+        -> Xray
+        -> Jitsi Web
+        -> BaGet
+
+internal network
+  -> PostgreSQL
+  -> Microsoft SQL Server Express
+  -> backend service database access
+
+isolated networks
+  -> jitsi-docker-jitsi-meet-aa76415_meet.jitsi
+  -> xray_default
+  -> gitea-runner_default
+  -> monitoring stack network
+```
+
+---
+
+## Default Docker Networks
+
+Docker default networks remain present:
+
+* `bridge`
+* `host`
+* `none`
+
+These are system defaults and are not the preferred routing model for SilverLinux application services.
+
+---
+
+## Verification Checklist
+
+Periodically verify:
+
+* PostgreSQL and MSSQL are attached only to private networks required by their consumers
+* Nginx Proxy Manager remains the only reverse proxy for HTTP services
+* Nginx Proxy Manager routes match the documented proxy-host table in `docs/domains.md`
+* DbGate authentication is enabled and tested
+* DbGate and Portainer have an access restriction policy
+* Xray exposure policy is documented and monitored
+* Jitsi Web is reachable through `proxy`
+* Jitsi internal components remain attached to `jitsi-docker-jitsi-meet-aa76415_meet.jitsi`
+* The desired Xray attachment is reconciled with the fact that `xray_default` currently has no running containers
+* Direct host-published ports are intentional and protected by firewall policy
+* UFW-only allowed ports have documented owners
+* Prometheus targets remain healthy
+* Grafana datasource connection remains healthy
+* Unused Docker networks are removed only after confirming they are not active dependencies
+
+---
+
+## Related Documentation
+
+* docs/services.md
+* docs/security.md
+* docs/monitoring.md
+* docs/decisions.md
+* nginx-proxy-manager/README.md
@@ -0,0 +1,359 @@
+# Roadmap
+
+## Overview
+
+This document tracks planned SilverLinux infrastructure improvements.
+
+The roadmap focuses on observability, access control, security hardening, platform services and backup maturity.
+
+---
+
+## Planned Capabilities
+
+### Collaboration And Storage
+
+#### Nextcloud
+
+Purpose:
+
+* Self-hosted file sharing
+* Team document storage
+* Optional collaboration layer for internal operations
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+### Observability
+
+#### Grafana And Prometheus
+
+Purpose:
+
+* Metrics collection
+* Dashboards and visualization
+* Host and container health monitoring
+* Alerting foundation
+
+Status:
+
+```text
+Completed
+```
+
+Completed components:
+
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+* Alertmanager
+
+Validation:
+
+* Prometheus UI validated
+* Prometheus targets validated
+* Grafana datasource connected to Prometheus
+* Initial dashboards created and validated
+* Alertmanager email routing validated
+
+Alerting:
+
+* Incident grouping implemented
+* Severity-based routing implemented
+* Alert deduplication implemented
+* Multi-recipient email alerts implemented
+
+---
+
+#### Loki
+
+Purpose:
+
+* Central log aggregation
+* Log exploration through Grafana
+* Faster incident investigation
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+#### Sentry
+
+Purpose:
+
+* Application error tracking
+* Release health visibility
+* Exception diagnostics for deployed apps
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+### Identity And Access
+
+#### Authelia
+
+Purpose:
+
+* Identity and access management for exposed admin areas
+* SSO gateway for sensitive services
+* Additional protection for DbGate, Portainer, Gitea admin areas and OpenProject admin areas
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+### Security Hardening
+
+#### Fail2Ban Upgrade
+
+Purpose:
+
+* Improve brute-force protection
+* Expand jail coverage beyond SSH where practical
+* Align detection with exposed administrative services
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+#### CrowdSec
+
+Purpose:
+
+* Collaborative intrusion detection
+* Behavior-based blocking
+* Additional protection for public endpoints
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+#### Cloudflare Rate Limiting And WAF
+
+Purpose:
+
+* Rate limiting for exposed admin areas
+* WAF rules for common attack patterns
+* Brute-force protection before traffic reaches SilverLinux
+
+Target areas:
+
+* DbGate
+* Portainer
+* Gitea administrator areas
+* OpenProject administrator areas
+* Prototype and demo endpoints where appropriate
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+### Platform Services
+
+#### Harbor
+
+Purpose:
+
+* Docker image registry
+* Internal image distribution
+* Container artifact management
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+#### SonarQube
+
+Purpose:
+
+* Code quality scanning
+* Security scanning
+* Technical debt tracking
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+#### Backstage
+
+Purpose:
+
+* Internal developer portal
+* Service catalog
+* Platform documentation entry point
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+### Performance
+
+#### Redis
+
+Purpose:
+
+* Caching layer for databases and high-traffic application reads
+* Session storage where supported by applications
+* Future scaling support
+
+Status:
+
+```text
+Planned
+```
+
+---
+
+### Backup And Restore
+
+#### Restic
+
+Purpose:
+
+* Versioned backup jobs
+* Restore workflows
+* Encrypted backups
+* Better backup retention management
+
+Status:
+
+```text
+Planned
+```
+
+Current backup status:
+
+```text
+Backup v5.3 production
+```
+
+Completed:
+
+* Daily cron backup
+* Email reports
+* HTML reports
+* Automatic archive generation
+* Automatic cleanup
+* Backup metrics export
+* Restore validation
+* Lock file protection
+* Disk space validation
+* Central logging
+* Structured failure handling
+
+---
+
+## Priority Phases
+
+### Phase 1 - Immediate Impact
+
+Items:
+
+* Loki
+* Redis
+* Authelia
+
+Completed:
+
+* Grafana
+* Prometheus
+* Node Exporter
+* cAdvisor
+* Alertmanager
+* Backup metrics integration
+* Incident intelligence
+
+---
+
+### Phase 2 - Security And Stability
+
+Items:
+
+* CrowdSec
+* Sentry
+* Fail2Ban upgrade
+* Cloudflare rate limiting and WAF
+
+Completed:
+
+* Backup v5.3 automation
+* Restore validation
+
+---
+
+### Phase 3 - Scale And Enterprise Services
+
+Items:
+
+* SonarQube
+* Harbor Docker registry
+* Backstage portal
+* Nextcloud
+
+---
+
+## Documentation Requirements
+
+Each roadmap item should receive documentation before deployment:
+
+* Service purpose
+* Docker Compose location
+* Data and backup paths
+* Network attachments
+* Public exposure policy
+* Authentication model
+* Secrets required
+* Restore procedure
+
+---
+
+## Related Documentation
+
+* docs/services.md
+* docs/security.md
+* docs/network.md
+* docs/backups.md
+* docs/monitoring.md
+* docs/decisions.md
@@ -34,16 +34,46 @@ ubuntu

 UFW is enabled and used as the primary host firewall.

-Allowed ports:
+Verified allowed incoming ports from `sudo ufw status verbose`:

 | Port | Purpose |
 | ---- | ------- |
 | 22   | SSH     |
 | 80   | HTTP    |
 | 443  | HTTPS   |
+| 2222 | Gitea Git SSH |
+| 10000/udp | Jitsi JVB media |
+| 8085 | Explicitly allowed non-standard TCP port |
+| 8443 | Explicitly allowed non-standard HTTPS port |
+| 8444 | Explicitly allowed non-standard TCP port |
+| 8080 | Explicitly allowed non-standard TCP port |
+| 32443 | Xray public tunneling endpoint |
+
+Default policy:
+
+* deny incoming
+* allow outgoing
+* deny routed

 All other ports should remain blocked unless explicitly required.

+Known runtime-published non-standard ports from the 2026-06-24 Docker snapshot:
+
+| Port | Service | UFW state | Notes |
+| ---- | ------- | --------- | ----- |
+| 81   | Nginx Proxy Manager | Not listed as allowed | Admin interface is published by Docker, but not allowed by UFW snapshot |
+| 2222 | Gitea | Allowed | Git SSH |
+| 32443 | Xray | Allowed | Public tunneling endpoint |
+| 5555 | BaGet | Not listed as allowed | Direct HTTP port is published by Docker, but not allowed by UFW snapshot |
+| 8000 | Jitsi Web | Not listed as allowed | Direct HTTP port is published by Docker, but not allowed by UFW snapshot |
+| 8443 | Jitsi Web | Allowed | Direct HTTPS port |
+| 10000/udp | Jitsi JVB | Allowed | Required media bridge traffic |
+| 8082 | OpenProject | Not listed as allowed | Direct HTTP port is published by Docker, but not allowed by UFW snapshot |
+
+These ports must either be intentionally allowed and documented in UFW, or blocked at the host firewall if only Nginx Proxy Manager should be public.
+
+UFW also allows `8080`, `8085` and `8444`, but no matching Docker-published service was identified in the 2026-06-24 Docker snapshot. These rules should be reviewed and removed if they are no longer required.
+
 ---

 ## Intrusion Protection
@@ -58,6 +88,12 @@ Purpose:
 * Automatically block malicious IP addresses
 * Protect SSH access

+Planned improvements:
+
+* Upgrade Fail2Ban coverage
+* Add CrowdSec for collaborative intrusion detection
+* Add Cloudflare rate limiting and WAF for exposed administrative endpoints
+
 ---

 ## Secrets Management
@@ -100,6 +136,43 @@ All public services should be accessible through HTTPS only.

 ---

+## Public Exposure
+
+Public HTTP and HTTPS services are routed through the `proxy` Docker network and Nginx Proxy Manager.
+
+Publicly exposed services:
+
+* Gitea
+* OpenProject
+* Portainer
+* DbGate
+* Xray
+* Jitsi Web
+* BaGet
+
+High-sensitivity exposure:
+
+* DbGate exposes a database administration interface and must always require authentication and HTTPS.
+* Xray exposes tunneling capability and must be monitored for abuse, unexpected traffic and unauthorized use.
+* Portainer exposes Docker management and must be restricted to trusted administrators.
+
+Nginx Proxy Manager findings from the 2026-06-24 proxy-host snapshot:
+
+* No proxy host had an Nginx Proxy Manager access list assigned.
+* DbGate was enabled without SSL Force, HTTP/2 or HSTS.
+* Gitea, Jitsi Meet and BaGet were enabled without SSL Force.
+* Plane.so has been removed from containers, Nginx Proxy Manager and server certificate storage.
+* `cicd.silveressence.net` is intentionally enabled as a frequently changing prototype deployment endpoint.
+* `exactsyncer.silveressence.net` is intentionally enabled as a client-facing Exact Online syncer demo endpoint.
+
+Priority hardening:
+
+1. Enable SSL Force for all HTTPS public service routes unless there is a documented exception.
+2. Add an access list or equivalent restriction for DbGate and Portainer.
+3. Define explicit exposure and authentication expectations for prototype and demo endpoints.
+
+---
+
 ## Email Security

 Outgoing email is provided through Gmail SMTP.
@@ -137,6 +210,7 @@ Future OAuth providers:

 * Google OAuth2
 * GitHub OAuth2
+* Authelia SSO for exposed admin areas

 Requirements:

@@ -144,6 +218,13 @@ Requirements:
 * Use least-privilege scopes
 * Disable unused authentication providers

+Planned protected areas:
+
+* DbGate
+* Portainer
+* Gitea administrator areas
+* OpenProject administrator areas
+
 ### Disabled

 * OpenID authentication
@@ -180,6 +261,19 @@ DbGate is the public administration entry point and must require authentication

 ---

+## Connectivity Layer Security
+
+Xray is classified as a network tunneling and connectivity layer, not as an application service.
+
+Requirements:
+
+* Keep the Xray service isolated from private database networks unless a documented need exists.
+* Monitor logs and traffic patterns for abuse.
+* Review access policy after configuration changes.
+* Keep secrets, UUIDs, keys and endpoint credentials outside the repository.
+
+---
+
 ## Docker Security

 Services are isolated through Docker containers.
@@ -195,6 +289,16 @@ Networks:

 * proxy
 * internal
+* jitsi-docker-jitsi-meet-aa76415_meet.jitsi
+* xray_default
+* gitea-runner_default
+
+Network model:
+
+* `internal` is for private database and backend traffic.
+* `proxy` is for public HTTP and HTTPS services controlled by Nginx Proxy Manager.
+* Isolated stack networks are used for Jitsi, Xray and CI/CD runner separation.
+* `xray_default` currently exists without a running container attached; the active `xray` container is attached to `proxy`.

 ---

@@ -232,11 +336,20 @@ Periodically verify:
 * SSH keys are current
 * Unused accounts are removed
 * Fail2Ban is active
+* Planned CrowdSec and Cloudflare WAF status is reviewed
 * UFW is active
 * SSL certificates are valid
 * Secrets are not stored in repositories
 * Backups are functioning
 * Administrator accounts have 2FA enabled
+* DbGate login is enforced
+* DbGate proxy route has SSL Force and an access restriction policy
+* Xray exposure policy is still valid
+* Public services are still routed only through Nginx Proxy Manager
+* Removed proxy hosts and unused certificates stay removed unless explicitly approved
+* Prototype and demo proxy hosts have documented owners and exposure expectations
+* Direct host-published ports are intentional or blocked by UFW
+* UFW rules `8080`, `8085` and `8444` still have a documented owner
 * Review application logs
 * Review Docker logs
 * Review authentication failures
@@ -262,4 +375,6 @@ If a credential is exposed:
 * docs/secrets.md
 * docs/backups.md
 * docs/email.md
+* docs/network.md
+* docs/roadmap.md
 * nginx-proxy-manager/README.md
@@ -70,6 +70,15 @@ Purpose:

 * Firewall management

+### Monitoring Stack
+
+Purpose:
+
+* Metrics collection
+* Monitoring dashboards
+* Host metrics
+* Docker container metrics
+
 ---

 ## Docker Networks
@@ -103,6 +112,45 @@ Examples:

 ---

+### jitsi-docker-jitsi-meet-aa76415_meet.jitsi
+
+Purpose:
+
+Isolated Jitsi communication cluster.
+
+Examples:
+
+* jitsi-web
+* jitsi-jvb
+* jitsi-jicofo
+* jitsi-prosody
+
+---
+
+### xray_default
+
+Purpose:
+
+Compose-created tunneling and connectivity network for Xray.
+
+Examples:
+
+* No running containers were attached in the 2026-06-24 runtime snapshot
+
+---
+
+### gitea-runner_default
+
+Purpose:
+
+Default Compose network for the Gitea Actions runner stack.
+
+Examples:
+
+* gitea-runner
+
+---
+
 ## Running Services

 * PostgreSQL
@@ -115,6 +163,11 @@ Examples:
 * Gitea Actions Runner
 * Microsoft SQL Server 2022 Express
 * DbGate
+* Xray
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor

 ---

@@ -139,6 +192,8 @@ Contains:
 /srv/docker/gitea-runner
 /srv/docker/mssql
 /srv/docker/dbgate
+/srv/docker/xray
+/srv/docker/monitoring
 ```

 ---
@@ -257,6 +312,59 @@ silverlinux-runner

 ---

+### Xray
+
+```text
+/srv/docker/xray
+```
+
+Network:
+
+```text
+proxy
+```
+
+Published port:
+
+```text
+0.0.0.0:32443 -> 8080/tcp
+```
+
+Image:
+
+```text
+ghcr.io/xtls/xray-core:latest
+```
+
+Runtime note:
+
+The `xray_default` network exists, but the running `xray` container was attached only to `proxy` in the 2026-06-24 runtime snapshot.
+
+---
+
+### Monitoring Stack
+
+```text
+/srv/docker/monitoring
+```
+
+Components:
+
+* Prometheus
+* Grafana
+* Node Exporter
+* cAdvisor
+
+Validated:
+
+* Prometheus targets
+* Grafana Prometheus datasource
+* Backup metrics
+* Host metrics
+* Docker metrics
+
+---
+
 ## SSH Access

 Primary user:
@@ -283,6 +391,7 @@ Implemented:
 * PostgreSQL not exposed publicly
 * MSSQL port 1433 not exposed publicly
 * Database administration is available only through authenticated HTTPS access to DbGate
+* Xray is not attached to the private database network and is monitored as a tunneling endpoint
 * OpenID disabled in Gitea

 ---
@@ -329,4 +438,5 @@ Recovery priorities:
 * docs/security.md
 * docs/backups.md
 * docs/services.md
+* docs/network.md
 * docs/domains.md
@@ -4,6 +4,12 @@

 This document lists all services running on SilverLinux and their purpose.

+Generated state reference:
+
+```text
+2026-06-25 infrastructure review
+```
+
 ---

 ## Running Services
@@ -12,7 +18,7 @@ This document lists all services running on SilverLinux and their purpose.

 Purpose:

-* Shared database platform
+* Primary shared database platform
 * Application data storage

 Used by:
@@ -28,6 +34,40 @@ Running

 ---

+### Microsoft SQL Server Express
+
+Purpose:
+
+* Secondary database platform for SQL Server compatibility
+* Support future applications requiring Microsoft SQL Server
+* Extend the platform beyond PostgreSQL-only workloads
+
+Image:
+
+```text
+mcr.microsoft.com/mssql/server:2022-latest
+```
+
+Network:
+
+```text
+internal
+```
+
+Exposure:
+
+* Port 1433 is not published
+* Not routed through Nginx Proxy Manager
+* Private database access only
+
+Status:
+
+```text
+Operational
+```
+
+---
+
 ### Gitea

 URL:
@@ -135,34 +175,123 @@ Running

 ---

-### Microsoft SQL Server Express
+### Prometheus

 Purpose:

-* Support applications requiring Microsoft SQL Server
-* Run alongside PostgreSQL as a private database platform
+* Metrics collection
+* Alert rule engine
+* Monitoring data storage
+* Scrape targets for system, Docker and backup metrics

-Image:
+Scrapes:

-```text
-mcr.microsoft.com/mssql/server:2022-latest
-```
+* Prometheus
+* Node Exporter
+* cAdvisor
+* SilverLinux backup metrics

-Network:
+Alert categories:

-```text
-internal
-```
-
-Exposure:
-
-* Port 1433 is not published
-* Not routed through Nginx Proxy Manager
+* Backup
+* System pressure
+* Monitoring health
+* Database availability
+* Container health

 Status:

 ```text
-Operational
+Running
+```
+
+---
+
+### Grafana
+
+Purpose:
+
+* Monitoring dashboards
+* Metrics visualization
+* Prometheus datasource frontend
+
+Dashboards:
+
+* Custom backup dashboard
+* Custom system dashboard
+* Docker monitoring dashboard
+
+Note:
+
+The imported Prometheus 2.0 Overview dashboard is not compatible with the installed Prometheus version `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem.
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Alertmanager
+
+Purpose:
+
+* Alert routing
+* Email notifications
+* Incident grouping
+* Alert deduplication
+* Severity-based routing
+
+Recipients:
+
+* `oclick021@gmail.com`
+* `martin@silversolutions.nl`
+
+Incident groups:
+
+* `backup_failure`
+* `system_pressure`
+* `storage_pressure`
+* `database_outage`
+* `monitoring_failure`
+* `container_health`
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Node Exporter
+
+Purpose:
+
+* Host metrics collection
+* CPU, memory and disk metrics
+* Textfile Collector endpoint for SilverLinux backup metrics
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### cAdvisor
+
+Purpose:
+
+* Docker container metrics
+* Container CPU, memory and runtime visibility
+
+Status:
+
+```text
+Running
 ```

 ---
@@ -244,8 +373,9 @@ https://dbgate.silveressence.net

 Purpose:

-* Browser-based MSSQL administration
 * Browser-based PostgreSQL administration
+* Browser-based MSSQL administration
+* Database explorer and debugging interface

 Networks:

@@ -254,8 +384,10 @@ Networks:

 Security:

+* Publicly reachable through the proxy layer
 * HTTPS enforced
 * DbGate username and password required
+* Authentication must remain hardened and periodically verified
 * Database ports remain private

 Status:
@@ -266,6 +398,54 @@ Operational

 ---

+### Xray
+
+Purpose:
+
+* Connectivity and tunneling layer
+* Improve long-distance connectivity stability to SilverLinux
+* Maintain secure routing paths when direct connectivity is limited
+
+Classification:
+
+```text
+Network tunneling / connectivity layer
+```
+
+Networks:
+
+* `proxy`
+
+Runtime ports:
+
+```text
+0.0.0.0:32443 -> 8080/tcp
+```
+
+Image:
+
+```text
+ghcr.io/xtls/xray-core:latest
+```
+
+Runtime note:
+
+* The `xray_default` network exists, but the running `xray` container was attached only to `proxy` in the 2026-06-24 runtime snapshot.
+
+Security:
+
+* Public tunneling endpoint exposure must be monitored
+* Access policy and abuse monitoring must be reviewed regularly
+* Not classified as an application backend service
+
+Status:
+
+```text
+Operational
+```
+
+---
+
 ## Shared Infrastructure

 ### Docker
@@ -297,6 +477,42 @@ Running

 ---

+### Proxy Docker Network
+
+Purpose:
+
+* HTTP and HTTPS service exposure through Nginx Proxy Manager
+* Shared public routing layer for web-facing services
+
+Status:
+
+```text
+Running
+```
+
+---
+
+### Isolated Docker Networks
+
+Purpose:
+
+* Keep specialized infrastructure stacks isolated from application backends
+* Separate Jitsi, Xray and CI/CD runner traffic from shared database traffic
+
+Networks:
+
+* `jitsi-docker-jitsi-meet-aa76415_meet.jitsi`
+* `xray_default`
+* `gitea-runner_default`
+
+Status:
+
+```text
+Running
+```
+
+---
+
 ### Shared Secrets

 Location:
@@ -321,11 +537,42 @@ Active

 ## Removed Services

+## Planned Services
+
+See:
+
+```text
+docs/roadmap.md
+```
+
+Phase 1:
+
+* Loki
+* Redis
+* Authelia
+
+Phase 2:
+
+* CrowdSec
+* Sentry
+* Restic
+* Fail2Ban upgrade
+* Cloudflare rate limiting and WAF
+
+Phase 3:
+
+* SonarQube
+* Harbor
+* Backstage
+* Nextcloud
+
+---
+
 ### Plane.so

 Reason:

-OpenProject was selected as the primary project management platform.
+OpenProject was selected as the primary project management platform. Plane.so is not used and will not be used.

 Status:

@@ -333,11 +580,20 @@ Status:
 Removed
 ```

+Cleanup:
+
+* Containers removed
+* Nginx Proxy Manager route removed
+* Unused SSL certificate removed from the server
+
 ---

 ## Related Documentation

 * docs/server.md
+* docs/network.md
+* docs/roadmap.md
+* docs/monitoring.md
 * docs/domains.md
 * docs/security.md
 * docs/backups.md
Author	SHA1	Message	Date
Oclick021	a74909be25	docs: add monitoring alerting updates	2026-06-25 14:11:23 +03:30
Oclick021	4cf06da0a8	docs: update infrastructure state and roadmap	2026-06-25 12:10:56 +03:30