diff --git a/README.md b/README.md index c0e77be..cda3bd7 100644 --- a/README.md +++ b/README.md @@ -34,11 +34,26 @@ SilverLinux is the primary self-hosted platform for Silver Solutions and hosts s * BaGet * Gitea Actions Runner * DbGate +* Xray +* Prometheus +* Grafana +* Node Exporter +* cAdvisor ### Planned -* Sentry * Nextcloud +* Loki +* Redis +* Authelia +* CrowdSec +* Sentry +* Restic +* Harbor +* SonarQube +* Backstage + +See [docs/roadmap.md](docs/roadmap.md) for the phased infrastructure roadmap. ### Removed @@ -54,6 +69,9 @@ The following components are shared across multiple services: * Docker Compose * Internal Docker Network * Proxy Docker Network +* Isolated Docker Networks +* Monitoring Docker Stack +* Automated Backup System * Shared Secrets Management Secrets are stored outside the repository: @@ -130,6 +148,8 @@ This repository should contain enough information to rebuild the entire SilverLi See: * docs/server.md +* docs/network.md +* docs/roadmap.md * docs/backups.md * docs/security.md * docs/secrets.md diff --git a/SUMMARY.md b/SUMMARY.md index 30f5cad..10f2772 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,6 +1,6 @@ # SilverLinux Infrastructure Summary -Last Updated: 2026-06-22 +Last Updated: 2026-06-25 ## Overview @@ -8,6 +8,8 @@ SilverLinux is the primary infrastructure server for Silver Solutions. The platform hosts source control, project management, video conferencing, container management and supporting infrastructure services. +Monitoring, automated backups and restore validation are now operational. + --- ## Public Services @@ -20,6 +22,7 @@ The platform hosts source control, project management, video conferencing, conta | Jitsi Meet | https://meet.silveressence.net | | BaGet | https://nuget.silveressence.net | | DbGate | https://dbgate.silveressence.net | +| Xray | Public tunneling endpoint | --- @@ -130,6 +133,73 @@ Purpose: --- +### Monitoring Stack + +Purpose: + +* Metrics collection +* Monitoring dashboards +* Host and Docker visibility +* Backup metrics visibility + +Components: + +* Prometheus +* Grafana +* Node Exporter +* cAdvisor + +Validated: + +* Prometheus UI +* Prometheus targets +* Node Exporter +* cAdvisor +* Backup metrics +* Grafana datasource + +Dashboards: + +* Custom backup dashboard +* Custom system dashboard +* Docker monitoring dashboard + +Status: + +* Operational + +--- + +### Automated Backup System + +Purpose: + +* Daily backup execution +* Archive generation +* HTML reporting +* Email notification +* Restore validation +* Prometheus backup metrics + +Current version: + +* Backup v2 + +Validated: + +* Backup execution +* Restore archive extraction +* MSSQL backup integrity +* PostgreSQL backup +* Email notifications +* Cron automation + +Status: + +* Operational + +--- + ### BaGet Purpose: @@ -176,6 +246,25 @@ Status: --- +### Xray + +Purpose: + +* Connectivity and tunneling layer +* Improve stable access paths to SilverLinux + +Access: + +* Attached to `proxy` +* Publishes `32443 -> 8080/tcp` +* `xray_default` exists, but no running Xray container was attached to it in the 2026-06-24 runtime snapshot + +Status: + +* Operational + +--- + ## Shared Infrastructure ### Docker @@ -191,6 +280,18 @@ Purpose: * Private communication between services +### Proxy Network + +Purpose: + +* Public HTTP and HTTPS services through Nginx Proxy Manager + +### Isolated Networks + +Purpose: + +* Separate specialized stacks such as Jitsi, Xray and CI/CD runners where active runtime attachments support it + ### Shared Secrets Location: @@ -220,7 +321,7 @@ OpenProject is the primary project management platform. ### Plane.so Removed -Plane.so was removed to reduce maintenance complexity and duplicate functionality. +Plane.so was removed to reduce maintenance complexity and duplicate functionality. It is not used and will not be used. Its containers, Nginx Proxy Manager route and unused SSL certificate were removed. ### Centralized Secrets @@ -230,27 +331,55 @@ All shared credentials are stored in: /srv/secrets/company.env ``` +### Xray Connectivity Layer + +Xray is classified as a tunneling and connectivity layer rather than an application backend. + +### Multi-Network Segmentation + +SilverLinux uses `internal`, `proxy` and isolated infrastructure networks to separate database, public and specialized stack traffic. + --- ## Future Roadmap -### Phase 1 +See: -* Enable Google OAuth2 -* Enable GitHub OAuth2 -* Configure automated backups +```text +docs/roadmap.md +``` -### Phase 2 +### Phase 1 - Immediate Impact -* Expand package publishing to additional repositories +* Loki logging +* Redis +* Authelia SSO -### Phase 3 +Completed: -* Deploy Sentry +* Grafana + Prometheus +* Node Exporter +* cAdvisor +* Backup metrics integration -### Phase 4 +### Phase 2 - Security And Stability -* Automated deployment to Windows IIS servers +* CrowdSec +* Sentry +* Fail2Ban upgrade +* Cloudflare rate limiting and WAF + +Completed: + +* Backup v2 automation +* Restore validation + +### Phase 3 - Scale And Enterprise Services + +* SonarQube +* Harbor Docker registry +* Backstage portal +* Nextcloud --- diff --git a/docs/AI_CONTEXT.md b/docs/AI_CONTEXT.md index 87a9b6a..6af7861 100644 --- a/docs/AI_CONTEXT.md +++ b/docs/AI_CONTEXT.md @@ -28,6 +28,11 @@ Public IP: 51.255.83.140 * BaGet * Gitea Actions Runner (`silverlinux-runner`) * DbGate +* Xray +* Prometheus +* Grafana +* Node Exporter +* cAdvisor ## Domains @@ -43,6 +48,8 @@ nuget.silveressence.net -> BaGet (through Nginx Proxy Manager) dbgate.silveressence.net -> DbGate (through Nginx Proxy Manager) +Xray -> Public tunneling endpoint through proxy network + ## Secrets Secrets are stored in: @@ -57,9 +64,14 @@ Always use env_file. proxy internal +jitsi-docker-jitsi-meet-aa76415_meet.jitsi +xray_default +gitea-runner_default MSSQL is attached only to `internal`. DbGate is attached to both `internal` and `proxy`. +Jitsi uses `jitsi-docker-jitsi-meet-aa76415_meet.jitsi` for its internal cluster and `proxy` for web access. Xray is currently attached to `proxy`; `xray_default` exists but had no running containers attached in the 2026-06-24 runtime snapshot. + ## Authentication Gitea: @@ -91,6 +103,53 @@ MSSQL uses hostname `mssql` on the `internal` Docker network. Port 1433 is not p DbGate provides authenticated HTTPS administration for MSSQL and PostgreSQL at `https://dbgate.silveressence.net`. +Xray is a connectivity layer, not an application backend. It runs from `ghcr.io/xtls/xray-core:latest` and publishes `0.0.0.0:32443 -> 8080/tcp`. Its endpoint exposure and access policy should be reviewed during security checks. + +## Monitoring + +The monitoring stack is operational. + +Components: + +* Prometheus +* Grafana +* Node Exporter +* cAdvisor + +Validated: + +* Prometheus UI +* Prometheus targets +* Node Exporter +* cAdvisor +* Backup metrics +* Grafana datasource + +Grafana has custom backup, system and Docker monitoring dashboards. + +## Backup + +Backup v2 is operational. + +Implemented: + +* Daily cron backup +* HTML reports +* Email reports through Postfix and Gmail SMTP +* Automatic archive generation +* Automatic cleanup +* Prometheus metrics export +* Restore validation + +Verified: + +* PostgreSQL +* MSSQL +* Gitea +* BaGet +* OpenProject assets +* Docker configuration + ## Documentation Read all files in: @@ -99,6 +158,20 @@ docs/ and the README.md of each service folder before making infrastructure recommendations. +Roadmap: + +* Phase 1 completed: Grafana, Prometheus, Node Exporter, cAdvisor, backup metrics integration +* Phase 1 remaining: Loki, Redis, Authelia +* Phase 2 completed: Backup v2 automation, restore validation +* Phase 2 remaining: CrowdSec, Sentry, Fail2Ban upgrade, Cloudflare rate limiting and WAF +* Phase 3: SonarQube, Harbor, Backstage, Nextcloud + +See: + +```text +docs/roadmap.md +``` + ## Important Decisions * Plane.so removed @@ -110,3 +183,5 @@ and the README.md of each service folder before making infrastructure recommenda * Gitea Actions and a global self-hosted runner selected for package publishing * MSSQL Express deployed alongside PostgreSQL * DbGate selected as the database administration portal +* Xray introduced as a tunneling and connectivity layer +* Multi-network segmentation formalized: internal, proxy and isolated infrastructure networks diff --git a/docs/backups.md b/docs/backups.md index ec968da..224774b 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -6,6 +6,58 @@ This document describes the backup strategy for SilverLinux. The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios. +Current state: + +```text +Backup v2 operational +``` + +--- + +## Backup System Status + +### Backup v1.4 + +Implemented: + +* HTML backup reports +* Email notification after backup +* Automatic archive creation +* Backup logging +* Restore validation + +--- + +### Backup v1.5 + +Implemented: + +* MSSQL backup fix +* Runtime MSSQL data included correctly +* Restore tests verified MSSQL data is present in backups +* Improved backup structure + +--- + +### Backup v2 + +Implemented: + +* Backup metrics generation +* Prometheus metrics export +* Backup duration metric +* Backup size metric +* Backup success metric +* Automatic low disk space handling +* Automatic deletion of oldest backups when required +* Improved logging + +Status: + +```text +Operational +``` + --- ## Backup Priorities @@ -41,12 +93,13 @@ Required: * Native SQL Server database backups * MSSQL Compose configuration * `MSSQL_SA_PASSWORD` from the shared secrets backup +* Runtime MSSQL data included by the v1.5+ backup structure Importance: Critical -Do not rely on copying live database files as the only backup method. +Restore validation has verified that MSSQL data is present in current backup archives. --- @@ -241,8 +294,42 @@ Medium --- +#### Xray Configuration + +Location: + +```text +/srv/docker/xray +``` + +Contains: + +* Xray Docker Compose configuration +* Xray runtime configuration +* Endpoint credentials, UUIDs or keys if stored with the service + +Importance: + +Medium + +Xray credentials and keys must be backed up securely and must never be committed to Git. + +--- + ## Backup Storage +Current backup capabilities: + +* Automatic archive generation +* HTML backup report generation +* Backup log generation +* Email report delivery +* Prometheus metrics export +* Automatic cleanup when disk space is low +* Automatic deletion of oldest backups when required + +--- + Current Location: ```text @@ -262,6 +349,14 @@ Suggested Structure: ## Retention Policy +Current cleanup behavior: + +* Backup v2 monitors available disk space. +* If disk space is low, the oldest backups are deleted automatically. +* Cleanup is logged. + +--- + ### Daily Keep: @@ -303,6 +398,99 @@ Restore order: 9. BaGet 10. Gitea Actions Runner 11. DbGate +12. Xray + +--- + +## Restore Validation + +Restore procedure has been validated. + +Completed: + +* Dry-run restore +* Full archive validation +* Archive extraction + +Verified: + +* PostgreSQL +* Microsoft SQL Server +* Gitea +* BaGet +* OpenProject assets +* Docker configuration + +Validation results: + +* MSSQL backup integrity verified +* PostgreSQL backup verified +* Restore archive extraction verified + +--- + +## Monitoring Integration + +Backup v2 exports metrics for Prometheus. + +Current metrics: + +* Backup success +* Backup duration +* Backup size + +Export path: + +```text +Node Exporter Textfile Collector +``` + +Prometheus target status: + +```text +Validated +``` + +--- + +## Automation + +Current automation: + +* Daily backup execution +* Root cron configuration +* Backup wrapper command +* Email report +* HTML report +* Automatic cleanup +* Automatic archive generation + +Status: + +```text +Operational +``` + +--- + +## Email Notifications + +Postfix is configured to relay through Gmail SMTP. + +Implemented: + +* Gmail relay +* App Password authentication +* TLS encryption +* Successful email delivery +* Backup success emails +* Backup report emails + +Status: + +```text +Operational +``` --- @@ -363,3 +551,4 @@ Verify: * openproject/README.md * mssql/README.md * dbgate/README.md +* docs/roadmap.md diff --git a/docs/decisions.md b/docs/decisions.md index 86b2709..2eb7c5b 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -322,6 +322,229 @@ Implemented --- +## 2026-06-24 - Formalize Multi-Network Segmentation + +Decision: + +* Treat SilverLinux as a three-layer Docker network architecture. +* Use `internal` for private databases and backend service traffic. +* Use `proxy` for public HTTP and HTTPS services routed through Nginx Proxy Manager. +* Use isolated stack networks for Jitsi, Xray and CI/CD runner traffic where the active runtime attachments support it. + +Reason: + +* Keep databases private while allowing selected web interfaces to be public. +* Reduce unnecessary cross-service connectivity. +* Make specialized infrastructure stacks easier to reason about and audit. + +Status: + +```text +Accepted +``` + +--- + +## 2026-06-24 - Confirm Nginx Proxy Manager as the Single Reverse Proxy + +Decision: + +Use Nginx Proxy Manager as the sole reverse proxy and SSL termination layer for SilverLinux HTTP and HTTPS services. + +Reason: + +* The active platform uses Nginx Proxy Manager for domain routing. +* No Traefik or alternate reverse proxy is active. +* A single proxy layer keeps exposure and certificate management centralized. + +Status: + +```text +Confirmed +``` + +--- + +## 2026-06-24 - Add Xray Connectivity Layer + +Decision: + +* Classify Xray as a network tunneling and connectivity layer. +* Attach Xray to the `proxy` network for public endpoint access. +* Keep `xray_default` documented as the Compose-created Xray stack network. +* Monitor the service as an exposed connectivity endpoint rather than as an application backend. + +Reason: + +* Improve connectivity stability to SilverLinux. +* Support secure long-distance connection paths. +* Keep tunneling concerns separate from application and database services. + +Status: + +```text +Accepted +``` + +Runtime Note: + +The 2026-06-24 Docker snapshot showed the running `xray` container attached to `proxy` only. The `xray_default` network existed but had no running containers attached. + +--- + +## 2026-06-24 - Confirm Jitsi Meet Is Fully Deployed + +Decision: + +Treat the Jitsi Meet stack as an operational isolated collaboration stack. + +Components: + +* jitsi-web +* jitsi-jvb +* jitsi-jicofo +* jitsi-prosody + +Networks: + +* `proxy` +* `jitsi-docker-jitsi-meet-aa76415_meet.jitsi` + +Reason: + +* Jitsi Web is expected to be public. +* Internal Jitsi components should remain isolated inside the Jitsi stack network. + +Status: + +```text +Confirmed +``` + +--- + +## 2026-06-24 - Adopt Phased Infrastructure Roadmap + +Decision: + +Use a phased roadmap for planned SilverLinux platform improvements. + +Phases: + +* Phase 1: Grafana, Prometheus, Loki, Redis and Authelia. +* Phase 2: CrowdSec, Sentry, Restic, Fail2Ban upgrade and Cloudflare rate limiting/WAF. +* Phase 3: SonarQube, Harbor, Backstage and Nextcloud. + +Reason: + +* Prioritize observability, access control and shared runtime foundations first. +* Add security and backup maturity before larger platform expansion. +* Keep enterprise developer-platform tools grouped together. + +Status: + +```text +Accepted +``` + +--- + +## 2026-06-25 - Select Prometheus And Grafana Monitoring Stack + +Decision: + +Deploy a dedicated monitoring Docker stack using: + +* Prometheus for metrics collection +* Grafana for visualization +* Node Exporter for host metrics +* cAdvisor for Docker metrics + +Reason: + +* Prometheus provides a standard pull-based metrics system. +* Grafana provides flexible dashboards and datasource integration. +* Node Exporter provides host CPU, memory and disk visibility. +* cAdvisor provides Docker container visibility. + +Status: + +```text +Implemented +``` + +Validation: + +* Prometheus UI validated +* Prometheus targets validated +* Grafana datasource connected to Prometheus +* Initial monitoring dashboards created and validated + +--- + +## 2026-06-25 - Export Backup Metrics Through Node Exporter + +Decision: + +Export SilverLinux backup metrics through the Node Exporter Textfile Collector. + +Metrics: + +* Backup success +* Backup duration +* Backup size + +Reason: + +* Keep backup status visible in the same monitoring stack as host and container metrics. +* Avoid a separate metrics service for backup scripts. +* Allow Grafana dashboards to show backup health. + +Status: + +```text +Implemented +``` + +--- + +## 2026-06-25 - Validate Backup v2 And Restore Workflow + +Decision: + +Treat Backup v2 as the operational SilverLinux backup system. + +Implemented: + +* HTML backup reports +* Email notifications +* Automatic archive creation +* Backup logging +* Restore validation +* MSSQL data inclusion fix +* Backup metrics +* Automatic low disk space cleanup +* Daily cron execution + +Validation: + +* Dry-run restore completed +* Full archive validation completed +* PostgreSQL restore coverage verified +* MSSQL backup integrity verified +* Gitea backup verified +* BaGet backup verified +* OpenProject assets verified +* Docker configuration verified + +Status: + +```text +Implemented +``` + +--- + ## Future Decisions Document future decisions using the following template: diff --git a/docs/domains.md b/docs/domains.md index 2224e2c..001b68e 100644 --- a/docs/domains.md +++ b/docs/domains.md @@ -6,6 +6,12 @@ This document contains the public domains used by SilverLinux and the services t All public services are routed through Nginx Proxy Manager. +Verified proxy-host snapshot: + +```text +2026-06-24 +``` + --- ## Primary Domain @@ -22,7 +28,58 @@ Used for: --- -## Active Domains +## Nginx Proxy Manager Hosts + +| ID | Domain | Forward target | SSL Force | HTTP/2 | HSTS | WebSocket | Access List | Status | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | `portainer.silveressence.net` | `http://portainer:9000` | enabled | enabled | enabled | enabled | none | enabled | +| 2 | `git.silveressence.net` | `http://gitea:3000` | disabled | disabled | disabled | enabled | none | enabled | +| 3 | `meet.silveressence.net` | `http://jitsi-docker-jitsi-meet-aa76415-web-1:80` | disabled | disabled | disabled | enabled | none | enabled | +| 5 | `team.silveressence.net` | `http://openproject:80` | enabled | enabled | disabled | enabled | none | enabled | +| 6 | `nuget.silveressence.net` | `http://baget:80` | disabled | disabled | disabled | disabled | none | enabled | +| 8 | `dbgate.silveressence.net` | `http://dbgate:3000` | disabled | disabled | disabled | enabled | none | enabled | + +All listed active inventory hosts have Block Exploits enabled. + +--- + +## Additional Active Proxy Hosts + +These proxy hosts were present and enabled in Nginx Proxy Manager during the 2026-06-24 snapshot, but are not part of the current core infrastructure service inventory. + +| ID | Domain | Forward target | Reason | +| --- | --- | --- | --- | +| 7 | `cicd.silveressence.net` | `http://bobsAutowas:8080` | Active prototype deployment and test endpoint; may change frequently | +| 9 | `exactsyncer.silveressence.net` | `http://exact-syncer:8080` | Active client-facing demo for the Exact Online syncer application | + +Action: + +* Keep `cicd.silveressence.net` documented as a fast-changing prototype endpoint. +* Keep `exactsyncer.silveressence.net` documented as an intentional client-facing demo endpoint. +* Document owner, authentication model and exposure policy for active non-core endpoints. + +Plane.so cleanup: + +* Plane.so containers were removed. +* `plane.silveressence.net` was removed from Nginx Proxy Manager. +* The unused Plane.so SSL certificate was removed from the server. + +--- + +## Proxy Hardening Review + +Nginx Proxy Manager findings from the 2026-06-24 snapshot: + +* `portainer.silveressence.net` has SSL Force, HTTP/2 and HSTS enabled. +* `team.silveressence.net` has SSL Force and HTTP/2 enabled. +* `git.silveressence.net`, `meet.silveressence.net`, `nuget.silveressence.net` and `dbgate.silveressence.net` do not have SSL Force enabled. +* No listed proxy host uses an Nginx Proxy Manager access list. +* DbGate is a high-sensitivity database administration interface and should be prioritized for SSL Force and an access restriction policy. +* Plane.so has been removed from containers, Nginx Proxy Manager and server certificate storage. + +--- + +## Domain Details ### Gitea @@ -87,7 +144,7 @@ Purpose: Container Stack: ```text -jitsi +jitsi-docker-jitsi-meet-aa76415-web-1 ``` --- @@ -243,4 +300,5 @@ portainer * docs/server.md * docs/security.md +* docs/network.md * nginx-proxy-manager/README.md diff --git a/docs/email.md b/docs/email.md index 2e43e48..2b82422 100644 --- a/docs/email.md +++ b/docs/email.md @@ -10,6 +10,7 @@ Email is used for: * User invitations * Notifications * System alerts +* Backup reports * Future CI/CD notifications --- @@ -126,6 +127,26 @@ Configured --- +### Backup System + +Purpose: + +* Backup success emails +* HTML backup report emails +* Operational backup notifications + +Delivery: + +* Postfix relay through Gmail SMTP +* Gmail App Password authentication +* TLS encryption + +Status: + +Configured and validated + +--- + ## Future Applications Future services should use the centralized SMTP configuration whenever possible. @@ -134,7 +155,6 @@ Examples: * BaGet * CI/CD notifications -* Monitoring systems * Alerting systems --- @@ -186,6 +206,15 @@ Verify: * STARTTLS is enabled * Port 587 is used +### Backup Reports + +Verify: + +* Postfix relay is active +* Gmail App Password authentication succeeds +* TLS delivery succeeds +* Backup wrapper sends the HTML report after backup completion + --- ## Security Notes @@ -201,5 +230,6 @@ Verify: * docs/secrets.md * docs/security.md +* docs/backups.md * openproject/README.md * gitea/README.md diff --git a/docs/network.md b/docs/network.md new file mode 100644 index 0000000..3daab1e --- /dev/null +++ b/docs/network.md @@ -0,0 +1,296 @@ +# Network Architecture + +## Overview + +SilverLinux uses Docker networks to separate public entry points, private service communication and isolated infrastructure stacks. + +Current model: + +* `internal` for private databases and backend service traffic +* `proxy` for HTTP and HTTPS services routed by Nginx Proxy Manager +* Isolated stack networks for specialized infrastructure such as Jitsi, Xray, monitoring and CI/CD runners + +Verified runtime snapshot: + +```text +2026-06-24 10:15 CEST +``` + +--- + +## Network Layers + +### Internal Layer + +Purpose: + +* Private database access +* Backend-to-database communication +* Service traffic that should not be reachable from the internet + +Network: + +```text +internal +``` + +Services: + +* PostgreSQL +* Microsoft SQL Server Express +* Gitea backend connectivity +* OpenProject backend connectivity +* DbGate database connectivity + +Policy: + +* Database ports must not be published on the host +* Database engines must not be routed through Nginx Proxy Manager +* Only containers that require private backend access should join this network + +--- + +### Proxy Layer + +Purpose: + +* Public HTTP and HTTPS entry point +* SSL termination +* Domain routing through Nginx Proxy Manager + +Network: + +```text +proxy +``` + +Public services: + +* Gitea +* OpenProject +* Portainer +* DbGate +* Xray +* Jitsi Web +* BaGet + +Internal monitoring services: + +* Prometheus +* Grafana +* Node Exporter +* cAdvisor + +Verified subnet: + +```text +172.18.0.0/16 +``` + +Policy: + +* Nginx Proxy Manager is the single reverse proxy system +* HTTP services should be exposed through Nginx Proxy Manager instead of direct host ports +* Public management and administration services require strong authentication + +--- + +### Isolated Infrastructure Layers + +Purpose: + +* Keep specialized stacks separate from the shared application and database networks +* Limit cross-service traffic to only the attachments each stack requires + +Networks: + +```text +jitsi-docker-jitsi-meet-aa76415_meet.jitsi +xray_default +gitea-runner_default +monitoring stack network +``` + +#### jitsi-docker-jitsi-meet-aa76415_meet.jitsi + +Purpose: + +* Isolated Jitsi communication cluster + +Services: + +* jitsi-web +* jitsi-jvb +* jitsi-jicofo +* jitsi-prosody + +Public attachment: + +* All currently running Jitsi containers also attach to `proxy`. +* Public user access is expected through the Jitsi Web component. + +Verified subnet: + +```text +172.20.0.0/16 +``` + +#### xray_default + +Purpose: + +* Isolated tunneling and connectivity system +* Compose-created network for the Xray stack + +Services: + +* No running containers were attached in the 2026-06-24 runtime snapshot + +Public attachment: + +* The running `xray` container is attached to `proxy`. +* The running `xray` container was not attached to `xray_default` in the 2026-06-24 runtime snapshot. + +Verified subnet: + +```text +172.22.0.0/16 +``` + +#### gitea-runner_default + +Purpose: + +* Runner execution and workflow isolation +* Keep build traffic separate from public services and private databases unless explicit access is required + +Runtime attachment: + +* `gitea-runner` is attached to both `gitea-runner_default` and `internal`. + +Verified subnet: + +```text +172.21.0.0/16 +``` + +#### Monitoring Stack Network + +Purpose: + +* Dedicated monitoring stack communication +* Prometheus scraping of monitoring targets +* Grafana datasource connectivity to Prometheus + +Services: + +* Prometheus +* Grafana +* Node Exporter +* cAdvisor + +Validated: + +* Prometheus scrapes Prometheus +* Prometheus scrapes Node Exporter +* Prometheus scrapes cAdvisor +* Prometheus scrapes SilverLinux backup metrics +* Grafana connects to Prometheus + +--- + +## Runtime-Published Ports + +The current Docker runtime includes these host-published ports. + +| Service | Published port | Container port | UFW state | Notes | +| --- | --- | --- | --- | --- | +| Nginx Proxy Manager | `80`, `81`, `443` | `80`, `81`, `443` | `80` and `443` allowed; `81` not listed | Public proxy and NPM admin interface | +| Gitea | `2222` | `22` | Allowed | Git SSH access | +| OpenProject | `8082` | `80` | Not listed as allowed | Direct HTTP port also published | +| BaGet | `5555` | `80` | Not listed as allowed | Direct HTTP port also published | +| Jitsi Web | `8000`, `8443` | `80`, `443` | `8443` allowed; `8000` not listed | Direct HTTP and HTTPS ports also published | +| Jitsi JVB | `10000/udp` | `10000/udp` | Allowed | Required media bridge traffic | +| Xray | `32443` | `8080` | Allowed | Public tunneling endpoint | + +Additional UFW allowed ports with no matching Docker-published service in the 2026-06-24 Docker snapshot: + +* `8080/tcp` +* `8085/tcp` +* `8444/tcp` + +Database ports: + +* PostgreSQL exposes `5432/tcp` only inside Docker. +* MSSQL exposes `1433/tcp` only inside Docker. +* DbGate exposes `3000/tcp` only inside Docker. + +--- + +## Topology Summary + +```text +Internet + -> Nginx Proxy Manager + -> proxy network + -> Gitea + -> OpenProject + -> Portainer + -> DbGate + -> Xray + -> Jitsi Web + -> BaGet + +internal network + -> PostgreSQL + -> Microsoft SQL Server Express + -> backend service database access + +isolated networks + -> jitsi-docker-jitsi-meet-aa76415_meet.jitsi + -> xray_default + -> gitea-runner_default + -> monitoring stack network +``` + +--- + +## Default Docker Networks + +Docker default networks remain present: + +* `bridge` +* `host` +* `none` + +These are system defaults and are not the preferred routing model for SilverLinux application services. + +--- + +## Verification Checklist + +Periodically verify: + +* PostgreSQL and MSSQL are attached only to private networks required by their consumers +* Nginx Proxy Manager remains the only reverse proxy for HTTP services +* Nginx Proxy Manager routes match the documented proxy-host table in `docs/domains.md` +* DbGate authentication is enabled and tested +* DbGate and Portainer have an access restriction policy +* Xray exposure policy is documented and monitored +* Jitsi Web is reachable through `proxy` +* Jitsi internal components remain attached to `jitsi-docker-jitsi-meet-aa76415_meet.jitsi` +* The desired Xray attachment is reconciled with the fact that `xray_default` currently has no running containers +* Direct host-published ports are intentional and protected by firewall policy +* UFW-only allowed ports have documented owners +* Prometheus targets remain healthy +* Grafana datasource connection remains healthy +* Unused Docker networks are removed only after confirming they are not active dependencies + +--- + +## Related Documentation + +* docs/services.md +* docs/security.md +* docs/decisions.md +* nginx-proxy-manager/README.md diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..298e8a4 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,343 @@ +# Roadmap + +## Overview + +This document tracks planned SilverLinux infrastructure improvements. + +The roadmap focuses on observability, access control, security hardening, platform services and backup maturity. + +--- + +## Planned Capabilities + +### Collaboration And Storage + +#### Nextcloud + +Purpose: + +* Self-hosted file sharing +* Team document storage +* Optional collaboration layer for internal operations + +Status: + +```text +Planned +``` + +--- + +### Observability + +#### Grafana And Prometheus + +Purpose: + +* Metrics collection +* Dashboards and visualization +* Host and container health monitoring +* Alerting foundation + +Status: + +```text +Completed +``` + +Completed components: + +* Prometheus +* Grafana +* Node Exporter +* cAdvisor + +Validation: + +* Prometheus UI validated +* Prometheus targets validated +* Grafana datasource connected to Prometheus +* Initial dashboards created and validated + +--- + +#### Loki + +Purpose: + +* Central log aggregation +* Log exploration through Grafana +* Faster incident investigation + +Status: + +```text +Planned +``` + +--- + +#### Sentry + +Purpose: + +* Application error tracking +* Release health visibility +* Exception diagnostics for deployed apps + +Status: + +```text +Planned +``` + +--- + +### Identity And Access + +#### Authelia + +Purpose: + +* Identity and access management for exposed admin areas +* SSO gateway for sensitive services +* Additional protection for DbGate, Portainer, Gitea admin areas and OpenProject admin areas + +Status: + +```text +Planned +``` + +--- + +### Security Hardening + +#### Fail2Ban Upgrade + +Purpose: + +* Improve brute-force protection +* Expand jail coverage beyond SSH where practical +* Align detection with exposed administrative services + +Status: + +```text +Planned +``` + +--- + +#### CrowdSec + +Purpose: + +* Collaborative intrusion detection +* Behavior-based blocking +* Additional protection for public endpoints + +Status: + +```text +Planned +``` + +--- + +#### Cloudflare Rate Limiting And WAF + +Purpose: + +* Rate limiting for exposed admin areas +* WAF rules for common attack patterns +* Brute-force protection before traffic reaches SilverLinux + +Target areas: + +* DbGate +* Portainer +* Gitea administrator areas +* OpenProject administrator areas +* Prototype and demo endpoints where appropriate + +Status: + +```text +Planned +``` + +--- + +### Platform Services + +#### Harbor + +Purpose: + +* Docker image registry +* Internal image distribution +* Container artifact management + +Status: + +```text +Planned +``` + +--- + +#### SonarQube + +Purpose: + +* Code quality scanning +* Security scanning +* Technical debt tracking + +Status: + +```text +Planned +``` + +--- + +#### Backstage + +Purpose: + +* Internal developer portal +* Service catalog +* Platform documentation entry point + +Status: + +```text +Planned +``` + +--- + +### Performance + +#### Redis + +Purpose: + +* Caching layer for databases and high-traffic application reads +* Session storage where supported by applications +* Future scaling support + +Status: + +```text +Planned +``` + +--- + +### Backup And Restore + +#### Restic + +Purpose: + +* Versioned backup jobs +* Restore workflows +* Encrypted backups +* Better backup retention management + +Status: + +```text +Planned +``` + +Current backup status: + +```text +Backup v2 operational +``` + +Completed: + +* Daily cron backup +* Email reports +* HTML reports +* Automatic archive generation +* Automatic cleanup +* Backup metrics export +* Restore validation + +--- + +## Priority Phases + +### Phase 1 - Immediate Impact + +Items: + +* Loki +* Redis +* Authelia + +Completed: + +* Grafana +* Prometheus +* Node Exporter +* cAdvisor +* Backup metrics integration + +--- + +### Phase 2 - Security And Stability + +Items: + +* CrowdSec +* Sentry +* Fail2Ban upgrade +* Cloudflare rate limiting and WAF + +Completed: + +* Backup v2 automation +* Restore validation + +--- + +### Phase 3 - Scale And Enterprise Services + +Items: + +* SonarQube +* Harbor Docker registry +* Backstage portal +* Nextcloud + +--- + +## Documentation Requirements + +Each roadmap item should receive documentation before deployment: + +* Service purpose +* Docker Compose location +* Data and backup paths +* Network attachments +* Public exposure policy +* Authentication model +* Secrets required +* Restore procedure + +--- + +## Related Documentation + +* docs/services.md +* docs/security.md +* docs/network.md +* docs/backups.md +* docs/decisions.md diff --git a/docs/security.md b/docs/security.md index 1a4ef10..7de2af1 100644 --- a/docs/security.md +++ b/docs/security.md @@ -34,16 +34,46 @@ ubuntu UFW is enabled and used as the primary host firewall. -Allowed ports: +Verified allowed incoming ports from `sudo ufw status verbose`: | Port | Purpose | | ---- | ------- | | 22 | SSH | | 80 | HTTP | | 443 | HTTPS | +| 2222 | Gitea Git SSH | +| 10000/udp | Jitsi JVB media | +| 8085 | Explicitly allowed non-standard TCP port | +| 8443 | Explicitly allowed non-standard HTTPS port | +| 8444 | Explicitly allowed non-standard TCP port | +| 8080 | Explicitly allowed non-standard TCP port | +| 32443 | Xray public tunneling endpoint | + +Default policy: + +* deny incoming +* allow outgoing +* deny routed All other ports should remain blocked unless explicitly required. +Known runtime-published non-standard ports from the 2026-06-24 Docker snapshot: + +| Port | Service | UFW state | Notes | +| ---- | ------- | --------- | ----- | +| 81 | Nginx Proxy Manager | Not listed as allowed | Admin interface is published by Docker, but not allowed by UFW snapshot | +| 2222 | Gitea | Allowed | Git SSH | +| 32443 | Xray | Allowed | Public tunneling endpoint | +| 5555 | BaGet | Not listed as allowed | Direct HTTP port is published by Docker, but not allowed by UFW snapshot | +| 8000 | Jitsi Web | Not listed as allowed | Direct HTTP port is published by Docker, but not allowed by UFW snapshot | +| 8443 | Jitsi Web | Allowed | Direct HTTPS port | +| 10000/udp | Jitsi JVB | Allowed | Required media bridge traffic | +| 8082 | OpenProject | Not listed as allowed | Direct HTTP port is published by Docker, but not allowed by UFW snapshot | + +These ports must either be intentionally allowed and documented in UFW, or blocked at the host firewall if only Nginx Proxy Manager should be public. + +UFW also allows `8080`, `8085` and `8444`, but no matching Docker-published service was identified in the 2026-06-24 Docker snapshot. These rules should be reviewed and removed if they are no longer required. + --- ## Intrusion Protection @@ -58,6 +88,12 @@ Purpose: * Automatically block malicious IP addresses * Protect SSH access +Planned improvements: + +* Upgrade Fail2Ban coverage +* Add CrowdSec for collaborative intrusion detection +* Add Cloudflare rate limiting and WAF for exposed administrative endpoints + --- ## Secrets Management @@ -100,6 +136,43 @@ All public services should be accessible through HTTPS only. --- +## Public Exposure + +Public HTTP and HTTPS services are routed through the `proxy` Docker network and Nginx Proxy Manager. + +Publicly exposed services: + +* Gitea +* OpenProject +* Portainer +* DbGate +* Xray +* Jitsi Web +* BaGet + +High-sensitivity exposure: + +* DbGate exposes a database administration interface and must always require authentication and HTTPS. +* Xray exposes tunneling capability and must be monitored for abuse, unexpected traffic and unauthorized use. +* Portainer exposes Docker management and must be restricted to trusted administrators. + +Nginx Proxy Manager findings from the 2026-06-24 proxy-host snapshot: + +* No proxy host had an Nginx Proxy Manager access list assigned. +* DbGate was enabled without SSL Force, HTTP/2 or HSTS. +* Gitea, Jitsi Meet and BaGet were enabled without SSL Force. +* Plane.so has been removed from containers, Nginx Proxy Manager and server certificate storage. +* `cicd.silveressence.net` is intentionally enabled as a frequently changing prototype deployment endpoint. +* `exactsyncer.silveressence.net` is intentionally enabled as a client-facing Exact Online syncer demo endpoint. + +Priority hardening: + +1. Enable SSL Force for all HTTPS public service routes unless there is a documented exception. +2. Add an access list or equivalent restriction for DbGate and Portainer. +3. Define explicit exposure and authentication expectations for prototype and demo endpoints. + +--- + ## Email Security Outgoing email is provided through Gmail SMTP. @@ -137,6 +210,7 @@ Future OAuth providers: * Google OAuth2 * GitHub OAuth2 +* Authelia SSO for exposed admin areas Requirements: @@ -144,6 +218,13 @@ Requirements: * Use least-privilege scopes * Disable unused authentication providers +Planned protected areas: + +* DbGate +* Portainer +* Gitea administrator areas +* OpenProject administrator areas + ### Disabled * OpenID authentication @@ -180,6 +261,19 @@ DbGate is the public administration entry point and must require authentication --- +## Connectivity Layer Security + +Xray is classified as a network tunneling and connectivity layer, not as an application service. + +Requirements: + +* Keep the Xray service isolated from private database networks unless a documented need exists. +* Monitor logs and traffic patterns for abuse. +* Review access policy after configuration changes. +* Keep secrets, UUIDs, keys and endpoint credentials outside the repository. + +--- + ## Docker Security Services are isolated through Docker containers. @@ -195,6 +289,16 @@ Networks: * proxy * internal +* jitsi-docker-jitsi-meet-aa76415_meet.jitsi +* xray_default +* gitea-runner_default + +Network model: + +* `internal` is for private database and backend traffic. +* `proxy` is for public HTTP and HTTPS services controlled by Nginx Proxy Manager. +* Isolated stack networks are used for Jitsi, Xray and CI/CD runner separation. +* `xray_default` currently exists without a running container attached; the active `xray` container is attached to `proxy`. --- @@ -232,11 +336,20 @@ Periodically verify: * SSH keys are current * Unused accounts are removed * Fail2Ban is active +* Planned CrowdSec and Cloudflare WAF status is reviewed * UFW is active * SSL certificates are valid * Secrets are not stored in repositories * Backups are functioning * Administrator accounts have 2FA enabled +* DbGate login is enforced +* DbGate proxy route has SSL Force and an access restriction policy +* Xray exposure policy is still valid +* Public services are still routed only through Nginx Proxy Manager +* Removed proxy hosts and unused certificates stay removed unless explicitly approved +* Prototype and demo proxy hosts have documented owners and exposure expectations +* Direct host-published ports are intentional or blocked by UFW +* UFW rules `8080`, `8085` and `8444` still have a documented owner * Review application logs * Review Docker logs * Review authentication failures @@ -262,4 +375,6 @@ If a credential is exposed: * docs/secrets.md * docs/backups.md * docs/email.md +* docs/network.md +* docs/roadmap.md * nginx-proxy-manager/README.md diff --git a/docs/server.md b/docs/server.md index f9aacf4..e8ba87c 100644 --- a/docs/server.md +++ b/docs/server.md @@ -70,6 +70,15 @@ Purpose: * Firewall management +### Monitoring Stack + +Purpose: + +* Metrics collection +* Monitoring dashboards +* Host metrics +* Docker container metrics + --- ## Docker Networks @@ -103,6 +112,45 @@ Examples: --- +### jitsi-docker-jitsi-meet-aa76415_meet.jitsi + +Purpose: + +Isolated Jitsi communication cluster. + +Examples: + +* jitsi-web +* jitsi-jvb +* jitsi-jicofo +* jitsi-prosody + +--- + +### xray_default + +Purpose: + +Compose-created tunneling and connectivity network for Xray. + +Examples: + +* No running containers were attached in the 2026-06-24 runtime snapshot + +--- + +### gitea-runner_default + +Purpose: + +Default Compose network for the Gitea Actions runner stack. + +Examples: + +* gitea-runner + +--- + ## Running Services * PostgreSQL @@ -115,6 +163,11 @@ Examples: * Gitea Actions Runner * Microsoft SQL Server 2022 Express * DbGate +* Xray +* Prometheus +* Grafana +* Node Exporter +* cAdvisor --- @@ -139,6 +192,8 @@ Contains: /srv/docker/gitea-runner /srv/docker/mssql /srv/docker/dbgate +/srv/docker/xray +/srv/docker/monitoring ``` --- @@ -257,6 +312,59 @@ silverlinux-runner --- +### Xray + +```text +/srv/docker/xray +``` + +Network: + +```text +proxy +``` + +Published port: + +```text +0.0.0.0:32443 -> 8080/tcp +``` + +Image: + +```text +ghcr.io/xtls/xray-core:latest +``` + +Runtime note: + +The `xray_default` network exists, but the running `xray` container was attached only to `proxy` in the 2026-06-24 runtime snapshot. + +--- + +### Monitoring Stack + +```text +/srv/docker/monitoring +``` + +Components: + +* Prometheus +* Grafana +* Node Exporter +* cAdvisor + +Validated: + +* Prometheus targets +* Grafana Prometheus datasource +* Backup metrics +* Host metrics +* Docker metrics + +--- + ## SSH Access Primary user: @@ -283,6 +391,7 @@ Implemented: * PostgreSQL not exposed publicly * MSSQL port 1433 not exposed publicly * Database administration is available only through authenticated HTTPS access to DbGate +* Xray is not attached to the private database network and is monitored as a tunneling endpoint * OpenID disabled in Gitea --- @@ -329,4 +438,5 @@ Recovery priorities: * docs/security.md * docs/backups.md * docs/services.md +* docs/network.md * docs/domains.md diff --git a/docs/services.md b/docs/services.md index 4b3bec3..4d9db61 100644 --- a/docs/services.md +++ b/docs/services.md @@ -4,6 +4,12 @@ This document lists all services running on SilverLinux and their purpose. +Generated state reference: + +```text +2026-06-25 infrastructure review +``` + --- ## Running Services @@ -12,7 +18,7 @@ This document lists all services running on SilverLinux and their purpose. Purpose: -* Shared database platform +* Primary shared database platform * Application data storage Used by: @@ -28,6 +34,40 @@ Running --- +### Microsoft SQL Server Express + +Purpose: + +* Secondary database platform for SQL Server compatibility +* Support future applications requiring Microsoft SQL Server +* Extend the platform beyond PostgreSQL-only workloads + +Image: + +```text +mcr.microsoft.com/mssql/server:2022-latest +``` + +Network: + +```text +internal +``` + +Exposure: + +* Port 1433 is not published +* Not routed through Nginx Proxy Manager +* Private database access only + +Status: + +```text +Operational +``` + +--- + ### Gitea URL: @@ -135,34 +175,82 @@ Running --- -### Microsoft SQL Server Express +### Prometheus Purpose: -* Support applications requiring Microsoft SQL Server -* Run alongside PostgreSQL as a private database platform +* Metrics collection +* Monitoring data storage +* Scrape targets for system, Docker and backup metrics -Image: +Scrapes: -```text -mcr.microsoft.com/mssql/server:2022-latest -``` - -Network: - -```text -internal -``` - -Exposure: - -* Port 1433 is not published -* Not routed through Nginx Proxy Manager +* Prometheus +* Node Exporter +* cAdvisor +* SilverLinux backup metrics Status: ```text -Operational +Running +``` + +--- + +### Grafana + +Purpose: + +* Monitoring dashboards +* Metrics visualization +* Prometheus datasource frontend + +Dashboards: + +* Custom backup dashboard +* Custom system dashboard +* Docker monitoring dashboard + +Note: + +The imported Prometheus 2.0 Overview dashboard is not compatible with the installed Prometheus version `3.5.4`. This is a third-party dashboard compatibility issue, not an infrastructure problem. + +Status: + +```text +Running +``` + +--- + +### Node Exporter + +Purpose: + +* Host metrics collection +* CPU, memory and disk metrics +* Textfile Collector endpoint for SilverLinux backup metrics + +Status: + +```text +Running +``` + +--- + +### cAdvisor + +Purpose: + +* Docker container metrics +* Container CPU, memory and runtime visibility + +Status: + +```text +Running ``` --- @@ -244,8 +332,9 @@ https://dbgate.silveressence.net Purpose: -* Browser-based MSSQL administration * Browser-based PostgreSQL administration +* Browser-based MSSQL administration +* Database explorer and debugging interface Networks: @@ -254,8 +343,10 @@ Networks: Security: +* Publicly reachable through the proxy layer * HTTPS enforced * DbGate username and password required +* Authentication must remain hardened and periodically verified * Database ports remain private Status: @@ -266,6 +357,54 @@ Operational --- +### Xray + +Purpose: + +* Connectivity and tunneling layer +* Improve long-distance connectivity stability to SilverLinux +* Maintain secure routing paths when direct connectivity is limited + +Classification: + +```text +Network tunneling / connectivity layer +``` + +Networks: + +* `proxy` + +Runtime ports: + +```text +0.0.0.0:32443 -> 8080/tcp +``` + +Image: + +```text +ghcr.io/xtls/xray-core:latest +``` + +Runtime note: + +* The `xray_default` network exists, but the running `xray` container was attached only to `proxy` in the 2026-06-24 runtime snapshot. + +Security: + +* Public tunneling endpoint exposure must be monitored +* Access policy and abuse monitoring must be reviewed regularly +* Not classified as an application backend service + +Status: + +```text +Operational +``` + +--- + ## Shared Infrastructure ### Docker @@ -297,6 +436,42 @@ Running --- +### Proxy Docker Network + +Purpose: + +* HTTP and HTTPS service exposure through Nginx Proxy Manager +* Shared public routing layer for web-facing services + +Status: + +```text +Running +``` + +--- + +### Isolated Docker Networks + +Purpose: + +* Keep specialized infrastructure stacks isolated from application backends +* Separate Jitsi, Xray and CI/CD runner traffic from shared database traffic + +Networks: + +* `jitsi-docker-jitsi-meet-aa76415_meet.jitsi` +* `xray_default` +* `gitea-runner_default` + +Status: + +```text +Running +``` + +--- + ### Shared Secrets Location: @@ -321,11 +496,42 @@ Active ## Removed Services +## Planned Services + +See: + +```text +docs/roadmap.md +``` + +Phase 1: + +* Loki +* Redis +* Authelia + +Phase 2: + +* CrowdSec +* Sentry +* Restic +* Fail2Ban upgrade +* Cloudflare rate limiting and WAF + +Phase 3: + +* SonarQube +* Harbor +* Backstage +* Nextcloud + +--- + ### Plane.so Reason: -OpenProject was selected as the primary project management platform. +OpenProject was selected as the primary project management platform. Plane.so is not used and will not be used. Status: @@ -333,11 +539,19 @@ Status: Removed ``` +Cleanup: + +* Containers removed +* Nginx Proxy Manager route removed +* Unused SSL certificate removed from the server + --- ## Related Documentation * docs/server.md +* docs/network.md +* docs/roadmap.md * docs/domains.md * docs/security.md * docs/backups.md