# Backups ## Overview This document describes the backup strategy for SilverLinux. The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios. Current state: ```text Backup v5.3 production ``` --- ## Backup System Status ### Backup v1.4 Implemented: * HTML backup reports * Email notification after backup * Automatic archive creation * Backup logging * Restore validation --- ### Backup v1.5 Implemented: * MSSQL backup fix * Runtime MSSQL data included correctly * Restore tests verified MSSQL data is present in backups * Improved backup structure --- ### Backup v2 Implemented: * Backup metrics generation * Prometheus metrics export * Backup duration metric * Backup size metric * Backup success metric * Automatic low disk space handling * Automatic deletion of oldest backups when required * Improved logging Status: ```text Operational ``` --- ### Backup v5.3 Implemented: * PostgreSQL backup using `pg_dumpall` * MSSQL backup using `sqlcmd` and `docker cp` * Gitea backup * BaGet backup * OpenProject backup * Docker configuration backup * Final compressed archive * Lock file protection * Disk space validation * Central backup logging * Structured failure handling * Prometheus Textfile Collector metrics output * Backup success and failure tracking * Duration and size monitoring * Historical logging Status: ```text Production ``` --- ## Backup Priorities ### Critical These items must always be backed up. #### PostgreSQL Contains: * OpenProject database * Gitea database * Future application databases Importance: Critical --- #### Microsoft SQL Server Data location: ```text /srv/docker/mssql/data ``` Required: * Native SQL Server database backups * MSSQL Compose configuration * `MSSQL_SA_PASSWORD` from the shared secrets backup * Runtime MSSQL data included by the v1.5+ backup structure Importance: Critical Restore validation has verified that MSSQL data is present in current backup archives. --- #### Gitea Data Location: ```text /srv/docker/gitea/data ``` Contains: * Git repositories * Attachments * Configuration * User data Importance: Critical --- #### OpenProject Assets Location: ```text /srv/docker/openproject/assets ``` Contains: * Attachments * Uploaded files * User generated content Importance: Critical --- #### Shared Secrets Location: ```text /srv/secrets/company.env ``` Contains: * SMTP credentials * PostgreSQL passwords * MSSQL SA password * OpenProject secrets * Future DbGate password * Future OAuth secrets Importance: Critical Without this file applications may not start correctly. --- #### Nginx Proxy Manager Locations: ```text /srv/docker/nginx-proxy-manager/data /srv/docker/nginx-proxy-manager/letsencrypt ``` Contains: * Proxy configuration * SSL certificates * Domain routing Importance: Critical --- ### Medium Priority #### Portainer Volume: ```text portainer_portainer_data ``` Contains: * Portainer users * Portainer configuration * Stack definitions Importance: Medium Can be recreated if necessary. --- #### Jitsi Configuration Location: ```text /srv/docker/jitsi ``` Contains: * Jitsi configuration * Internal authentication configuration Importance: Medium --- #### BaGet Data Location: ```text /srv/docker/baget/data ``` Contains: * Hosted NuGet packages * SQLite database Importance: Medium --- #### Gitea Actions Runner Configuration Location: ```text /srv/docker/gitea-runner ``` Contains: * Runner deployment configuration * Runner state The registration token is stored separately in `/srv/secrets/company.env`. Importance: Medium --- #### DbGate Data Location: ```text /srv/docker/dbgate/data ``` Contains: * Saved connections * DbGate application state Also retain the Nginx Proxy Manager route and DbGate authentication credential. Importance: Medium --- #### Xray Configuration Location: ```text /srv/docker/xray ``` Contains: * Xray Docker Compose configuration * Xray runtime configuration * Endpoint credentials, UUIDs or keys if stored with the service Importance: Medium Xray credentials and keys must be backed up securely and must never be committed to Git. --- ## Backup Storage Current backup capabilities: * Automatic archive generation * HTML backup report generation * Backup log generation * Email report delivery * Prometheus metrics export * Automatic cleanup when disk space is low * Automatic deletion of oldest backups when required --- Current Location: ```text /srv/backups ``` Suggested Structure: ```text /srv/backups/ ├── daily ├── weekly └── monthly ``` --- ## Retention Policy Current cleanup behavior: * Backup v5.3 monitors available disk space. * If disk space is low, the oldest backups are deleted automatically. * Cleanup is logged. --- ### Daily Keep: ```text 7 days ``` ### Weekly Keep: ```text 4 weeks ``` ### Monthly Keep: ```text 12 months ``` --- ## Restore Priorities Restore order: 1. Shared secrets 2. PostgreSQL 3. Microsoft SQL Server 4. Nginx Proxy Manager 5. Gitea 6. OpenProject 7. Portainer 8. Jitsi 9. BaGet 10. Gitea Actions Runner 11. DbGate 12. Xray --- ## Restore Validation Restore procedure has been validated. Completed: * Dry-run restore * Full archive validation * Archive extraction Verified: * PostgreSQL * Microsoft SQL Server * Gitea * BaGet * OpenProject assets * Docker configuration Validation results: * MSSQL backup integrity verified * PostgreSQL backup verified * Restore archive extraction verified --- ## Monitoring Integration Backup v5.3 exports metrics for Prometheus. Current metrics: * Backup success * Backup duration * Backup size * Backup timestamp Metric names: * `silverlinux_backup_success` * `silverlinux_backup_duration_seconds` * `silverlinux_backup_size_bytes` * `silverlinux_backup_timestamp_seconds` Export path: ```text Node Exporter Textfile Collector ``` Prometheus target status: ```text Validated ``` --- ## Automation Current automation: * Daily backup execution * Root cron configuration * Backup wrapper command * Email report * HTML report * Automatic cleanup * Automatic archive generation Status: ```text Operational ``` --- ## Email Notifications Postfix is configured to relay through Gmail SMTP. Implemented: * Gmail relay * App Password authentication * TLS encryption * Successful email delivery * Backup success emails * Backup report emails Status: ```text Operational ``` --- ## Disaster Recovery Checklist ### Step 1 Install: * Ubuntu Server * Docker * Docker Compose ### Step 2 Restore: ```text /srv/secrets/company.env ``` ### Step 3 Restore: ```text /srv/docker ``` data directories. ### Step 4 Restore PostgreSQL and Microsoft SQL Server databases. ### Step 5 Deploy Docker stacks. ### Step 6 Verify: * DNS * SSL certificates * Email delivery * Application access --- ## Related Documentation * docs/server.md * docs/security.md * docs/secrets.md * docs/monitoring.md * postgres/README.md * gitea/README.md * openproject/README.md * mssql/README.md * dbgate/README.md * docs/roadmap.md