592 lines
7.0 KiB
Markdown
592 lines
7.0 KiB
Markdown
# Backups
|
|
|
|
## Overview
|
|
|
|
This document describes the backup strategy for SilverLinux.
|
|
|
|
The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios.
|
|
|
|
Current state:
|
|
|
|
```text
|
|
Backup v5.3 production
|
|
```
|
|
|
|
---
|
|
|
|
## Backup System Status
|
|
|
|
### Backup v1.4
|
|
|
|
Implemented:
|
|
|
|
* HTML backup reports
|
|
* Email notification after backup
|
|
* Automatic archive creation
|
|
* Backup logging
|
|
* Restore validation
|
|
|
|
---
|
|
|
|
### Backup v1.5
|
|
|
|
Implemented:
|
|
|
|
* MSSQL backup fix
|
|
* Runtime MSSQL data included correctly
|
|
* Restore tests verified MSSQL data is present in backups
|
|
* Improved backup structure
|
|
|
|
---
|
|
|
|
### Backup v2
|
|
|
|
Implemented:
|
|
|
|
* Backup metrics generation
|
|
* Prometheus metrics export
|
|
* Backup duration metric
|
|
* Backup size metric
|
|
* Backup success metric
|
|
* Automatic low disk space handling
|
|
* Automatic deletion of oldest backups when required
|
|
* Improved logging
|
|
|
|
Status:
|
|
|
|
```text
|
|
Operational
|
|
```
|
|
|
|
---
|
|
|
|
### Backup v5.3
|
|
|
|
Implemented:
|
|
|
|
* PostgreSQL backup using `pg_dumpall`
|
|
* MSSQL backup using `sqlcmd` and `docker cp`
|
|
* Gitea backup
|
|
* BaGet backup
|
|
* OpenProject backup
|
|
* Docker configuration backup
|
|
* Final compressed archive
|
|
* Lock file protection
|
|
* Disk space validation
|
|
* Central backup logging
|
|
* Structured failure handling
|
|
* Prometheus Textfile Collector metrics output
|
|
* Backup success and failure tracking
|
|
* Duration and size monitoring
|
|
* Historical logging
|
|
|
|
Status:
|
|
|
|
```text
|
|
Production
|
|
```
|
|
|
|
---
|
|
|
|
## Backup Priorities
|
|
|
|
### Critical
|
|
|
|
These items must always be backed up.
|
|
|
|
#### PostgreSQL
|
|
|
|
Contains:
|
|
|
|
* OpenProject database
|
|
* Gitea database
|
|
* Future application databases
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
#### Microsoft SQL Server
|
|
|
|
Data location:
|
|
|
|
```text
|
|
/srv/docker/mssql/data
|
|
```
|
|
|
|
Required:
|
|
|
|
* Native SQL Server database backups
|
|
* MSSQL Compose configuration
|
|
* `MSSQL_SA_PASSWORD` from the shared secrets backup
|
|
* Runtime MSSQL data included by the v1.5+ backup structure
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
Restore validation has verified that MSSQL data is present in current backup archives.
|
|
|
|
---
|
|
|
|
#### Gitea Data
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/gitea/data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Git repositories
|
|
* Attachments
|
|
* Configuration
|
|
* User data
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
#### OpenProject Assets
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/openproject/assets
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Attachments
|
|
* Uploaded files
|
|
* User generated content
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
#### Shared Secrets
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/secrets/company.env
|
|
```
|
|
|
|
Contains:
|
|
|
|
* SMTP credentials
|
|
* PostgreSQL passwords
|
|
* MSSQL SA password
|
|
* OpenProject secrets
|
|
* Future DbGate password
|
|
* Future OAuth secrets
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
Without this file applications may not start correctly.
|
|
|
|
---
|
|
|
|
#### Nginx Proxy Manager
|
|
|
|
Locations:
|
|
|
|
```text
|
|
/srv/docker/nginx-proxy-manager/data
|
|
/srv/docker/nginx-proxy-manager/letsencrypt
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Proxy configuration
|
|
* SSL certificates
|
|
* Domain routing
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
### Medium Priority
|
|
|
|
#### Portainer
|
|
|
|
Volume:
|
|
|
|
```text
|
|
portainer_portainer_data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Portainer users
|
|
* Portainer configuration
|
|
* Stack definitions
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
Can be recreated if necessary.
|
|
|
|
---
|
|
|
|
#### Jitsi Configuration
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/jitsi
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Jitsi configuration
|
|
* Internal authentication configuration
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### BaGet Data
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/baget/data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Hosted NuGet packages
|
|
* SQLite database
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### Gitea Actions Runner Configuration
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/gitea-runner
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Runner deployment configuration
|
|
* Runner state
|
|
|
|
The registration token is stored separately in `/srv/secrets/company.env`.
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### DbGate Data
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/dbgate/data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Saved connections
|
|
* DbGate application state
|
|
|
|
Also retain the Nginx Proxy Manager route and DbGate authentication credential.
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### Xray Configuration
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/xray
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Xray Docker Compose configuration
|
|
* Xray runtime configuration
|
|
* Endpoint credentials, UUIDs or keys if stored with the service
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
Xray credentials and keys must be backed up securely and must never be committed to Git.
|
|
|
|
---
|
|
|
|
## Backup Storage
|
|
|
|
Current backup capabilities:
|
|
|
|
* Automatic archive generation
|
|
* HTML backup report generation
|
|
* Backup log generation
|
|
* Email report delivery
|
|
* Prometheus metrics export
|
|
* Automatic cleanup when disk space is low
|
|
* Automatic deletion of oldest backups when required
|
|
|
|
---
|
|
|
|
Current Location:
|
|
|
|
```text
|
|
/srv/backups
|
|
```
|
|
|
|
Suggested Structure:
|
|
|
|
```text
|
|
/srv/backups/
|
|
├── daily
|
|
├── weekly
|
|
└── monthly
|
|
```
|
|
|
|
---
|
|
|
|
## Retention Policy
|
|
|
|
Current cleanup behavior:
|
|
|
|
* Backup v5.3 monitors available disk space.
|
|
* If disk space is low, the oldest backups are deleted automatically.
|
|
* Cleanup is logged.
|
|
|
|
---
|
|
|
|
### Daily
|
|
|
|
Keep:
|
|
|
|
```text
|
|
7 days
|
|
```
|
|
|
|
### Weekly
|
|
|
|
Keep:
|
|
|
|
```text
|
|
4 weeks
|
|
```
|
|
|
|
### Monthly
|
|
|
|
Keep:
|
|
|
|
```text
|
|
12 months
|
|
```
|
|
|
|
---
|
|
|
|
## Restore Priorities
|
|
|
|
Restore order:
|
|
|
|
1. Shared secrets
|
|
2. PostgreSQL
|
|
3. Microsoft SQL Server
|
|
4. Nginx Proxy Manager
|
|
5. Gitea
|
|
6. OpenProject
|
|
7. Portainer
|
|
8. Jitsi
|
|
9. BaGet
|
|
10. Gitea Actions Runner
|
|
11. DbGate
|
|
12. Xray
|
|
|
|
---
|
|
|
|
## Restore Validation
|
|
|
|
Restore procedure has been validated.
|
|
|
|
Completed:
|
|
|
|
* Dry-run restore
|
|
* Full archive validation
|
|
* Archive extraction
|
|
|
|
Verified:
|
|
|
|
* PostgreSQL
|
|
* Microsoft SQL Server
|
|
* Gitea
|
|
* BaGet
|
|
* OpenProject assets
|
|
* Docker configuration
|
|
|
|
Validation results:
|
|
|
|
* MSSQL backup integrity verified
|
|
* PostgreSQL backup verified
|
|
* Restore archive extraction verified
|
|
|
|
---
|
|
|
|
## Monitoring Integration
|
|
|
|
Backup v5.3 exports metrics for Prometheus.
|
|
|
|
Current metrics:
|
|
|
|
* Backup success
|
|
* Backup duration
|
|
* Backup size
|
|
* Backup timestamp
|
|
|
|
Metric names:
|
|
|
|
* `silverlinux_backup_success`
|
|
* `silverlinux_backup_duration_seconds`
|
|
* `silverlinux_backup_size_bytes`
|
|
* `silverlinux_backup_timestamp_seconds`
|
|
|
|
Export path:
|
|
|
|
```text
|
|
Node Exporter Textfile Collector
|
|
```
|
|
|
|
Prometheus target status:
|
|
|
|
```text
|
|
Validated
|
|
```
|
|
|
|
---
|
|
|
|
## Automation
|
|
|
|
Current automation:
|
|
|
|
* Daily backup execution
|
|
* Root cron configuration
|
|
* Backup wrapper command
|
|
* Email report
|
|
* HTML report
|
|
* Automatic cleanup
|
|
* Automatic archive generation
|
|
|
|
Status:
|
|
|
|
```text
|
|
Operational
|
|
```
|
|
|
|
---
|
|
|
|
## Email Notifications
|
|
|
|
Postfix is configured to relay through Gmail SMTP.
|
|
|
|
Implemented:
|
|
|
|
* Gmail relay
|
|
* App Password authentication
|
|
* TLS encryption
|
|
* Successful email delivery
|
|
* Backup success emails
|
|
* Backup report emails
|
|
|
|
Status:
|
|
|
|
```text
|
|
Operational
|
|
```
|
|
|
|
---
|
|
|
|
## Disaster Recovery Checklist
|
|
|
|
### Step 1
|
|
|
|
Install:
|
|
|
|
* Ubuntu Server
|
|
* Docker
|
|
* Docker Compose
|
|
|
|
### Step 2
|
|
|
|
Restore:
|
|
|
|
```text
|
|
/srv/secrets/company.env
|
|
```
|
|
|
|
### Step 3
|
|
|
|
Restore:
|
|
|
|
```text
|
|
/srv/docker
|
|
```
|
|
|
|
data directories.
|
|
|
|
### Step 4
|
|
|
|
Restore PostgreSQL and Microsoft SQL Server databases.
|
|
|
|
### Step 5
|
|
|
|
Deploy Docker stacks.
|
|
|
|
### Step 6
|
|
|
|
Verify:
|
|
|
|
* DNS
|
|
* SSL certificates
|
|
* Email delivery
|
|
* Application access
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
* docs/server.md
|
|
* docs/security.md
|
|
* docs/secrets.md
|
|
* docs/monitoring.md
|
|
* postgres/README.md
|
|
* gitea/README.md
|
|
* openproject/README.md
|
|
* mssql/README.md
|
|
* dbgate/README.md
|
|
* docs/roadmap.md
|