737 lines
11 KiB
Markdown
737 lines
11 KiB
Markdown
# Backups
|
|
|
|
## Overview
|
|
|
|
This document describes the backup strategy for SilverLinux.
|
|
|
|
The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios.
|
|
|
|
Current state:
|
|
|
|
```text
|
|
Backup v5.3 production
|
|
```
|
|
|
|
---
|
|
|
|
## Backup System Status
|
|
|
|
### Backup v1.4
|
|
|
|
Implemented:
|
|
|
|
* HTML backup reports
|
|
* Email notification after backup
|
|
* Automatic archive creation
|
|
* Backup logging
|
|
* Restore validation
|
|
|
|
---
|
|
|
|
### Backup v1.5
|
|
|
|
Implemented:
|
|
|
|
* MSSQL backup fix
|
|
* Runtime MSSQL data included correctly
|
|
* Restore tests verified MSSQL data is present in backups
|
|
* Improved backup structure
|
|
|
|
---
|
|
|
|
### Backup v2
|
|
|
|
Implemented:
|
|
|
|
* Backup metrics generation
|
|
* Prometheus metrics export
|
|
* Backup duration metric
|
|
* Backup size metric
|
|
* Backup success metric
|
|
* Automatic low disk space handling
|
|
* Automatic deletion of oldest backups when required
|
|
* Improved logging
|
|
|
|
Status:
|
|
|
|
```text
|
|
Operational
|
|
```
|
|
|
|
---
|
|
|
|
### Backup v5.3
|
|
|
|
Implemented:
|
|
|
|
* PostgreSQL backup using `pg_dumpall`
|
|
* MSSQL backup using `sqlcmd` and `docker cp`
|
|
* Gitea backup
|
|
* BaGet backup
|
|
* OpenProject backup
|
|
* Docker configuration backup
|
|
* Final compressed archive
|
|
* Lock file protection
|
|
* Disk space validation
|
|
* Central backup logging
|
|
* Structured failure handling
|
|
* Prometheus Textfile Collector metrics output
|
|
* Backup success and failure tracking
|
|
* Duration and size monitoring
|
|
* Historical logging
|
|
|
|
Status:
|
|
|
|
```text
|
|
Production
|
|
```
|
|
|
|
---
|
|
|
|
## Backup Priorities
|
|
|
|
### Critical
|
|
|
|
These items must always be backed up.
|
|
|
|
#### PostgreSQL
|
|
|
|
Contains:
|
|
|
|
* OpenProject database
|
|
* Gitea database
|
|
* Future application databases
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
#### Microsoft SQL Server
|
|
|
|
Data location:
|
|
|
|
```text
|
|
/srv/docker/mssql/data
|
|
```
|
|
|
|
Required:
|
|
|
|
* Native SQL Server database backups
|
|
* MSSQL Compose configuration
|
|
* `MSSQL_SA_PASSWORD` from Infisical recovery data
|
|
* Runtime MSSQL data included by the v1.5+ backup structure
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
Restore validation has verified that MSSQL data is present in current backup archives.
|
|
|
|
---
|
|
|
|
#### Gitea Data
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/gitea/data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Git repositories
|
|
* Attachments
|
|
* Configuration
|
|
* User data
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
#### OpenProject Assets
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/openproject/assets
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Attachments
|
|
* Uploaded files
|
|
* User generated content
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
#### Shared Secrets
|
|
|
|
Source:
|
|
|
|
```text
|
|
Infisical
|
|
```
|
|
|
|
Contains:
|
|
|
|
* SMTP credentials
|
|
* PostgreSQL passwords
|
|
* MSSQL SA password
|
|
* OpenProject secrets
|
|
* DbGate password
|
|
* Future OAuth secrets
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
Without Infisical recovery and `/etc/infisical/token` reauthorization, applications may not start correctly.
|
|
|
|
Legacy note:
|
|
|
|
* `/srv/secrets/company.env` is no longer part of the active secret model and should be removed if it still exists.
|
|
|
|
---
|
|
|
|
#### Nextcloud
|
|
|
|
Volumes:
|
|
|
|
```text
|
|
nextcloud_nextcloud_data
|
|
nextcloud_nextcloud_db
|
|
```
|
|
|
|
Compose location:
|
|
|
|
```text
|
|
/srv/docker/nextcloud
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Uploaded files and user data
|
|
* Nextcloud application configuration
|
|
* Installed apps and themes
|
|
* PostgreSQL metadata database
|
|
* Compose configuration and environment references
|
|
|
|
Required:
|
|
|
|
* Logical PostgreSQL dump from `nextcloud-db`
|
|
* Archive of `nextcloud_nextcloud_data`
|
|
* Archive or cold snapshot of `nextcloud_nextcloud_db`
|
|
* `/srv/docker/nextcloud/docker-compose.yml`
|
|
* `/srv/docker/nextcloud/.env` only if it contains non-secret runtime configuration
|
|
* Infisical `/nextcloud` secrets
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
The 2026-07-08 runtime snapshot verified `nextcloud_nextcloud_data` and `nextcloud_nextcloud_db` as Docker-managed named volumes.
|
|
|
|
---
|
|
|
|
#### Sentry
|
|
|
|
Deployment model:
|
|
|
|
```text
|
|
Upstream Sentry self-hosted install.sh
|
|
```
|
|
|
|
Contains:
|
|
|
|
* `/srv/docker/sentry`
|
|
* Generated Sentry configuration
|
|
* PostgreSQL metadata
|
|
* ClickHouse event data
|
|
* Redis cache and coordination data
|
|
* Kafka data if persistence is enabled
|
|
* SeaweedFS attachments and object storage
|
|
|
|
Required:
|
|
|
|
* Backup of `/srv/docker/sentry`
|
|
* Backup of Sentry generated config and secret files
|
|
* Backup of persistent Sentry Docker volumes
|
|
* Nginx Proxy Manager route and certificate state for `sentry.silveressence.net`
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
Sentry is not a normal SilverLinux `slcompose` service. Preserve its upstream self-hosted installer layout and restore it as one deployment unit.
|
|
|
|
Verified configuration files:
|
|
|
|
```text
|
|
/srv/docker/sentry/.env
|
|
/srv/docker/sentry/docker-compose.yml
|
|
/srv/docker/sentry/sentry/sentry.conf.py
|
|
```
|
|
|
|
---
|
|
|
|
#### Nginx Proxy Manager
|
|
|
|
Locations:
|
|
|
|
```text
|
|
/srv/docker/nginx-proxy-manager/data
|
|
/srv/docker/nginx-proxy-manager/letsencrypt
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Proxy configuration
|
|
* SSL certificates
|
|
* Domain routing
|
|
|
|
Importance:
|
|
|
|
Critical
|
|
|
|
---
|
|
|
|
### Medium Priority
|
|
|
|
#### Portainer
|
|
|
|
Volume:
|
|
|
|
```text
|
|
portainer_portainer_data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Portainer users
|
|
* Portainer configuration
|
|
* Stack definitions
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
Can be recreated if necessary.
|
|
|
|
---
|
|
|
|
#### Jitsi Configuration
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/jitsi
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Jitsi configuration
|
|
* Internal authentication configuration
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### BaGet Data
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/baget/data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Hosted NuGet packages
|
|
* SQLite database
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### Gitea Actions Runner Configuration
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/gitea-runner
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Runner deployment configuration
|
|
* Runner state
|
|
|
|
The registration token is stored in Infisical.
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### DbGate Data
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/dbgate/data
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Saved connections
|
|
* DbGate application state
|
|
|
|
Also retain the Nginx Proxy Manager route and DbGate authentication credential.
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
---
|
|
|
|
#### Xray Configuration
|
|
|
|
Location:
|
|
|
|
```text
|
|
/srv/docker/xray
|
|
```
|
|
|
|
Contains:
|
|
|
|
* Xray Docker Compose configuration
|
|
* Xray runtime configuration
|
|
* Endpoint credentials, UUIDs or keys if stored with the service
|
|
|
|
Importance:
|
|
|
|
Medium
|
|
|
|
Xray credentials and keys must be backed up securely and must never be committed to Git.
|
|
|
|
---
|
|
|
|
## Backup Storage
|
|
|
|
Current backup capabilities:
|
|
|
|
* Automatic archive generation
|
|
* HTML backup report generation
|
|
* Backup log generation
|
|
* Email report delivery
|
|
* Prometheus metrics export
|
|
* Automatic cleanup when disk space is low
|
|
* Automatic deletion of oldest backups when required
|
|
|
|
---
|
|
|
|
Current Location:
|
|
|
|
```text
|
|
/srv/backups
|
|
```
|
|
|
|
Suggested Structure:
|
|
|
|
```text
|
|
/srv/backups/
|
|
├── daily
|
|
├── weekly
|
|
└── monthly
|
|
```
|
|
|
|
---
|
|
|
|
## Retention Policy
|
|
|
|
Current cleanup behavior:
|
|
|
|
* Backup v5.3 monitors available disk space.
|
|
* If disk space is low, the oldest backups are deleted automatically.
|
|
* Cleanup is logged.
|
|
|
|
---
|
|
|
|
### Daily
|
|
|
|
Keep:
|
|
|
|
```text
|
|
7 days
|
|
```
|
|
|
|
---
|
|
|
|
## Nextcloud Backup Recipe
|
|
|
|
This recipe shows practical commands to back up the operational Nextcloud PostgreSQL database and application data volume. Run these on the host where Docker runs. Adjust paths, filenames and the backup target directory as needed.
|
|
|
|
1) Backup PostgreSQL (from running `nextcloud-db` container). This creates a compressed SQL dump:
|
|
|
|
```bash
|
|
# Backup directory on host
|
|
BACKUP_DIR=/srv/backups/nextcloud
|
|
mkdir -p "$BACKUP_DIR"
|
|
docker exec -t nextcloud-db pg_dump -U nextcloud nextcloud | gzip > "$BACKUP_DIR/nextcloud_db_$(date +%F).sql.gz"
|
|
```
|
|
|
|
If the DB user or DB name differ from the example, use the values from the Nextcloud Infisical path or sanitized Compose config. For a full cluster dump, use `pg_dumpall` with the appropriate PostgreSQL user.
|
|
|
|
2) Backup application data volume (`nextcloud_nextcloud_data`):
|
|
|
|
```bash
|
|
BACKUP_DIR=/srv/backups/nextcloud
|
|
mkdir -p "$BACKUP_DIR"
|
|
docker run --rm -v nextcloud_nextcloud_data:/data -v "$BACKUP_DIR":/backup alpine \
|
|
sh -c "cd /data && tar czf /backup/nextcloud_data_$(date +%F).tar.gz ."
|
|
```
|
|
|
|
3) Backup `docker-compose.yml` and `.env` if the `.env` file contains only non-secret runtime configuration:
|
|
|
|
```bash
|
|
cp /srv/docker/nextcloud/docker-compose.yml "$BACKUP_DIR/docker-compose.yml.$(date +%F)"
|
|
if [ -f /srv/docker/nextcloud/.env ]; then
|
|
cp /srv/docker/nextcloud/.env "$BACKUP_DIR/.env.$(date +%F)"
|
|
fi
|
|
```
|
|
|
|
Do not preserve plaintext passwords from `.env`; migrate them to Infisical and remove them from the file.
|
|
|
|
4) Optional: Export Postgres data directory snapshot (cold snapshot required — stop DB or use filesystem snapshot):
|
|
|
|
```bash
|
|
# Stop DB to take a consistent file-level snapshot (or use LVM/ZFS snapshot instead)
|
|
docker compose -f /srv/docker/nextcloud/docker-compose.yml stop nextcloud-db
|
|
tar czf "$BACKUP_DIR/nextcloud_db_files_$(date +%F).tar.gz" -C /var/lib/docker/volumes/nextcloud_nextcloud_db/_data .
|
|
docker compose -f /srv/docker/nextcloud/docker-compose.yml start nextcloud-db
|
|
```
|
|
|
|
5) Retention and verification
|
|
|
|
- Keep backups in `/srv/backups/nextcloud` with rolling retention (e.g., daily 7, weekly 4, monthly 6).
|
|
- Verify SQL dumps by restoring to a staging DB and checking the Nextcloud application with `occ status` and a login test.
|
|
|
|
Security note: backups contain secrets (DB passwords stored in config, mail credentials in `config.php`). Protect backups with appropriate filesystem permissions and store them encrypted if possible.
|
|
|
|
|
|
### Weekly
|
|
|
|
Keep:
|
|
|
|
```text
|
|
4 weeks
|
|
```
|
|
|
|
### Monthly
|
|
|
|
Keep:
|
|
|
|
```text
|
|
12 months
|
|
```
|
|
|
|
---
|
|
|
|
## Restore Priorities
|
|
|
|
Restore order:
|
|
|
|
1. Shared secrets
|
|
2. PostgreSQL
|
|
3. Microsoft SQL Server
|
|
4. Nginx Proxy Manager
|
|
5. Gitea
|
|
6. OpenProject
|
|
7. Portainer
|
|
8. Jitsi
|
|
9. BaGet
|
|
10. Gitea Actions Runner
|
|
11. DbGate
|
|
12. Xray
|
|
13. Nextcloud
|
|
14. Sentry
|
|
|
|
---
|
|
|
|
## Restore Validation
|
|
|
|
Restore procedure has been validated.
|
|
|
|
Completed:
|
|
|
|
* Dry-run restore
|
|
* Full archive validation
|
|
* Archive extraction
|
|
|
|
Verified:
|
|
|
|
* PostgreSQL
|
|
* Microsoft SQL Server
|
|
* Gitea
|
|
* BaGet
|
|
* OpenProject assets
|
|
* Docker configuration
|
|
|
|
Nextcloud restore validation is pending after its 2026-07-08 operational documentation update.
|
|
|
|
Sentry restore validation is pending after its upstream `install.sh` deployment documentation update.
|
|
|
|
Validation results:
|
|
|
|
* MSSQL backup integrity verified
|
|
* PostgreSQL backup verified
|
|
* Restore archive extraction verified
|
|
|
|
---
|
|
|
|
## Monitoring Integration
|
|
|
|
Backup v5.3 exports metrics for Prometheus.
|
|
|
|
Current metrics:
|
|
|
|
* Backup success
|
|
* Backup duration
|
|
* Backup size
|
|
* Backup timestamp
|
|
|
|
Metric names:
|
|
|
|
* `silverlinux_backup_success`
|
|
* `silverlinux_backup_duration_seconds`
|
|
* `silverlinux_backup_size_bytes`
|
|
* `silverlinux_backup_timestamp_seconds`
|
|
|
|
Export path:
|
|
|
|
```text
|
|
Node Exporter Textfile Collector
|
|
```
|
|
|
|
Prometheus target status:
|
|
|
|
```text
|
|
Validated
|
|
```
|
|
|
|
---
|
|
|
|
## Automation
|
|
|
|
Current automation:
|
|
|
|
* Daily backup execution
|
|
* Root cron configuration
|
|
* Backup wrapper command
|
|
* Email report
|
|
* HTML report
|
|
* Automatic cleanup
|
|
* Automatic archive generation
|
|
|
|
Status:
|
|
|
|
```text
|
|
Operational
|
|
```
|
|
|
|
---
|
|
|
|
## Email Notifications
|
|
|
|
Postfix is configured to relay through Gmail SMTP.
|
|
|
|
Implemented:
|
|
|
|
* Gmail relay
|
|
* App Password authentication
|
|
* TLS encryption
|
|
* Successful email delivery
|
|
* Backup success emails
|
|
* Backup report emails
|
|
|
|
Status:
|
|
|
|
```text
|
|
Operational
|
|
```
|
|
|
|
---
|
|
|
|
## Disaster Recovery Checklist
|
|
|
|
### Step 1
|
|
|
|
Install:
|
|
|
|
* Ubuntu Server
|
|
* Docker
|
|
* Docker Compose
|
|
|
|
### Step 2
|
|
|
|
Restore or reauthorize:
|
|
|
|
```text
|
|
Infisical access and service secret paths
|
|
```
|
|
|
|
### Step 3
|
|
|
|
Restore:
|
|
|
|
```text
|
|
/srv/docker
|
|
```
|
|
|
|
data directories.
|
|
|
|
### Step 4
|
|
|
|
Restore PostgreSQL and Microsoft SQL Server databases.
|
|
|
|
### Step 5
|
|
|
|
Deploy Docker stacks.
|
|
|
|
### Step 6
|
|
|
|
Verify:
|
|
|
|
* DNS
|
|
* SSL certificates
|
|
* Email delivery
|
|
* Application access
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
* docs/server.md
|
|
* docs/security.md
|
|
* docs/secrets.md
|
|
* docs/monitoring.md
|
|
* postgres/README.md
|
|
* gitea/README.md
|
|
* openproject/README.md
|
|
* mssql/README.md
|
|
* dbgate/README.md
|
|
* docs/roadmap.md
|