SilverSolutions/silverlinux-infra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ea638c0eae670dfe343803e1621e7fdc35a7162d
silverlinux-infra/docs/backups.md
T

11 KiB
Raw Blame History

Backups

Overview

This document describes the backup strategy for SilverLinux.

The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios.

Current state:

Backup v5.3 production

Backup System Status

Backup v1.4

Implemented:

  • HTML backup reports
  • Email notification after backup
  • Automatic archive creation
  • Backup logging
  • Restore validation

Backup v1.5

Implemented:

  • MSSQL backup fix
  • Runtime MSSQL data included correctly
  • Restore tests verified MSSQL data is present in backups
  • Improved backup structure

Backup v2

Implemented:

  • Backup metrics generation
  • Prometheus metrics export
  • Backup duration metric
  • Backup size metric
  • Backup success metric
  • Automatic low disk space handling
  • Automatic deletion of oldest backups when required
  • Improved logging

Status:

Operational

Backup v5.3

Implemented:

  • PostgreSQL backup using pg_dumpall
  • MSSQL backup using sqlcmd and docker cp
  • Gitea backup
  • BaGet backup
  • OpenProject backup
  • Docker configuration backup
  • Final compressed archive
  • Lock file protection
  • Disk space validation
  • Central backup logging
  • Structured failure handling
  • Prometheus Textfile Collector metrics output
  • Backup success and failure tracking
  • Duration and size monitoring
  • Historical logging

Status:

Production

Backup Priorities

Critical

These items must always be backed up.

PostgreSQL

Contains:

  • OpenProject database
  • Gitea database
  • Future application databases

Importance:

Critical

Microsoft SQL Server

Data location:

/srv/docker/mssql/data

Required:

  • Native SQL Server database backups
  • MSSQL Compose configuration
  • MSSQL_SA_PASSWORD from Infisical recovery data
  • Runtime MSSQL data included by the v1.5+ backup structure

Importance:

Critical

Restore validation has verified that MSSQL data is present in current backup archives.

Gitea Data

Location:

/srv/docker/gitea/data

Contains:

  • Git repositories
  • Attachments
  • Configuration
  • User data

Importance:

Critical

OpenProject Assets

Location:

/srv/docker/openproject/assets

Contains:

  • Attachments
  • Uploaded files
  • User generated content

Importance:

Critical

Shared Secrets

Source:

Infisical

Contains:

  • SMTP credentials
  • PostgreSQL passwords
  • MSSQL SA password
  • OpenProject secrets
  • DbGate password
  • Future OAuth secrets

Importance:

Critical

Without Infisical recovery and /etc/infisical/token reauthorization, applications may not start correctly.

Legacy note:

  • /srv/secrets/company.env is no longer part of the active secret model and should be removed if it still exists.

Nextcloud

Volumes:

nextcloud_nextcloud_data
nextcloud_nextcloud_db

Compose location:

/srv/docker/nextcloud

Contains:

  • Uploaded files and user data
  • Nextcloud application configuration
  • Installed apps and themes
  • PostgreSQL metadata database
  • Compose configuration and environment references

Required:

  • Logical PostgreSQL dump from nextcloud-db
  • Archive of nextcloud_nextcloud_data
  • Archive or cold snapshot of nextcloud_nextcloud_db
  • /srv/docker/nextcloud/docker-compose.yml
  • /srv/docker/nextcloud/.env only if it contains non-secret runtime configuration
  • Infisical /nextcloud secrets

Importance:

Critical

The 2026-07-08 runtime snapshot verified nextcloud_nextcloud_data and nextcloud_nextcloud_db as Docker-managed named volumes.

Sentry

Deployment model:

Upstream Sentry self-hosted install.sh

Contains:

  • /srv/docker/sentry
  • Generated Sentry configuration
  • PostgreSQL metadata
  • ClickHouse event data
  • Redis cache and coordination data
  • Kafka data if persistence is enabled
  • SeaweedFS attachments and object storage

Required:

  • Backup of /srv/docker/sentry
  • Backup of Sentry generated config and secret files
  • Backup of persistent Sentry Docker volumes
  • Nginx Proxy Manager route and certificate state for sentry.silveressence.net

Importance:

Critical

Sentry is not a normal SilverLinux slcompose service. Preserve its upstream self-hosted installer layout and restore it as one deployment unit.

Verified configuration files:

/srv/docker/sentry/.env
/srv/docker/sentry/docker-compose.yml
/srv/docker/sentry/sentry/sentry.conf.py

Nginx Proxy Manager

Locations:

/srv/docker/nginx-proxy-manager/data
/srv/docker/nginx-proxy-manager/letsencrypt

Contains:

  • Proxy configuration
  • SSL certificates
  • Domain routing

Importance:

Critical

Medium Priority

Portainer

Volume:

portainer_portainer_data

Contains:

  • Portainer users
  • Portainer configuration
  • Stack definitions

Importance:

Medium

Can be recreated if necessary.

Jitsi Configuration

Location:

/srv/docker/jitsi

Contains:

  • Jitsi configuration
  • Internal authentication configuration

Importance:

Medium

BaGet Data

Location:

/srv/docker/baget/data

Contains:

  • Hosted NuGet packages
  • SQLite database

Importance:

Medium

Gitea Actions Runner Configuration

Location:

/srv/docker/gitea-runner

Contains:

  • Runner deployment configuration
  • Runner state

The registration token is stored in Infisical.

Importance:

Medium

DbGate Data

Location:

/srv/docker/dbgate/data

Contains:

  • Saved connections
  • DbGate application state

Also retain the Nginx Proxy Manager route and DbGate authentication credential.

Importance:

Medium

Xray Configuration

Location:

/srv/docker/xray

Contains:

  • Xray Docker Compose configuration
  • Xray runtime configuration
  • Endpoint credentials, UUIDs or keys if stored with the service

Importance:

Medium

Xray credentials and keys must be backed up securely and must never be committed to Git.

Backup Storage

Current backup capabilities:

  • Automatic archive generation
  • HTML backup report generation
  • Backup log generation
  • Email report delivery
  • Prometheus metrics export
  • Automatic cleanup when disk space is low
  • Automatic deletion of oldest backups when required

Current Location:

/srv/backups

Suggested Structure:

/srv/backups/
├── daily
├── weekly
└── monthly

Retention Policy

Current cleanup behavior:

  • Backup v5.3 monitors available disk space.
  • If disk space is low, the oldest backups are deleted automatically.
  • Cleanup is logged.

Daily

Keep:

7 days

Nextcloud Backup Recipe

This recipe shows practical commands to back up the operational Nextcloud PostgreSQL database and application data volume. Run these on the host where Docker runs. Adjust paths, filenames and the backup target directory as needed.

  1. Backup PostgreSQL (from running nextcloud-db container). This creates a compressed SQL dump:
# Backup directory on host
BACKUP_DIR=/srv/backups/nextcloud
mkdir -p "$BACKUP_DIR"
docker exec -t nextcloud-db pg_dump -U nextcloud nextcloud | gzip > "$BACKUP_DIR/nextcloud_db_$(date +%F).sql.gz"

If the DB user or DB name differ from the example, use the values from the Nextcloud Infisical path or sanitized Compose config. For a full cluster dump, use pg_dumpall with the appropriate PostgreSQL user.

  1. Backup application data volume (nextcloud_nextcloud_data):
BACKUP_DIR=/srv/backups/nextcloud
mkdir -p "$BACKUP_DIR"
docker run --rm -v nextcloud_nextcloud_data:/data -v "$BACKUP_DIR":/backup alpine \
	sh -c "cd /data && tar czf /backup/nextcloud_data_$(date +%F).tar.gz ."
  1. Backup docker-compose.yml and .env if the .env file contains only non-secret runtime configuration:
cp /srv/docker/nextcloud/docker-compose.yml "$BACKUP_DIR/docker-compose.yml.$(date +%F)"
if [ -f /srv/docker/nextcloud/.env ]; then
  cp /srv/docker/nextcloud/.env "$BACKUP_DIR/.env.$(date +%F)"
fi

Do not preserve plaintext passwords from .env; migrate them to Infisical and remove them from the file.

  1. Optional: Export Postgres data directory snapshot (cold snapshot required — stop DB or use filesystem snapshot):
# Stop DB to take a consistent file-level snapshot (or use LVM/ZFS snapshot instead)
docker compose -f /srv/docker/nextcloud/docker-compose.yml stop nextcloud-db
tar czf "$BACKUP_DIR/nextcloud_db_files_$(date +%F).tar.gz" -C /var/lib/docker/volumes/nextcloud_nextcloud_db/_data .
docker compose -f /srv/docker/nextcloud/docker-compose.yml start nextcloud-db
  1. Retention and verification
  • Keep backups in /srv/backups/nextcloud with rolling retention (e.g., daily 7, weekly 4, monthly 6).
  • Verify SQL dumps by restoring to a staging DB and checking the Nextcloud application with occ status and a login test.

Security note: backups contain secrets (DB passwords stored in config, mail credentials in config.php). Protect backups with appropriate filesystem permissions and store them encrypted if possible.

Weekly

Keep:

4 weeks

Monthly

Keep:

12 months

Restore Priorities

Restore order:

  1. Shared secrets
  2. PostgreSQL
  3. Microsoft SQL Server
  4. Nginx Proxy Manager
  5. Gitea
  6. OpenProject
  7. Portainer
  8. Jitsi
  9. BaGet
  10. Gitea Actions Runner
  11. DbGate
  12. Xray
  13. Nextcloud
  14. Sentry

Restore Validation

Restore procedure has been validated.

Completed:

  • Dry-run restore
  • Full archive validation
  • Archive extraction

Verified:

  • PostgreSQL
  • Microsoft SQL Server
  • Gitea
  • BaGet
  • OpenProject assets
  • Docker configuration

Nextcloud restore validation is pending after its 2026-07-08 operational documentation update.

Sentry restore validation is pending after its upstream install.sh deployment documentation update.

Validation results:

  • MSSQL backup integrity verified
  • PostgreSQL backup verified
  • Restore archive extraction verified

Monitoring Integration

Backup v5.3 exports metrics for Prometheus.

Current metrics:

  • Backup success
  • Backup duration
  • Backup size
  • Backup timestamp

Metric names:

  • silverlinux_backup_success
  • silverlinux_backup_duration_seconds
  • silverlinux_backup_size_bytes
  • silverlinux_backup_timestamp_seconds

Export path:

Node Exporter Textfile Collector

Prometheus target status:

Validated

Automation

Current automation:

  • Daily backup execution
  • Root cron configuration
  • Backup wrapper command
  • Email report
  • HTML report
  • Automatic cleanup
  • Automatic archive generation

Status:

Operational

Email Notifications

Postfix is configured to relay through Gmail SMTP.

Implemented:

  • Gmail relay
  • App Password authentication
  • TLS encryption
  • Successful email delivery
  • Backup success emails
  • Backup report emails

Status:

Operational

Disaster Recovery Checklist

Step 1

Install:

  • Ubuntu Server
  • Docker
  • Docker Compose

Step 2

Restore or reauthorize:

Infisical access and service secret paths

Step 3

Restore:

/srv/docker

data directories.

Step 4

Restore PostgreSQL and Microsoft SQL Server databases.

Step 5

Deploy Docker stacks.

Step 6

Verify:

  • DNS
  • SSL certificates
  • Email delivery
  • Application access

Related Documentation

  • docs/server.md
  • docs/security.md
  • docs/secrets.md
  • docs/monitoring.md
  • postgres/README.md
  • gitea/README.md
  • openproject/README.md
  • mssql/README.md
  • dbgate/README.md
  • docs/roadmap.md
Reference in New Issue View Git Blame Copy Permalink