# Backups

## Overview

This document describes the backup strategy for SilverLinux.

The goal is to ensure that all critical services can be restored after hardware failure, accidental deletion, corruption, or disaster recovery scenarios.

Current state:

```text
Backup v5.3 production
```

---

## Backup System Status

### Backup v1.4

Implemented:

* HTML backup reports
* Email notification after backup
* Automatic archive creation
* Backup logging
* Restore validation

---

### Backup v1.5

Implemented:

* MSSQL backup fix
* Runtime MSSQL data included correctly
* Restore tests verified MSSQL data is present in backups
* Improved backup structure

---

### Backup v2

Implemented:

* Backup metrics generation
* Prometheus metrics export
* Backup duration metric
* Backup size metric
* Backup success metric
* Automatic low disk space handling
* Automatic deletion of oldest backups when required
* Improved logging

Status:

```text
Operational
```

---

### Backup v5.3

Implemented:

* PostgreSQL backup using `pg_dumpall`
* MSSQL backup using `sqlcmd` and `docker cp`
* Gitea backup
* BaGet backup
* OpenProject backup
* Docker configuration backup
* Final compressed archive
* Lock file protection
* Disk space validation
* Central backup logging
* Structured failure handling
* Prometheus Textfile Collector metrics output
* Backup success and failure tracking
* Duration and size monitoring
* Historical logging

Status:

```text
Production
```

---

## Backup Priorities

### Critical

These items must always be backed up.

#### PostgreSQL

Contains:

* OpenProject database
* Gitea database
* Future application databases

Importance:

Critical

---

#### Microsoft SQL Server

Data location:

```text
/srv/docker/mssql/data
```

Required:

* Native SQL Server database backups
* MSSQL Compose configuration
* `MSSQL_SA_PASSWORD` from Infisical recovery data
* Runtime MSSQL data included by the v1.5+ backup structure

Importance:

Critical

Restore validation has verified that MSSQL data is present in current backup archives.

---

#### Gitea Data

Location:

```text
/srv/docker/gitea/data
```

Contains:

* Git repositories
* Attachments
* Configuration
* User data

Importance:

Critical

---

#### OpenProject Assets

Location:

```text
/srv/docker/openproject/assets
```

Contains:

* Attachments
* Uploaded files
* User generated content

Importance:

Critical

---

#### Shared Secrets

Source:

```text
Infisical
```

Contains:

* SMTP credentials
* PostgreSQL passwords
* MSSQL SA password
* OpenProject secrets
* DbGate password
* Future OAuth secrets

Importance:

Critical

Without Infisical recovery and `/etc/infisical/token` reauthorization, applications may not start correctly.

Legacy note:

* `/srv/secrets/company.env` is no longer part of the active secret model and should be removed if it still exists.

---

#### Nextcloud

Volumes:

```text
nextcloud_nextcloud_data
nextcloud_nextcloud_db
```

Compose location:

```text
/srv/docker/nextcloud
```

Contains:

* Uploaded files and user data
* Nextcloud application configuration
* Installed apps and themes
* PostgreSQL metadata database
* Compose configuration and environment references

Required:

* Logical PostgreSQL dump from `nextcloud-db`
* Archive of `nextcloud_nextcloud_data`
* Archive or cold snapshot of `nextcloud_nextcloud_db`
* `/srv/docker/nextcloud/docker-compose.yml`
* `/srv/docker/nextcloud/.env` only if it contains non-secret runtime configuration
* Infisical `/nextcloud` secrets

Importance:

Critical

The 2026-07-08 runtime snapshot verified `nextcloud_nextcloud_data` and `nextcloud_nextcloud_db` as Docker-managed named volumes.

---

#### Sentry

Deployment model:

```text
Upstream Sentry self-hosted install.sh
```

Contains:

* `/srv/docker/sentry`
* Generated Sentry configuration
* PostgreSQL metadata
* ClickHouse event data
* Redis cache and coordination data
* Kafka data if persistence is enabled
* SeaweedFS attachments and object storage

Required:

* Backup of `/srv/docker/sentry`
* Backup of Sentry generated config and secret files
* Backup of persistent Sentry Docker volumes
* Nginx Proxy Manager route and certificate state for `sentry.silveressence.net`

Importance:

Critical

Sentry is not a normal SilverLinux `slcompose` service. Preserve its upstream self-hosted installer layout and restore it as one deployment unit.

Verified configuration files:

```text
/srv/docker/sentry/.env
/srv/docker/sentry/docker-compose.yml
/srv/docker/sentry/sentry/sentry.conf.py
```

---

#### Nginx Proxy Manager

Locations:

```text
/srv/docker/nginx-proxy-manager/data
/srv/docker/nginx-proxy-manager/letsencrypt
```

Contains:

* Proxy configuration
* SSL certificates
* Domain routing

Importance:

Critical

---

### Medium Priority

#### Portainer

Volume:

```text
portainer_portainer_data
```

Contains:

* Portainer users
* Portainer configuration
* Stack definitions

Importance:

Medium

Can be recreated if necessary.

---

#### Jitsi Configuration

Location:

```text
/srv/docker/jitsi
```

Contains:

* Jitsi configuration
* Internal authentication configuration

Importance:

Medium

---

#### BaGet Data

Location:

```text
/srv/docker/baget/data
```

Contains:

* Hosted NuGet packages
* SQLite database

Importance:

Medium

---

#### Gitea Actions Runner Configuration

Location:

```text
/srv/docker/gitea-runner
```

Contains:

* Runner deployment configuration
* Runner state

The registration token is stored in Infisical.

Importance:

Medium

---

#### DbGate Data

Location:

```text
/srv/docker/dbgate/data
```

Contains:

* Saved connections
* DbGate application state

Also retain the Nginx Proxy Manager route and DbGate authentication credential.

Importance:

Medium

---

#### Xray Configuration

Location:

```text
/srv/docker/xray
```

Contains:

* Xray Docker Compose configuration
* Xray runtime configuration
* Endpoint credentials, UUIDs or keys if stored with the service

Importance:

Medium

Xray credentials and keys must be backed up securely and must never be committed to Git.

---

## Backup Storage

Current backup capabilities:

* Automatic archive generation
* HTML backup report generation
* Backup log generation
* Email report delivery
* Prometheus metrics export
* Automatic cleanup when disk space is low
* Automatic deletion of oldest backups when required

---

Current Location:

```text
/srv/backups
```

Suggested Structure:

```text
/srv/backups/
├── daily
├── weekly
└── monthly
```

---

## Retention Policy

Current cleanup behavior:

* Backup v5.3 monitors available disk space.
* If disk space is low, the oldest backups are deleted automatically.
* Cleanup is logged.

---

### Daily

Keep:

```text
7 days
```

---

## Nextcloud Backup Recipe

This recipe shows practical commands to back up the operational Nextcloud PostgreSQL database and application data volume. Run these on the host where Docker runs. Adjust paths, filenames and the backup target directory as needed.

1) Backup PostgreSQL (from running `nextcloud-db` container). This creates a compressed SQL dump:

```bash
# Backup directory on host
BACKUP_DIR=/srv/backups/nextcloud
mkdir -p "$BACKUP_DIR"
docker exec -t nextcloud-db pg_dump -U nextcloud nextcloud | gzip > "$BACKUP_DIR/nextcloud_db_$(date +%F).sql.gz"
```

If the DB user or DB name differ from the example, use the values from the Nextcloud Infisical path or sanitized Compose config. For a full cluster dump, use `pg_dumpall` with the appropriate PostgreSQL user.

2) Backup application data volume (`nextcloud_nextcloud_data`):

```bash
BACKUP_DIR=/srv/backups/nextcloud
mkdir -p "$BACKUP_DIR"
docker run --rm -v nextcloud_nextcloud_data:/data -v "$BACKUP_DIR":/backup alpine \
	sh -c "cd /data && tar czf /backup/nextcloud_data_$(date +%F).tar.gz ."
```

3) Backup `docker-compose.yml` and `.env` if the `.env` file contains only non-secret runtime configuration:

```bash
cp /srv/docker/nextcloud/docker-compose.yml "$BACKUP_DIR/docker-compose.yml.$(date +%F)"
if [ -f /srv/docker/nextcloud/.env ]; then
  cp /srv/docker/nextcloud/.env "$BACKUP_DIR/.env.$(date +%F)"
fi
```

Do not preserve plaintext passwords from `.env`; migrate them to Infisical and remove them from the file.

4) Optional: Export Postgres data directory snapshot (cold snapshot required — stop DB or use filesystem snapshot):

```bash
# Stop DB to take a consistent file-level snapshot (or use LVM/ZFS snapshot instead)
docker compose -f /srv/docker/nextcloud/docker-compose.yml stop nextcloud-db
tar czf "$BACKUP_DIR/nextcloud_db_files_$(date +%F).tar.gz" -C /var/lib/docker/volumes/nextcloud_nextcloud_db/_data .
docker compose -f /srv/docker/nextcloud/docker-compose.yml start nextcloud-db
```

5) Retention and verification

- Keep backups in `/srv/backups/nextcloud` with rolling retention (e.g., daily 7, weekly 4, monthly 6).
- Verify SQL dumps by restoring to a staging DB and checking the Nextcloud application with `occ status` and a login test.

Security note: backups contain secrets (DB passwords stored in config, mail credentials in `config.php`). Protect backups with appropriate filesystem permissions and store them encrypted if possible.


### Weekly

Keep:

```text
4 weeks
```

### Monthly

Keep:

```text
12 months
```

---

## Restore Priorities

Restore order:

1. Shared secrets
2. PostgreSQL
3. Microsoft SQL Server
4. Nginx Proxy Manager
5. Gitea
6. OpenProject
7. Portainer
8. Jitsi
9. BaGet
10. Gitea Actions Runner
11. DbGate
12. Xray
13. Nextcloud
14. Sentry

---

## Restore Validation

Restore procedure has been validated.

Completed:

* Dry-run restore
* Full archive validation
* Archive extraction

Verified:

* PostgreSQL
* Microsoft SQL Server
* Gitea
* BaGet
* OpenProject assets
* Docker configuration

Nextcloud restore validation is pending after its 2026-07-08 operational documentation update.

Sentry restore validation is pending after its upstream `install.sh` deployment documentation update.

Validation results:

* MSSQL backup integrity verified
* PostgreSQL backup verified
* Restore archive extraction verified

---

## Monitoring Integration

Backup v5.3 exports metrics for Prometheus.

Current metrics:

* Backup success
* Backup duration
* Backup size
* Backup timestamp

Metric names:

* `silverlinux_backup_success`
* `silverlinux_backup_duration_seconds`
* `silverlinux_backup_size_bytes`
* `silverlinux_backup_timestamp_seconds`

Export path:

```text
Node Exporter Textfile Collector
```

Prometheus target status:

```text
Validated
```

---

## Automation

Current automation:

* Daily backup execution
* Root cron configuration
* Backup wrapper command
* Email report
* HTML report
* Automatic cleanup
* Automatic archive generation

Status:

```text
Operational
```

---

## Email Notifications

Postfix is configured to relay through Gmail SMTP.

Implemented:

* Gmail relay
* App Password authentication
* TLS encryption
* Successful email delivery
* Backup success emails
* Backup report emails

Status:

```text
Operational
```

---

## Disaster Recovery Checklist

### Step 1

Install:

* Ubuntu Server
* Docker
* Docker Compose

### Step 2

Restore or reauthorize:

```text
Infisical access and service secret paths
```

### Step 3

Restore:

```text
/srv/docker
```

data directories.

### Step 4

Restore PostgreSQL and Microsoft SQL Server databases.

### Step 5

Deploy Docker stacks.

### Step 6

Verify:

* DNS
* SSL certificates
* Email delivery
* Application access

---

## Related Documentation

* docs/server.md
* docs/security.md
* docs/secrets.md
* docs/monitoring.md
* postgres/README.md
* gitea/README.md
* openproject/README.md
* mssql/README.md
* dbgate/README.md
* docs/roadmap.md