Backup and Recovery

The backup strategy has three tiers: hourly local dumps on Caroline, daily rsync to Companion-Cube (the Synology NAS), and encrypted offsite to Cloudflare R2. An untested backup is a hypothesis. A tested restore procedure exists; run it.

Backup Tiers

Tier 1: Hourly Local (Caroline)

scripts/backup-hourly.sh runs on Caroline as a systemd timer. It produces a timestamped pg_dump of the primary database and includes WAL cleanup.

It also handles the Authelia users.yml backup (Phase 1b). This file is excluded from GitOps checkout because Authelia rehashes passwords on login, making any version restored from git invalid. The hourly backup is the only reliable source for users.yml recovery.

Output directory: ~/backups/ on Caroline.

Tier 2: Daily NAS Sync

scripts/maintenance/daily-backup.sh and scripts/backup/rsync-to-nas.sh rsync backup data from Caroline to Companion-Cube (the Synology NAS) for a second physical copy.

Tier 3: Offsite R2

scripts/backup/push-offsite.sh encrypts and pushes database dumps to Cloudflare R2 storage. This provides geographic redundancy and a recovery point independent of the local network.

scripts/backup/dr-drill.sh runs restore drills against the R2 backups to verify they are actually recoverable. Results are logged to dr_drill_log.

Authelia Users File

Caution

authelia/users.yml cannot be restored from git. Authelia rehashes passwords on every login. A git-restored version will have the hash from the last commit, not the current hash, and will reject all login attempts.

Recovery source: the most recent hourly backup from ~/backups/ on Caroline.

The file is excluded from GitOps checkout via the pre-pull hook in gitops-converge.sh. If a git reset --hard accidentally overwrites it, the hook restores the backup automatically. If that fails, restore manually from the hourly backup directory.

Recovery Procedures

Postgres Recovery

1. SSH to Caroline
2. Stop dependent containers: docker compose stop n8n authelia grafana dashboard-api mcp-proxy-postgres mcp-proxy-memory discord-bot
3. Restore the database from the backup dump using pg_restore as the superuser role
4. Restart services: docker compose up -d
5. Verify: make pi-status and check container health

Authelia Users Recovery

1. SSH to Caroline
2. Copy the backup from ~/backups/authelia-users-<timestamp>.yml to authelia/users.yml in the project directory
3. Restart Authelia: docker compose restart authelia
4. Verify login works before closing the session

R2 Restore (Full DR)

1. Run scripts/backup/dr-drill.sh — it documents the full restore procedure and tests it.
2. Retrieve the encrypted dump from R2 using the credentials in .env.
3. Decrypt and restore per the drill script output.

Backup Verification

scripts/backup/verify-restore.sh tests that a backup file can actually be restored into a temporary database. Run this periodically or after any change to the backup pipeline.

# Check backup log for recent runs
make pi-db-shell
# In psql:
SELECT created_at, status, size_bytes, notes
FROM backup_log
ORDER BY created_at DESC
LIMIT 10;

Scripts Reference

Script	Purpose
scripts/backup-hourly.sh	Hourly pg_dump + Authelia users.yml backup on Caroline
scripts/backup/rsync-to-nas.sh	Rsync backups to NAS
scripts/backup/push-offsite.sh	Encrypt and push to Cloudflare R2
scripts/backup/dr-drill.sh	Full disaster recovery drill
scripts/backup/local-pg-dump.sh	Manual local Postgres dump
scripts/backup/verify-restore.sh	Verify a backup file is restorable
scripts/backup/retention-local.sh	Prune old local backups
scripts/backup/setup-wal-archive.sh	WAL archiving setup