Backup and Recovery

Implement comprehensive backup and recovery strategies using rsync, Restic, and cloud storage backends. Covers the 3-2-1 rule, automated scheduling, S3/B2 backends, encryption, restore procedures, and verification testing.

When to Use

Designing a backup strategy for servers, databases, or application data
Setting up Restic for encrypted, deduplicated backups to local or cloud storage
Automating backups with systemd timers or cron
Restoring data after accidental deletion, corruption, or disaster
Migrating data between environments using backup/restore workflows
Verifying backup integrity and testing recovery procedures

Prerequisites

rsync installed (included in most Linux distributions)
restic installed (v0.16+ recommended)
Cloud CLI configured for the backend: AWS CLI for S3, b2 CLI for Backblaze B2
Sufficient storage at the backup destination (2-3x source size for retention)
SSH access for remote rsync targets
systemd or cron for scheduling

The 3-2-1 Backup Rule

3 copies of your data (1 primary + 2 backups)
2 different storage media or types (e.g., local disk + cloud)
1 copy offsite (cloud storage, remote datacenter)

rsync Backups

Basic Operations

# Sync a local directory to a backup location
rsync -avz --delete /data/ /backup/data/

# Flags explained:
# -a  archive mode (preserves permissions, ownership, timestamps, symlinks)
# -v  verbose output
# -z  compress data during transfer
# --delete  remove files at destination that no longer exist at source

# Sync to a remote server over SSH
rsync -avz -e "ssh -i ~/.ssh/backup_key" /data/ backup@remote:/backups/server01/

# Exclude patterns
rsync -avz --delete \
  --exclude='*.tmp' \
  --exclude='*.log' \
  --exclude='.cache/' \
  --exclude='node_modules/' \
  /data/ /backup/data/

# Use an exclude file for complex patterns
rsync -avz --delete --exclude-from=/etc/backup-excludes.txt /data/ /backup/data/

# Dry run (preview what would change)
rsync -avzn --delete /data/ /backup/data/

# Limit bandwidth to 50 MB/s and show progress
rsync -avz --bwlimit=50000 --progress /data/ backup@remote:/backups/

Incremental Backups with Hard Links

# Incremental: unchanged files hard-linked to previous backup (saves space)
rsync -avz --delete \
  --link-dest=/backup/daily/latest \
  /data/ /backup/daily/$(date +%Y-%m-%d)/

# Update the 'latest' symlink
ln -snf /backup/daily/$(date +%Y-%m-%d) /backup/daily/latest

# Remove backups older than 30 days
find /backup/daily -maxdepth 1 -type d -name "20*" -mtime +30 -exec rm -rf {} \;

Restic Backup

Installation

# Debian / Ubuntu
apt install -y restic

# RHEL / CentOS
dnf install -y restic

# Or download the latest binary
curl -L https://github.com/restic/restic/releases/latest/download/restic_0.17.3_linux_amd64.bz2 \
  | bunzip2 > /usr/local/bin/restic
chmod +x /usr/local/bin/restic

# Verify installation
restic version

Initialize a Repository

# Local repository
restic init --repo /backup/restic-repo

# AWS S3 backend
export AWS_ACCESS_KEY_ID="AKIAEXAMPLE"
export AWS_SECRET_ACCESS_KEY="secretkey"
restic init --repo s3:s3.amazonaws.com/my-backup-bucket

# S3-compatible (MinIO)
export AWS_ACCESS_KEY_ID="minioadmin"
export AWS_SECRET_ACCESS_KEY="miniosecret"
restic init --repo s3:http://minio.example.com:9000/backup-bucket

# Backblaze B2 backend
export B2_ACCOUNT_ID="accountid"
export B2_ACCOUNT_KEY="accountkey"
restic init --repo b2:my-backup-bucket:server01

# SFTP backend
restic init --repo sftp:backup@remote:/backups/server01

# Restic will prompt for a repository password -- store it securely
# Use a password file for automation
echo "my-secure-repo-password" > /etc/restic/password.txt
chmod 600 /etc/restic/password.txt

Backup Operations

# Basic backup
restic backup /data --repo /backup/restic-repo --password-file /etc/restic/password.txt

# Backup multiple directories
restic backup /data /etc /var/lib/postgresql \
  --repo s3:s3.amazonaws.com/my-backup-bucket \
  --password-file /etc/restic/password.txt

# Backup with exclusions
restic backup /data \
  --exclude='*.tmp' \
  --exclude='*.log' \
  --exclude-file=/etc/restic/excludes.txt \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

# Backup with tags (useful for filtering snapshots later)
restic backup /data \
  --tag server01 --tag production --tag daily \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

# Backup stdin (e.g., database dump)
pg_dump -U postgres mydb | restic backup --stdin --stdin-filename mydb.sql \
  --repo s3:s3.amazonaws.com/my-backup-bucket \
  --password-file /etc/restic/password.txt

# Verbose output showing files processed
restic backup /data -v \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

Snapshot Management

# List all snapshots (add --tag <tag> to filter)
restic snapshots --repo /backup/restic-repo --password-file /etc/restic/password.txt

# Browse files in the latest snapshot
restic ls latest --repo /backup/restic-repo --password-file /etc/restic/password.txt

# Compare two snapshots
restic diff abc123 def456 --repo /backup/restic-repo --password-file /etc/restic/password.txt

Retention Policy (forget + prune)

# Apply retention policy and reclaim space
restic forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 3 \
  --prune \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

# Dry run to preview what would be removed
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 \
  --dry-run --repo /backup/restic-repo --password-file /etc/restic/password.txt

Restore Procedures

# Restore the latest snapshot to a target directory
restic restore latest --target /restore \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

# Restore a specific snapshot by ID
restic restore abc123 --target /restore \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

# Restore only specific files or directories
restic restore latest --target /restore --include "/data/config" \
  --repo /backup/restic-repo \
  --password-file /etc/restic/password.txt

# Mount a snapshot as a FUSE filesystem (browse and copy individual files)
mkdir -p /mnt/restic
restic mount /mnt/restic --repo /backup/restic-repo --password-file /etc/restic/password.txt &
# Browse: ls /mnt/restic/snapshots/latest/
# Unmount when done: umount /mnt/restic

Verification

# Verify repository integrity (checks all data and metadata)
restic check --repo /backup/restic-repo --password-file /etc/restic/password.txt

# Full data verification (reads all pack files -- slow but thorough)
restic check --read-data --repo /backup/restic-repo --password-file /etc/restic/password.txt

# Verify a random subset of data (faster than full read)
restic check --read-data-subset=5% --repo /backup/restic-repo --password-file /etc/restic/password.txt

Automated Backup with Environment File

/etc/restic/env

# Repository configuration
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/my-backup-bucket"
export RESTIC_PASSWORD_FILE="/etc/restic/password.txt"
export AWS_ACCESS_KEY_ID="AKIAEXAMPLE"
export AWS_SECRET_ACCESS_KEY="secretkey"

# Optional: set cache directory
export RESTIC_CACHE_DIR="/var/cache/restic"

Backup Script

#!/bin/bash
# /usr/local/bin/restic-backup.sh
set -euo pipefail
source /etc/restic/env
LOG="/var/log/restic-backup.log"

echo "$(date): Starting backup" >> "$LOG"

restic backup /data /etc /var/lib/postgresql \
  --exclude-file=/etc/restic/excludes.txt \
  --tag "$(hostname)" --tag daily \
  --verbose >> "$LOG" 2>&1

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 \
  --prune >> "$LOG" 2>&1

# Weekly integrity check (Sundays)
[ "$(date +%u)" -eq 7 ] && restic check --read-data-subset=10% >> "$LOG" 2>&1

echo "$(date): Backup completed" >> "$LOG"

chmod +x /usr/local/bin/restic-backup.sh

Scheduled Backups

Systemd Timer (Recommended)

# /etc/systemd/system/restic-backup.service
[Unit]
Description=Restic backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/restic-backup.sh
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Run Restic backup daily at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=900

[Install]
WantedBy=timers.target

# Enable and start the timer
systemctl daemon-reload
systemctl enable --now restic-backup.timer

# Check timer status
systemctl list-timers restic-backup.timer

# Run manually for testing
systemctl start restic-backup.service
journalctl -u restic-backup.service -f

Database Backups with Restic

# PostgreSQL: stream dump directly into restic (no temp file)
pg_dump -U postgres -Fc mydb | restic backup --stdin --stdin-filename mydb.dump \
  --tag postgres --tag mydb \
  --repo s3:s3.amazonaws.com/my-backup-bucket \
  --password-file /etc/restic/password.txt

# MySQL / MariaDB: stream dump into restic
mysqldump --all-databases --single-transaction | \
  restic backup --stdin --stdin-filename all-databases.sql \
  --tag mysql \
  --repo s3:s3.amazonaws.com/my-backup-bucket \
  --password-file /etc/restic/password.txt

# Restore PostgreSQL from restic
restic dump latest mydb.dump \
  --repo s3:s3.amazonaws.com/my-backup-bucket \
  --password-file /etc/restic/password.txt \
  | pg_restore -U postgres -d mydb --clean --if-exists

Troubleshooting

Symptom	Diagnostic Command	Common Fix
"repository not initialized"	`restic cat config --repo <repo>`	Run `restic init --repo <repo>` first
"wrong password"	Check env vars / password file	Verify RESTIC_PASSWORD_FILE contents and permissions
Backup is slow	`restic backup -v` for progress	Check network bandwidth; exclude large unneeded dirs
S3 permission denied	`aws s3 ls s3://bucket/`	Check IAM policy includes s3:GetObject, s3:PutObject
"unable to create lock"	`restic unlock --repo <repo>`	A previous backup crashed; unlock the repository
Restore shows empty dirs	`restic ls <snapshot-id>`	Verify correct snapshot ID; check --include path syntax
Repository growing too large	`restic stats --repo <repo>`	Run `restic forget --prune` with stricter retention
Check fails with pack errors	`restic check --read-data`	Rebuild index: `restic rebuild-index`; restore from another copy

Related Skills

linux-administration -- Server maintenance and log management
systemd-services -- Scheduling backups with systemd timers
object-storage -- S3 and MinIO as backup destinations
block-storage -- LVM snapshots for consistent backups
nfs-storage -- Backing up NFS-shared data

backup-recovery