The 3-2-1 Rule
- 3 copies of your data
- 2 different storage types
- 1 offsite location
Example:
- Production server (primary)
- Local backup on separate disk (second copy, different media)
- Remote backup at different provider (offsite)
Recovery Time Objectives
| Metric | Definition | Your Target |
|---|---|---|
| RTO | How long until service is restored | < 1 hour? < 4 hours? |
| RPO | How much data loss is acceptable | 0? Last hour? Last day? |
Tip Your RTO and RPO determine your backup strategy and cost. Real-time replication for RPO=0 is expensive. Daily backups for RPO=24h are cheap. Choose based on business impact.
What to Back Up
Critical (back up hourly or more)
- Database (full + WAL/binlog for point-in-time)
- User-uploaded files
- Application secrets and certificates
Important (back up daily)
- Application code (also in git)
- Server configuration files
- Cron jobs and scripts
Nice to Have (back up weekly)
- Log files
- Monitoring data
- Development environments
Automated Backup Script
#!/bin/bash
# /opt/backup/disaster-recovery.sh
set -euo pipefail
REMOTE="backup-server:/backups/$(hostname)"
DATE=$(date +%Y%m%d_%H%M)
LOCAL="/backup/$DATE"
mkdir -p "$LOCAL"
# 1. Database
echo "[$(date)] Backing up database..."
mysqldump --all-databases --single-transaction | gzip > "$LOCAL/mysql.sql.gz"
# 2. Application files
echo "[$(date)] Backing up app files..."
tar czf "$LOCAL/www.tar.gz" -C /var/www html --exclude='node_modules' --exclude='.git'
# 3. Configuration
echo "[$(date)] Backing up config..."
tar czf "$LOCAL/etc.tar.gz" /etc/nginx /etc/letsencrypt /etc/systemd/system/*.service 2>/dev/null
# 4. Push to remote
echo "[$(date)] Syncing to remote..."
rsync -az --delete "$LOCAL/" "$REMOTE/$DATE/"
# 5. Cleanup local (keep 3 days)
find /backup -maxdepth 1 -type d -mtime +3 -exec rm -rf {} \;
echo "[$(date)] Backup complete: $LOCAL"
Recovery Runbook
Document these steps BEFORE you need them:
- Provision new server — same plan, same region
- Install base packages — restore
/etcconfigs - Restore database — import latest dump
- Restore application files — extract www archive
- Restore SSL certificates — or re-issue via Let's Encrypt
- Update DNS — point domain to new server IP
- Verify — check all endpoints, test login, verify data
Testing Recovery
Schedule a quarterly drill:
# Spin up a test server
# Restore from backup
# Verify everything works
# Document issues found
# Destroy test server
Danger An untested backup is not a backup. If you've never practiced a full restore, you don't know if it works, how long it takes, or what steps you'll forget under pressure.