S3-Compatible Database Backups with WAL-G

WAL-G is a high-performance archival and restoration tool for PostgreSQL (and MySQL/MariaDB, MongoDB, and SQL Server) that provides compressed, encrypted backups to S3-compatible object storage. It handles both base backups and continuous WAL archiving, enabling point-in-time recovery (PITR) with minimal storage costs. This guide covers setup, configuration, and operational procedures.

Why WAL-G?

• Speed — parallel compression and upload using all CPU cores
• Efficiency — delta backups store only changed pages, reducing backup size by 80-90%
• S3-compatible — works with AWS S3, MinIO, Wasabi, Backblaze B2, DigitalOcean Spaces
• PITR — continuous WAL archiving enables recovery to any point in time
• Encryption — built-in AES-256 encryption at rest

Installation

# Download latest release
WAL_G_VERSION=v3.0.3
curl -L "https://github.com/wal-g/wal-g/releases/download/${WAL_G_VERSION}/wal-g-pg-ubuntu-20.04-amd64.tar.gz" | tar xz
sudo mv wal-g-pg-ubuntu-20.04-amd64 /usr/local/bin/wal-g
sudo chmod +x /usr/local/bin/wal-g

# Verify
wal-g --version

Configuration

WAL-G reads configuration from environment variables. Create a configuration file:

# /etc/wal-g/env.conf (sourced by systemd or wrapper script)
WALG_S3_PREFIX=s3://my-backups/postgres-prod
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_REGION=us-east-1

# For S3-compatible storage (MinIO, Wasabi, etc.)
AWS_ENDPOINT=https://s3.wasabisys.com
AWS_S3_FORCE_PATH_STYLE=true

# Compression (lz4 is fastest, zstd offers better ratio)
WALG_COMPRESSION_METHOD=zstd

# Encryption (optional but recommended)
WALG_LIBSODIUM_KEY=your-32-byte-hex-encryption-key

# Delta backups (store only changed pages)
WALG_DELTA_MAX_STEPS=5

# Parallel upload
WALG_UPLOAD_CONCURRENCY=4
WALG_DOWNLOAD_CONCURRENCY=4

# PostgreSQL connection
PGHOST=/var/run/postgresql
PGUSER=postgres
PGDATABASE=postgres

Configure PostgreSQL for WAL Archiving

# postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'source /etc/wal-g/env.conf && wal-g wal-push %p'
archive_timeout = 60  # Force archiving at least every 60 seconds

# Restart PostgreSQL
sudo systemctl restart postgresql

Creating Backups

# Full base backup
source /etc/wal-g/env.conf
wal-g backup-push /var/lib/postgresql/16/main

# Delta backup (only changed pages since last full backup)
wal-g backup-push --delta /var/lib/postgresql/16/main

# List backups
wal-g backup-list

# Output:
# name                          modified             wal_segment_backup_start
# base_000000010000000000000010 2025-01-15T03:00:00Z 000000010000000000000010
# base_000000010000000000000015 2025-01-16T03:00:00Z 000000010000000000000015

Automated Backup Schedule

#!/bin/bash
# /usr/local/bin/walg-backup.sh
source /etc/wal-g/env.conf

DAY_OF_WEEK=$(date +%u)  # 1=Monday, 7=Sunday

if [ "$DAY_OF_WEEK" -eq 1 ]; then
    # Full backup on Mondays
    wal-g backup-push /var/lib/postgresql/16/main
else
    # Delta backups other days
    wal-g backup-push --delta /var/lib/postgresql/16/main
fi

# Retain last 7 full backups
wal-g delete retain FULL 7 --confirm

# Crontab
0 3 * * * /usr/local/bin/walg-backup.sh >> /var/log/walg-backup.log 2>&1

Restoring from Backup

Full Restore

# Stop PostgreSQL
sudo systemctl stop postgresql

# Clear data directory
sudo rm -rf /var/lib/postgresql/16/main/*

# Restore latest backup
source /etc/wal-g/env.conf
wal-g backup-fetch /var/lib/postgresql/16/main LATEST

# Or restore a specific backup
wal-g backup-fetch /var/lib/postgresql/16/main base_000000010000000000000010

# Create recovery signal file
touch /var/lib/postgresql/16/main/recovery.signal

# Configure recovery in postgresql.conf (or postgresql.auto.conf)
cat >> /var/lib/postgresql/16/main/postgresql.auto.conf > /var/lib/postgresql/16/main/postgresql.auto.conf