The Linux backup heaven. snapper + btrbk. Incremental btrfs backups to a remote ssh server

This guide explains how to configure btrbk to incrementally send existing Snapper snapshots to a remote Btrfs server using SSH. This setup assumes you already have Snapper configured and creating snapshots on your system.

Table of Contents

1. Overview
2. Prerequisites
3. Understanding the Setup
4. SSH Setup
5. btrbk Configuration
6. Remote Backup Cleanup
7. Testing and Verification
8. Automation
9. Monitoring and Maintenance
10. Troubleshooting

Overview

This setup allows you to:

• Incrementally send existing Snapper snapshots to a remote Btrfs server
• Preserve local snapshots according to Snapper's cleanup policies
• Maintain separate retention policies for local and remote backups
• Transfer efficiently using Btrfs send/receive with SSH
• Automate cleanup of old remote backups

Key Concept

btrbk works with your existing Snapper setup - it doesn't create or manage snapshots, it only sends them to remote storage and handles remote backup cleanup.

Prerequisites

On the Client (Source System)

# Install required packages
sudo apt install btrbk openssh-client  # Debian/Ubuntu
sudo dnf install btrbk openssh-clients  # Fedora
sudo pacman -S btrbk openssh           # Arch Linux

# Verify Btrfs tools are installed
btrfs --version

On the Server (Remote Backup System)

# Install required packages
sudo apt install btrfs-progs openssh-server  # Debian/Ubuntu
sudo dnf install btrfs-progs openssh-server  # Fedora
sudo pacman -S btrfs-progs openssh           # Arch Linux

# Create Btrfs filesystem for backups (if not already exists)
sudo mkfs.btrfs /dev/sdX1  # Replace with your backup disk
sudo mkdir -p /backups/omarchy/home
sudo mount /dev/sdX1 /backups

Verify Snapper Configuration

# Check existing Snapper configs
sudo snapper list-configs

# Verify snapshots are being created
sudo snapper -c home list  # Assuming home config exists

# Check snapshot directory structure
ls -la /home/.snapshots/

Understanding the Setup

How btrbk Works with Snapper

• Snapper creates snapshots in /home/.snapshots/
• btrbk reads existing snapshots and sends them incrementally
• btrbk manages remote backup cleanup based on retention policies
• Local snapshots remain under Snapper's control

Typical Snapper Structure

/home/.snapshots/           # Snapper home snapshots
├── 1/
│   ├── snapshot.xml
│   └── snapshot
├── 2/
│   ├── snapshot.xml
│   └── snapshot
├── 3/
│   ├── snapshot.xml
│   └── snapshot
└── info.xml

Data Flow

Snapper creates snapshot → btrbk detects snapshot → btrbk sends incrementally → Remote server stores backup → btrbk cleans up old remote backups

SSH Setup

1. Create SSH Key Pair on Client

# Create dedicated btrbk SSH key
sudo mkdir -p /etc/btrbk/ssh
sudo ssh-keygen -t ed25519 -f /etc/btrbk/ssh/id_ed25519 -C "btrbk@$(hostname)" -N ""

# Set proper permissions
sudo chmod 600 /etc/btrbk/ssh/id_ed25519
sudo chmod 644 /etc/btrbk/ssh/id_ed25519.pub

2. Configure Remote User on Server

Create remoteuser and Setup Permissions

# On server, create remoteuser
sudo useradd -r -s /usr/sbin/nologin remoteuser
sudo mkdir -p /home/remoteuser/.ssh
sudo chmod 700 /home/remoteuser/.ssh

# Add SSH key from client
sudo tee /home/remoteuser/.ssh/authorized_keys << 'EOF'
# Paste the content of /etc/btrbk/ssh/id_ed25519.pub from client
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... btrbk@client-hostname
EOF

sudo chmod 600 /home/remoteuser/.ssh/authorized_keys
sudo chown -R remoteuser:remoteuser /home/remoteuser/.ssh

# Configure sudo for btrbk operations
sudo tee /etc/sudoers.d/remoteuser << 'EOF'
remoteuser ALL=(ALL) NOPASSWD: /bin/btrfs, /usr/bin/btrfs
EOF

Create Backup Directory

# Create backup directory structure
sudo mkdir -p /backups/omarchy/home
sudo chown remoteuser:remoteuser /backups/omarchy/home
sudo chmod 755 /backups/omarchy/home

3. Test SSH Connection

# From client, test connection to remoteuser
sudo -u btrbk ssh -i /etc/btrbk/ssh/id_ed25519 remoteuser@backup-server "btrfs version"

# Test btrfs command execution
sudo -u btrbk ssh -i /etc/btrbk/ssh/id_ed25519 remoteuser@backup-server "sudo btrfs version"

btrbk Configuration

Create /etc/btrbk/btrbk.conf:

# /etc/btrbk/btrbk.conf
# Configuration for sending Snapper home snapshots to remote server

# Global settings
timestamp_format        long
snapshot_preserve_min   2d
snapshot_preserve      14d

# Remote backup retention policy
target_preserve_min    no
target_preserve        30d 12w *m

# SSH settings for remote connection
ssh_identity           /etc/btrbk/ssh/id_ed25519
ssh_user               remoteuser

# Lockfile to prevent concurrent runs
lockfile               /run/lock/btrbk.lock

# Transaction logging to systemd journal
transaction_syslog     user

# Home directory configuration (works with existing Snapper setup)
volume /home
    # Point to Snapper's snapshot directory
    snapshot_dir        .snapshots
    
    # IMPORTANT: Don't create snapshots - let Snapper handle it
    snapshot_create     no
    
    # Configure subvolume (home directory)
    subvolume .
        # Custom snapshot name for btrbk tracking
        snapshot_name    home-snapper
        
        # Remote backup target
        target send-receive ssh://backup-server/backups/omarchy/home
        
        # Incremental backups (default and recommended)
        incremental      yes
        
        # Compression for network transfer
        stream_compress  zstd
        stream_compress_level 3

Configuration Explanation

Remote Backup Cleanup

Automatic Cleanup Mechanism

btrbk automatically handles cleanup of remote backups based on the target_preserve policy. No separate cleanup commands are needed.

How Cleanup Works

1. When btrbk runs, it creates new incremental backups
2. Then evaluates existing remote backups against the retention policy
3. Deletes backups that don't match the retention policy
4. Logs all cleanup actions for verification

Retention Policy Example

target_preserve        30d 12w *m

This means:

• Keep all backups for minimum time (if target_preserve_min is set)
• Keep 30 daily backups (first backup of each day)
• Keep 12 weekly backups (first backup of each week)
• Keep all monthly backups (first backup of each month) forever

Manual Cleanup (if needed)

List Remote Backups

# List all remote backups
sudo btrbk -c /etc/btrbk/btrbk.conf list backups

# Detailed backup information
sudo btrbk -c /etc/btrbk/btrbk.conf list -L backups

Force Cleanup Run

# Run btrbk normally (includes automatic cleanup)
sudo btrbk -c /etc/btrbk/btrbk.conf run

# Run with verbose output to see cleanup actions
sudo btrbk -c /etc/btrbk/btrbk.conf -v run

Manual Cleanup on Remote Server

# Connect to remote server and list backups
ssh remoteuser@backup-server "sudo btrfs subvolume list /backups/omarchy/home"

# Manually delete a specific backup (use with caution)
ssh remoteuser@backup-server "sudo btrfs subvolume delete /backups/omarchy/home/home-snapper.20240101T1200"

# Check disk usage
ssh remoteuser@backup-server "sudo btrfs filesystem df /backups"

Cleanup Verification

Dry Run to See What Would Be Deleted

# See what actions would be performed (including deletions)
sudo btrbk -c /etc/btrbk/btrbk.conf -n -v run

# Check for deletion messages in dry run output
sudo btrbk -c /etc/btrbk/btrbk.conf -n -v run | grep -i delete

Monitor Cleanup in Logs

# Watch btrbk journal for cleanup actions
sudo journalctl -u btrbk -f | grep -E "(delete|cleanup|remove)"

# Check recent cleanup activity
sudo journalctl -u btrbk --since "1 day ago" | grep -E "(delete|cleanup|remove)"

Verify Remote Backup Count

# Count remote backups
ssh remoteuser@backup-server "sudo btrfs subvolume list /backups/omarchy/home | wc -l"

# List remote backups with dates
ssh remoteuser@backup-server "sudo btrfs subvolume list /backups/omarchy/home | grep home-snapper"

Testing and Verification

1. Check Configuration Syntax

# Verify configuration file syntax
sudo btrbk -c /etc/btrbk/btrbk.conf config print

# Check what would be done (dry run)
sudo btrbk -c /etc/btrbk/btrbk.conf -n -v run

2. List Existing Snapper Snapshots

# List Snapper snapshots
sudo snapper -c home list

# List snapshots as seen by btrbk
sudo btrbk -c /etc/btrbk/btrbk.conf list snapshots

3. Test Initial Backup

# Run first backup (may take longer for initial transfer)
sudo btrbk -c /etc/btrbk/btrbk.conf -v run

# Check results
sudo btrbk -c /etc/btrbk/btrbk.conf list backups

# Verify on remote server
ssh remoteuser@backup-server "sudo btrfs subvolume list /backups/omarchy/home"

4. Test Incremental Backup

# Create a new Snapper snapshot
sudo snapper -c home create --description "Test snapshot for btrbk"

# Run btrbk again (should be incremental and faster)
sudo btrbk -c /etc/btrbk/btrbk.conf -v run

# Verify incremental transfer (check logs for "incremental")
sudo journalctl -u btrbk --since "10 minutes ago" | grep -i incremental

5. Test Cleanup Mechanism

# Create multiple snapshots to trigger cleanup
for i in {1..5}; do
    sudo snapper -c home create --description "Test snapshot $i for cleanup"
    sleep 2
done

# Run btrbk to trigger cleanup
sudo btrbk -c /etc/btrbk/btrbk.conf -v run

# Verify cleanup happened
sudo btrbk -c /etc/btrbk/btrbk.conf list backups
ssh remoteuser@backup-server "sudo btrfs subvolume list /backups/omarchy/home"

Automation

Option 1: Cron Job

Create /etc/cron.hourly/btrbk:

#!/bin/bash
# /etc/cron.hourly/btrbk
# Send Snapper snapshots to remote server

exec /usr/bin/btrbk -q -c /etc/btrbk/btrbk.conf run

Make it executable:

sudo chmod +x /etc/cron.hourly/btrbk

Option 2: Systemd Timer

Create /etc/systemd/system/btrbk.service:

[Unit]
Description=Send Snapper snapshots to remote server with btrbk
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/btrbk -c /etc/btrbk/btrbk.conf run
StandardOutput=journal
StandardError=journal
SyslogIdentifier=btrbk

Create /etc/systemd/system/btrbk.timer:

[Unit]
Description=Run btrbk hourly for Snapper snapshot backup
Requires=btrbk.service

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now btrbk.timer

Scheduling Recommendations

Monitoring and Maintenance

Log Monitoring

# Monitor btrbk activity in systemd journal
sudo journalctl -u btrbk -f

# Check for errors in recent logs
sudo journalctl -u btrbk --since "1 hour ago" | grep -i error

# Check recent backup activity
sudo journalctl -u btrbk --since "1 hour ago"

# View all btrbk logs
sudo journalctl -u btrbk

Backup Verification

# List all backups
sudo btrbk -c /etc/btrbk/btrbk.conf list backups

# Check backup statistics
sudo btrbk -c /etc/btrbk/btrbk.conf stats

# Verify remote backups exist
ssh remoteuser@backup-server "sudo btrfs subvolume list /backups/omarchy/home"

Regular Maintenance

# On backup server, run regular Btrfs scrub
sudo btrfs scrub start /backups

# Check filesystem status
sudo btrfs filesystem show

# Monitor disk usage
ssh remoteuser@backup-server "df -h /backups"

Health Monitoring Script

Create /usr/local/bin/btrbk-health.sh:

#!/bin/bash
# Health check for btrbk backup system

LOG_FILE="/var/log/btrbk-health.log"
echo "$(date): Starting btrbk health check" | tee -a $LOG_FILE

# Check if last backup was successful
if sudo journalctl -u btrbk --since "1 hour ago" | grep -q "ERROR"; then
    echo "$(date): ERROR detected in btrbk logs" >> $LOG_FILE
    exit 1
fi

# Check remote server connectivity
if ! sudo -u btrbk ssh -i /etc/btrbk/ssh/id_ed25519 -o ConnectTimeout=10 remoteuser@backup-server "echo OK" 2>/dev/null; then
    echo "$(date): ERROR: Cannot connect to remote server" >> $LOG_FILE
    exit 1
fi

# Check remote disk space
REMOTE_USAGE=$(ssh remoteuser@backup-server "df /backups | tail -1 | awk '{print \$5}' | sed 's/%//'")
if [ "$REMOTE_USAGE" -gt 90 ]; then
    echo "$(date): WARNING: Remote disk usage is ${REMOTE_USAGE}%" >> $LOG_FILE
fi

echo "$(date): Health check completed successfully" >> $LOG_FILE

Make it executable and add to cron:

sudo chmod +x /usr/local/bin/btrbk-health.sh
# Add to crontab for daily checks

Troubleshooting

Common Issues and Solutions

1. SSH Connection Problems

# Test SSH connection manually
sudo -u btrbk ssh -i /etc/btrbk/ssh/id_ed25519 -v remoteuser@backup-server "btrfs version"

# Check permissions
ls -la /etc/btrbk/ssh/
ls -la /home/remoteuser/.ssh/

# Fix permissions if needed
sudo chmod 600 /etc/btrbk/ssh/id_ed25519
sudo chmod 600 /home/remoteuser/.ssh/authorized_keys

2. Permission Issues on Remote Server

# Test btrfs command execution
ssh remoteuser@backup-server "sudo btrfs version"

# Check sudo configuration
ssh remoteuser@backup-server "sudo -l"

# Verify backup directory permissions
ssh remoteuser@backup-server "ls -la /backups/omarchy/home"

3. Snapper Snapshots Not Found

# Verify Snapper snapshot directory exists
ls -la /home/.snapshots/

# Check Snapper config
sudo snapper -c home get-config | grep SUBVOLUME

# Verify snapshots exist
sudo snapper -c home list

4. Remote Backup Directory Issues

# Check if remote directory exists and is accessible
ssh remoteuser@backup-server "ls -la /backups/omarchy/home"

# Test write permissions
ssh remoteuser@backup-server "touch /backups/omarchy/home/testfile && rm /backups/omarchy/home/testfile"

# Verify Btrfs filesystem
ssh remoteuser@backup-server "sudo btrfs filesystem df /backups"

5. Cleanup Not Working

# Check retention policy configuration
sudo btrbk -c /etc/btrbk/btrbk.conf config print | grep preserve

# Run with verbose output to see cleanup decisions
sudo btrbk -c /etc/btrbk/btrbk.conf -vvv run | grep -i preserve

# Manually trigger cleanup
sudo btrbk -c /etc/btrbk/btrbk.conf run

6. Incremental Transfer Issues

# Check if incremental is enabled
sudo btrbk -c /etc/btrbk/btrbk.conf config print | grep incremental

# Verify parent-child relationships
sudo btrbk -c /etc/btrbk/btrbk.conf -v run | grep -i parent

# Force full backup (if incremental is broken)
sudo btrbk -c /etc/btrbk/btrbk.conf --override incremental=no run

Debug Mode

# Run with maximum verbosity
sudo btrbk -c /etc/btrbk/btrbk.conf -vvv run

# Check transaction log in systemd journal
sudo journalctl -u btrbk -f

# Dry run with debug output
sudo btrbk -c /etc/btrbk/btrbk.conf -n -vvv run

Performance Issues

# Check network bandwidth
ssh remoteuser@backup-server "iperf -s"  # On server
iperf -c backup-server                  # On client

# Adjust compression if CPU is bottleneck
# Edit btrbk.conf and change stream_compress_level

# Add rate limiting if network is saturated
# Add to btrbk.conf: rate_limit 10m

Advanced Configuration Options

Different Retention Policies

# Keep remote backups longer than local snapshots
target_preserve_min   7d
target_preserve       60d 24w *m

# Or keep fewer remote backups
target_preserve_min   no
target_preserve       7d 4w *m

Compression Options

# Different compression algorithms
stream_compress  gzip    # Good compatibility
stream_compress  zstd    # Best compression/speed ratio
stream_compress  lz4     # Fastest compression

# Compression levels
stream_compress_level 1  # Fastest
stream_compress_level 9  # Best compression

Rate Limiting

# Limit network usage for large transfers
rate_limit            10m  # 10 MB/s
rate_limit_remote     5m   # 5 MB/s on remote side

Buffering for Performance

# Add buffering for better performance
stream_buffer        256m  # 256MB buffer
stream_buffer_remote 128m  # 128MB buffer on remote side

Multiple Backup Targets

subvolume .
    # Local backup
    target send-receive /mnt/local-backup/home
    
    # Remote backup
    target send-receive ssh://backup-server/backups/omarchy/home
    
    # Secondary remote backup
    target send-receive ssh://backup-server2/backups/omarchy/home

Security Considerations

SSH Key Protection

• Store SSH keys with proper permissions (600)
• Consider using hardware tokens for high-security environments
• Regularly rotate SSH keys

Network Security

• Use VPN or dedicated network links for backup transfers
• Configure firewall rules to restrict SSH access
• Monitor SSH logs for unauthorized access attempts

Data Integrity

• Regularly verify backup integrity

Use Btrfs scrub on backup server:

sudo btrfs scrub start /backups

Access Control

• Use dedicated remoteuser with minimal privileges
• Implement sudo rules for btrfs commands only
• Consider using ssh_filter_btrbk.sh for command restriction

Conclusion

This setup provides a robust, automated solution for sending your existing Snapper snapshots to a remote Btrfs server. The key benefits are:

• Efficient incremental transfers using Btrfs send/receive
• Automatic cleanup of old remote backups based on retention policies
• Non-intrusive - works with your existing Snapper setup
• Secure SSH-based transfers with dedicated user access
• Automated with cron or systemd timers

Remember to regularly test your backup restoration process and monitor the system logs to ensure your backups are reliable when you need them.

Further Reading

• btrbk Official Documentation
• Snapper Documentation