Tim's blah blah blah

My 1 2 3 Offsite Backup Routine

While I prefer to self-host, I also like to not lose my data. To this end, I backup my stuff on my server, a local hard disk, and a hard disk at an offsite location. I mark parts of the files read-only to have some mild protection against ransomware, and use par2 files to have some bitrot protection for critical datasets.

Backup strategy

Data sources

I have three data sources:

  1. Time Machine backup
  2. Pictures & projects
  3. Virtual machine images

Data storage

I store backups on three locations:

  1. Server storage (NAS) virtual machine
  2. External HDD at home
  3. External HDD at offsite location

The first allows me to do frequent backups, e.g. Time Machine, and additionally sync data irregularly that’s always available. The second storage is irregularly updated from the main NAS backup, and rotated with the offsite backup.

Backup protection

I mark critical data as read-only to have some protection against ransomware or accidental deletion. On MacOS I use chflags schg and on the NAS I use chattr +i on the hypervisor (proxmox). Both require a root password, and the NAS files are locked on the hypervisor, which means even root on the virtual machine cannot delete them. I only lock files such that I can still add files to the directories.

Backup scripts

On pve 

sudo tar cJvf "/var/lib/vz/dump/pve-conf-$(date +\%Y_\%m_\%d-\%H_\%M_\%S).tar.xz" /etc

Combined

To WD5TB (pics, work, images, TM):

caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/Pictures /Volumes/WD5TB-Data-Tim/Tim/; caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs /Volumes/WD5TB-Data-Tim/; caffeinate rsync -avH --partial --progress --delete-after pve:/var/lib/vz/dump/ /Volumes/WD5TB-Data-Tim/Images/; caffeinate rsync --rsync-path 'sudo -u backupmba rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' proteus2:/mnt/backup/mba/ /Volumes/WD5TB-Data/Helene;

To SG5TB (pics, work, images, TM):

caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/Pictures /Volumes/SG5TB-Data/Tim/; caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs /Volumes/SG5TB-Data/Tim/; caffeinate rsync -avH --partial --progress --delete-after pve:/var/lib/vz/dump/ /Volumes/SG5TB-Data/Tim/Images/; caffeinate rsync --rsync-path 'sudo -u backupmba rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' proteus2:/mnt/backup/mba/ /Volumes/SG5TB-Data/Helene;

To 3TB (pics, workdocs, images):

caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/Pictures /Volumes/3TB/Tim/; caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs /Volumes/3TB-Data-Enc/; caffeinate rsync -avH --partial --progress --delete-after pve:/var/lib/vz/dump/ /Volumes/3TB-Data-Enc/Images/; caffeinate rsync --rsync-path 'sudo -u backupmba rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' proteus2:/mnt/backup/mba/ /Volumes/3TB/Helene;

To server (pics, workdocs)

caffeinate rsync --rsync-path 'sudo -u backupmbp rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/Pictures proteus2:/mnt/backup/data-tim/; caffeinate rsync --rsync-path 'sudo -u backupmbp rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs proteus2:/mnt/backup/data-tim/

From laptop to server

Sync pictures & workdocs to proteus

caffeinate rsync --rsync-path 'sudo -u backupmbp rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/Pictures proteus2:/mnt/backup/data-tim/;

caffeinate rsync --rsync-path 'sudo -u backupmbp rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs proteus2:/mnt/backup/data-tim/

From laptop to disk 

caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/Pictures /Volumes/3TB/Tim/; caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs /Volumes/WD5TB-Data-Tim/; caffeinate rsync -avH --partial --progress --delete pve:/var/lib/vz/dump/ /Volumes/WD5TB-Data-Tim/Images/; 

caffeinate rsync -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' ~/workdocs /Volumes/3TB/Tim/

From server to disk

Sync virtual machine images to disk

caffeinate rsync -avH --partial --progress --delete pve:/var/lib/vz/dump/ /Volumes/SG5TB-Data/Tim/Images/;
caffeinate rsync -avH --partial --progress --delete pve:/var/lib/vz/dump/ /Volumes/3TB-Data-Enc/Images/;

Sync TM

caffeinate rsync --rsync-path 'sudo -u backupmba rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' proteus2:/mnt/backup/mba/ /Volumes/SG5TB-TM-Helene/

caffeinate rsync --rsync-path 'sudo -u backupmba rsync' -avH --partial --progress --exclude '.Spotlight-V100' --exclude '.DS_Store*' --exclude '.Trashes' --exclude '.fseventsd' /Volumes/SG5TB-TM-Helene/ proteus2:/mnt/backup/mba/

Lock archive

Prevent accidental deletion / mild ransomware modification prevention.

Lock MacOS disks

On Mac:

find ~/Pictures/2025 -type f -print0 | xargs -0 sudo chflags schg
find /Volumes/WD5TB-Data/Helene/ -type f -print0 | xargs -0 sudo chflags schg
find /Volumes/WD5TB-Data/Tim/workdocs/ /Volumes/WD5TB-Data/Tim/Pictures/ -type f -print0 | xargs -0 sudo chflags schg

Lock NAS server

On pve:

sudo -u backupmba find /tank/backups/data-helene -type f -print0 | xargs -0 sudo chattr +i
sudo -u backupmbp find /tank/backups/data-tim -type f -print0 | xargs -0 sudo chattr +i

Parity

Make par2 files for all files.

TODO: add exception for .DS_Store and some hidden files.

Create PAR2

One photo dir:

rm *par2 && par2 c -r1 -q -- "$(basename $(pwd)).par2" *

One photo year dir:

find . -type d -print0 | xargs -0 -I {} -P 4 par2 c -r1 -q -- "{}/{}.par2" "{}/*"

All dirs:

for dir in 20*<12-24>*; do
    cd /Users/tim/Pictures/${dir}; pwd; find . -type d -print0 | xargs -0 -I {} -P 4 par2 c -r1 -q -- "{}/{}.par2" "{}/*"
done

Check / repair files

One photo dir:

par2 v -q -- "$(basename $(pwd)).par2"

One photo year dir:

find . -type d -print0 | xargs -0 -I {} -P 4 par2 v -q -- "{}/{}.par2" | grep "Target.*- missing.\|Target.*- damaged."

All dirs:

for dir in 20*<12-24>*; do
    cd /Users/tim/Pictures/${dir}; pwd; find . -type d -print0 | xargs -0 -I {} -P 4 par2 v -q -- "{}/{}.par2" | grep "Target.*- missing.\|Target.*- damaged."
done

#Adblock #Arduino #Css #Curl #Debian #Diy #Dyndns #E-Mail #Electricity #Epex #ESP8266 #Gandi #Grafana #Heating #Home-Assistant #Home-Improvement #Html #Htpc #Hugo #Influxdb #Ios #Letsencrypt #Linux #Mac #Markdown #Networking #Nextcloud #Nginx #Photography #Proxmox #RaspberryPi #Routeros #Security #Server #Smarthome #Solar #Soverin #Sqlite #Ssh #Transip #Ubuntu #Unix #Vyos #Windows