Using logrotate to organize backups

29 Apr, 2024

Some of my data is precious to me, so I want to back this up against loss or user errors. In addition to one backup, I want to keep a history of backups just in case. Here I document my approach, based on this SO solution (stackexchange.com) using logrotate (die.net).

Requirements ¶

I have the following goals:

Automate backups of InfluxDB & Home Assistant
Keep several daily & several monthly backups
Use trusted tooling

Regarding 3: I used xolox’ rotate-backups (pypi.org) before, but since the xz debacle (ycombinator.com) I’m a bit (more) hesitant of dorment projects. In addition, this seemed like a nice challenge.

Create backup ¶

Here I create my InfluxDB backup and tar gzip it. The hardlink is required for logrotate below.

/usr/bin/influxd backup -portable /home/tim/backup/influx/influx_snapshot && tar --remove-files -cvzf /home/tim/backup/influx/influx_snapshot.tar.gz -C /home/tim/backup/influx/ influx_snapshot && ln /home/tim/backup/influx/influx_snapshot.tar.gz /home/tim/backup/influx/influx_snapshot-monthly.tar.gz

Use logrotate to rotate them ¶

logrotate is a tool for rotating log files (gosh), i.e. splitting logs in several files so you can compress older logs you won’t look at so often, and keep the most recent one available as text. However logrotate can be used for any file (serverfault.com).

Target filename structure ¶

I want a few daily backups and a few weekly backups, with this naming scheme:

influx_snapshot.tar.gz
influx_snapshot.1.tar.gz
influx_snapshot-monthly.1.tar.gz

Testing config ¶

This is possible with this logrotate setup. Below is a test setup where I use a fake state file to trick logrotate to backup the file.

cat << 'EOF' > logrotate_full
/home/tim/backup/logrotate/influx_snapshot.tar.gz {
    su tim tim
    daily
    rotate 3
    copy
    nocompress
    ifempty
    missingok
    nocreate
    extension .tar.gz
}

/home/tim/backup/logrotate/influx_snapshot-monthly.tar.gz {
    su tim tim
    weekly
    rotate 3
    copy
    nocompress
    ifempty
    missingok
    extension .tar.gz
}
EOF

cat << 'EOF' > logrotate_state
logrotate state -- version 2
"/home/tim/backup/logrotate/influx_snapshot.tar.gz" 2024-1-21-21:00:00
"/home/tim/backup/logrotate/influx_snapshot-monthly.tar.gz" 2024-1-21-21:00:00
EOF

echo "helloworld" > influx_snapshot.tar.gz
ln influx_snapshot.tar.gz  influx_snapshot-monthly.tar.gz 

/usr/sbin/logrotate -v --state logrotate_state logrotate_full

Final config ¶

This is the final config I use. Now we wait.

cat << 'EOF' | sudo tee /etc/logrotate.d/tim-backups
/home/tim/backup/influx/influx_snapshot.tar.gz {
    su tim tim
    daily
    rotate 5
    copy
    nocompress
    ifempty
    missingok
    nocreate
    extension .tar.gz
}
/home/tim/backup/influx/influx_snapshot-monthly.tar.gz {
    su tim tim
    monthly
    rotate 5
    copy
    nocompress
    ifempty
    missingok
    extension .tar.gz
}
/home/tim/backup/grafana/grafana_snapshot.tar.gz {
    su tim tim
    daily
    rotate 5
    copy
    nocompress
    ifempty
    missingok
    nocreate
    extension .tar.gz
}
/home/tim/backup/grafana/grafana_snapshot-monthly.tar.gz {
    su tim tim
    monthly
    rotate 12
    copy
    nocompress
    ifempty
    missingok
    extension .tar.gz
}
EOF

Todo ¶

I could not get addextension and extension to work together, addextension could be used to indicate whether this is the daily or monthly backup.

Automate Home Assistant backup, ideally without resorting to root ownership to edit files. Possible solution (using most specific glob wildcard to reduce having false hits):

Move a HA backup to a specific path (NB this fails if there’s multiple backups - should use some find | xargs-construct instead)
Delete all (other) backups that we don’t need anymore

mv /var/lib/homeassistant/backups/????????.tar /home/tim/backup/homeassistant/homeassistant_snapshot.tar; rm /var/lib/homeassistant/backups/????????.tar

ls -At /var/lib/homeassistant/backups | head -n 1 | xargs -I {} mv '{}' /home/tim/backup/homeassistant/homeassistant_snapshot.tar && chown tim:tim /home/tim/backup/homeassistant/homeassistant_snapshot.tar 

mv /var/lib/homeassistant/backups/????????.tar /home/tim/backup/homeassistant/homeassistant_snapshot.tar && chown tim:tim /home/tim/backup/homeassistant/homeassistant_snapshot.tar && rm /var/lib/homeassistant/backups/????????.tar && ln /home/tim/backup/homeassistant/homeassistant_snapshot.tar /home/tim/backup/homeassistant/homeassistant_snapshot-monthly.tar

ln influx_snapshot.tar.gz influx_snapshot-monthly.tar.gz

#Debian #Linux #Server