This is post number 2 in my FreeBSD Tribal Knowledge series. They may or may not be best practices, but they are my practices, given my homelab environment. I am a huge believer in backups. At work, I manage north of 7.5PB of storage, and have yet to (my lips to God's ears) lost any data.
So I do several levels of backups. I have been running a tool called BackupPC since about 2004. It is perl based, has a web interface, and has been bulletproof since I started using it. At the time, my homelab was fully linux. I continue to maintain the BackupPC install, first, because I still have a couple of linux boxes on my network that are not running ZFS, and second, because it gives me a diversity of solutions. I don't think anything bad is going to happen with ZFS, but I don't think it is a bad thing to have a couple of approaches, and I have the space to spare. Since I only have a couple of linux systems, I went ahead and put the BSD boxes in BackupPC...I have the bandwidth and space for it, having that second arrow in my backup quiver gives me a case of the warm fuzzies.
Then in 2014, I started converting to FreeBSD. With ZFS. On boot. I started by replacing my long-in-the-tooth file server to FreeNAS (post ixsystems' acquisition of the name). I also began with the FreeBSD-based desktop. Started dabbling with PC-BSD, which became TrueOS. When TrueOS went the way of the dodo, I decided, after years of running Debian instead of ubuntu or linux mint or any of the "curated" Debians, to self-curate. So I installed FreeBSD and installed a core set of packages. I am a KDE guy, so I wasn't interested in Gnome, because I have never gotten along with gnome over the years. Plus the developers saying "you will use it like we distribute it, and you will like it." I like KDE, with the ability to tune everything, yet having reasonable defaults.
In any case, I have multiple FreeBSD hosts and jails, and all except my Pi 3 are running ZFS. So I configured snapshots on all hosts. For my FreeBSD hosts, I use zfstools, though I may look for another tool. Currently, I have my snapshots set up as follows:
- Frequent, every quarter hour - keep 4
- Hourly - keep 24
- Daily - keep 7
- Weekly - keep 4
- monthly - keep 12
...and cleanup everything else. So I have snapshots that drop off in frequency, going back a year. I also keep weekly snapshots of my system, the base system and the userspace packages under /usr/local. FreeBSD has a concept of the base system, which is a minimal system that is smaller than most operating systems, but includes everything the system needs to operate. The userspace stuff all lives under the /usr/local tree, so is separate from the base system. Thus it is possible to run two different versions of ssh, or run openssl, while the base system runs libressl.
The other reason snapshots are superior to backuppc is that recovery is much easier and far, far quicker from a snapshot than from the backuppc web interface. Say, for example, something goes sideways wtih a package upgrade. I can not only reboot into my older boot environment, but suppose firefox (which scatters files all over one's home directory) was upgraded. What happens at that point is that when you roll back the boot environment, firefox will complain that a newer version of firefox has run and tell you to create a new profile. It is easier to go back to an earlier incarnation of it than to figure out what files need to get unchanged in all of those megs of firefox data. Because doing a
zfs rollback <datasetname>@<snapshotname>
takes 1-2 seconds, and you are back to the way your home directory looked before firefox changed files. And depending on the cadence of your frequent snapshots, then unless you have an extremely busy home directory, the before/after changes should be minimal.
So I also have two NAS boxes running. I had old hardware and wanted to test out other NAS distributions. But now, both are running zVault. I replicate the system and /home snapshots of the from all of the FreeBSD hosts to both NAS boxes. I also replicate the datasets from the primary NAS to the secondary one.
This way, I have local snapshots, plus archives of them, and at some point, the secondary NAS will be moving offsite...Which will be the the third leg of the "backup triad"...
The first leg of the triad is making backups at all, and as people wiser than I have said, one copy is not a backup. Or as my buddy Jimmy used to say, "two's one and one's none."
The second leg of the triad is storing backups away from the host, as in my case, saving the datasets from one nas on the other nas, and replicating critical datasets (like /home and the base system) to both nas boxes.
The third is offsite storage, so if the meteor hits your primary location, you are able to reconstitute your hosts.
So my recommendation is if you are using ZFS, make copious snapshots and push them offsite, and any other OSes, BackupPC is a good option. In my case, I do both for my FreeBSD hosts.