Following on from: https://forum.qubes-os.org/t/simple-zfs-mirror-on-qubes-4-3/38703/2
I went ahead and organized my notes for posterity. This isn't a replacement for @Rudd-O's guide, which is super helpful, rather an annotation/overlay. If you're referring to these notes to make system changes, you should be referring to his guide first.
Here's what I have differently than the scenario in the guide:
- BIOS boot (coreboot+Heads) instead of EFI
- NVMe disks instead of SATA (so: /dev/nvme0n1 paths instead of /dev/sda, otherwise no different)
- Goal of a pool consisting of a mirror vdev rather than a pool of two single device vdevs. Only matters at the end. The process is also the same for raidz1 etc.
- Qubes 4.3 (ZFS software available through qubes-dom0-update) instead of early 4.2 (manual build). Simplies installation quite a lot.
Fair warning: the process, especially the ZFS migration, is not trivial. If you try it yourself, likely issues will pop up that you will have to troubleshoot.
Warning 2: probably best to do this on a fresh install, before restoring a backup. The qvm-migrate script is fast enough when there are few installed VMs but quite slow if there are many.
I installed Qubes 4.3 onto one of my two SSDs. Normal process.
For clarity I will call the installation SSD "Foo" and the other drive "Bar". I can't refer to these as /dev/nvme0n1 and /dev/nvme1n1 because the dev assignments can switch across boots.
You can read through the link for past context but in 4.3 you only need to run the commands below.
sudo qubes-dom0-update zfs # Note: also installs zfs-dkms as a dependency
# Warning: installs a LOT of packages to dom0.
# For /boot, and later maybe also dom0 root:
sudo qubes-dom0-update rsync
# If you do plan to migrate dom0 to ZFS, then:
sudo qubes-dom0-update zfs-dracut
I used blivet-gui to partition Bar such that there are /dev/nvme1n1p1 and /dev/nvme1n1p2 of size and format to match the partitions the Qubes installer created on Foo, /dev/nvme1n0p1 (1MB BIOS boot) and /dev/nvme1n0p2 (the /boot mountpoint). And the rest of Bar as a large regular partition (not LVM2, unlike Foo), with Encrypt checked to LUKS-ify it. Use the same encryption password you did during Part 0.
(I skip the cryptsetup luksFormat and cryptsetup luksOpen commands from the guide, as blivet-gui already did them for me.)
Run the following commands as root. (Ditto for the rest of the post, I think)
Synchronize Bar's boot to Foo's boot:
mkdir -p /tmp/x
mount /dev/nvme1n1p2 /tmp/x
rsync -vaxHAXSP /boot/ /tmp/x
umount /tmp/x
/boot/efi/ stuff from the guide because it's not required for a BIOS boot system
Set Bar's new, empty encrypted partition to be decrypted at boot:
dev=`blkid -s UUID -o value /dev/nvme1n1p3`
echo luks-$dev UUID=$dev none discard >> /etc/crypttab
Create the ZFS pool on Bar's new, empty encrypted partition:
zpool create -O atime=off -O relatime=off laptop /dev/disk/by-id/dm-uuid-*-luks-`blkid -s UUID -o value /dev/nvme1n1p3`
atime and relatime as a minor optimization. Not required.
zfs set mountpoint=none laptop
zpool status
Create the new Qubes pool, backed by the ZFS pool:
qvm-pool add -o container=laptop/vm-pool vm-pool-zfs zfs
laptop named vm-pool. Verify with zfs list.
- Note: the Qubes pool is named vm-pool-zfs while the backing ZFS dataset is named vm-pool. This makes sense to me but they can be named as you like.
Check your work:
qvm-pool list
qvm-pool info vm-pool-zfs
Set this pool to be the default pool for new qubes:
qubes-prefs default_pool vm-pool-zfs
# Note: also sets:
# default_pool_private D vm-pool-zfs
# default_pool_root D vm-pool-zfs
# default_pool_volatile D vm-pool-zfs
Use Rudd-O's qvm-migrate script to migrate all installed VMs (the underlying volumes) from the old default vm-pool to vm-pool-zfs. The script is still ok for 4.3 and it helps a lot; but, this is still error-prone, manual work. Lessons:
- Before you start, shut down everything except sys-usb (and dom0). If you don't need sys-usb for keyboard, then shut it down too.
- Everything that touches sys-usb is dicey (if you need it for input); you need it to be shutdown while you migrate 1) its template, 2) its default_dispvm, and 3) sys-usb itself.
- When you're done, check qvm-pool list to make sure all volumes are actually migrated. For me, sys-usb's volatile volume did not transfer to the new pool on my first attempt, not sure why, but probably user error.
I did something like this when migrating sys-usb:
# qvm-shutdown --wait --force sys-usb ; ./m.bash sys-usb ; qvm-start sys-usb ; qvm-start sys-usb-zfs
qvm-start sys-usb-zfs at the end just in case something goes wrong half way)
Here was my m.bash:
#!/bin/bash
set -e
set -x
vm="$1"
qvm-check -q "$vm"
qvm-check -q --running "$vm" && exit 1
~/qvm-migrate.bash $vm ${vm}-zfs
~/qvm-migrate.bash ${vm}-zfs $vm
(sudo zfs destroy -r laptop/vm-pool/${vm}-zfs || true)
qvm-service --disable $vm qubes-update-check
qvm-run -a -p --nogui $1 'sudo fstrim -v / ; sudo fstrim -v /rw' logic from the Optional section of the guide, as starting a qube can cause a chain of networking VMs to start up, and now you're playing whack-a-mole.
Start a few qubes and/or reboot now just to see if the system is currently sound, which it should be.
Then, delete the old Qubes pool:
qvm-pool remove vm-pool
As things stand: - Foo: used for booting and for dom0 - Bar: used to host the ZFS pool backing the Qubes pool for all VMs except dom0
Install zfs-dracut if you didn't install it before:
sudo qubes-dom0-update zfs-dracut
Probably should install Rudd-O's grub-zfs-fixer package too. There are many versions of the grub-zfs-fixer package listed on Rudd-O's index: https://repo.rudd-o.com/q4.1/packages/ . I downloaded the one with the most recent version, grub-zfs-fixer-0.0.7-23.fc32.noarch.rpm, and the one with the most recent timestamp, grub-zfs-fixer-0.0.7-19.fc32.noarch.rpm. The extracted RPMs are functionally identical. This package adds small innocuous patches to three files:
- /usr/sbin/grub2-mkconfig
- /etc/grub.d/10_linux
- /etc/grub.d/20_linux_xen
Without the patch to grub2-mkconfig, I got an error later on that I will point out.
Create a toplevel dataset for dom0 root in the ZFS pool on Bar:
zfs create -o mountpoint=/laptop -p laptop/ROOT/os
ROOT" seems to be the convention for the parent dataset of OS root filesystems
- "/os" could instead be /qubes-os, or whatever you like, as long as you're consistent
Create swap:
zfs create -o primarycache=metadata -o compression=zle \
-o logbias=throughput -o sync=always \
-o secondarycache=none \
-o com.sun:auto-snapshot=false -V 8G laptop/dom0-swap
mkswap /dev/zvol/laptop/dom0-swap
Sync to the live root (!):
rsync -vaxHAXSP / /laptop/ ; rmdir /laptop/laptop
Make fstab edits for the new /boot, /, and swap on Bar, as in the guide:
vim /laptop/etc/fstab
/boot/efi/ stuff because it's not required for a BIOS boot system
Mounting and GRUB stuff:
mount --bind /dev /laptop/dev
mount --bind /proc /laptop/proc
mount --bind /sys /laptop/sys
mount --bind /run /laptop/run
mount /dev/nvme0n1p2 /laptop/boot
chroot /laptop grub2-mkconfig -o /boot/grub2/grub.cfg
Here I got an error:
/usr/sbin/grub2-probe: error: cannot find a device for / (is /dev mounted?).
grub-zfs-fixer rectifies. If you get this error, install the package and rerun chroot /laptop grub2-mkconfig -o /boot/grub2/grub.cfg.
Tweak grub.cfg per the guide:
vim /laptop/boot/grub2/grub.cfg
root=laptop/ROOT/os". However, for me it showed root=ZFS=laptop/ROOT/os. I kept the ZFS= and that worked, so maybe there is a typo in the guide here, or maybe it works either way.
Remove or comment out the old luks-* entry in crypttab that was created during installation (Part 0).
vim /laptop/etc/crypttab
Recreate the RAM disk:
chroot /laptop dracut -fv --regenerate-all
...
dracut[I]: *** Including module: zfs ***
dracut-install: Failed to find module 'zfs'
dracut[E]: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.SPCl6s/initramfs --kerneldir /lib/modules/6.12.59-1.qubes.fc41.x86_64/ -m zfs
...
dracut[I]: *** Including module: zfs ***
dracut-install: Failed to find module 'zfs'
dracut[E]: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.CTIOsr/initramfs --kerneldir /lib/modules/6.12.63-1.qubes.fc41.x86_64/ -m zfs
...
zfs module for my running kernel and did produce /boot/6.17.9-1.qubes.fc41.x86_64.img.
I skipped the efibootmgr stuff, because BIOS boot.
Now, reboot. Hopefully the computer will choose to boot from Bar instead of Foo. You will know if it chose the 'wrong' disk because after logging in you will have no qubes except dom0. If so, reboot again and maybe choose the boot drive manually.
Follow the export/import commands in 'Unlock the system and set the root file system mount point'. Then double check the right things are mounted:
mount | grep zfs
mount | grep /boot
Create the GRUB config once again:
grub2-mkconfig -o /boot/grub2/grub.cfg
For me it was normal. At this point I used blivet-gui to delete the /boot partition on Foo, to avoid confusion later, and created a new empty/filler partition of the same size/extent in its place.
The next and nearly last step is to repurpose the rest of Foo to create the mirror for the ZFS pool currently hosted only on Bar.
I used blivet-gui to delete the old LUKS+LVM2 partition on Foo, and created a new partition in the same space, unformatted, Encrypted, same password as before. This partition should have the same (or greater) size/extent as the the LUKS partition on Bar. (As before, I skip the cryptsetup luksFormat and cryptsetup luksOpen commands from the guide, as blivet-gui already did them for me.)
Add a new entry to crypttab for this encrypted disk:
vim /etc/crypttab
Before creating the mirror, check the status of the pool:
user@dom0:~$ zpool status
pool: laptop
state: ONLINE
...
config:
NAME STATE READ WRITE CKSUM
laptop ONLINE 0 0 0
dm-uuid-CRYPT-LUKS2-blahBARblah ONLINE 0 0 0
errors: No known data errors
Magically create the mirror by attaching the new LUKS partition to the single device vdev:
zpool attach laptop /dev/disk/by-id/<device path for Bar's LUKS partition> /dev/disk/by-id/<device path for Foo's LUKS partition>
See that laptop consists of a mirror vdev now, and resilvering is in process:
user@dom0:~$ zpool status
pool: laptop
state: ONLINE
...
config:
NAME STATE READ WRITE CKSUM
laptop ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
dm-uuid-CRYPT-LUKS2-blahBARblah ONLINE 0 0 0
dm-uuid-CRYPT-LUKS2-blahFOOblah ONLINE 0 0 0 (resilvering)
errors: No known data errors
Recreate the RAM disk:
dracut -fv --regenerate-all
Add the new Foo LUKS partition to GRUB_CMDLINE_LINUX, as in the guide:
vim /etc/defaults/grub
Create the GRUB config a last time:
grub2-mkconfig -o /boot/grub2/grub.cfg
Once the resilvering is done, reboot to see all is well 👍
Do you see something like this (I did)?
user@dom0:~$ zpool status
pool: laptop
state: ONLINE
status: Mismatch between pool hostid and system hostid on imported pool.
This pool was previously imported into a system with a different hostid,
and then was verbatim imported into this system.
action: Export this pool on all systems on which it is imported.
Then import it to correct the mismatch.
...
root# zgenhostid "$(hostid)"