Hi all, As I am sure I will have forgotten what I've done when the next version of Qubes rolls out, I wrote this howto for myself, and as a help for anyone who has similar ideas. Goal: build a system that is redundant like raid, but that, unlike raid, doesn't wear down all drives. In other words: have an unused hot-swap disk on standby, which doesn't wear out as it is not used My laptop has: 1 x M.2 2280 PCIe Gen4x4 1 x M.2 2280 PCIe Gen3x4 1 x 2.5″ SATA HDD support

I tried LVM raid, which worked well, until I removed one of the ssd's as a test, then the laptop turns into a brick. So I opted for btrfs instead

What I did: use a different liveUSB to partition every SSD, leaving 10% empty space between every partition for over-provisioning, so every partition can grow/shrink when needed. (efi 256Mb, boot 512Mb, rest luks with btrfs) Useful commands to make the UUID more human readable: mlabel -s -N ef1aaaaa :: -i '/dev/nvme0n1p1' tune2fs -U aaaaaaaa-0000-0002-0000-aaaaaaaaaaaa /dev/nvme0n1p2 cryptsetup' with '--uuid=aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa mkfs.btrfs -f -L mylabel -U aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /dev/mapper/ssd0_crypt, or btrfstune -U aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /dev/mapper/ssd0_crypt

create a subvolume for swap, with a swapfile as explained here https://www.jwillikers.com/btrfs-swapfile create subvolumes @qubes-4.1.2-dom0 and @qubes-4.1.2-varlibqubes and @qubes-4.2.0-dom0 and @qubes-4.2.0-varlibqubes (tested with 4.2.0-RC2 and 4.2.0 (leaving previous R4.1.2 as is)) make sure to chown and chmod @qubes-4.X-varlibqubes: chown root:qubes @qubes-4.2.0-varlibqubes/ chmod 2770 @qubes-4.2.0-varlibqubes/ boot qubes installer, and when it asks where to install Qubes, select "advanced custom (blivet-gui)" Format /boot and /boot/efi with the labels you want (remember to change the UUID later) decrypt the luks container select @qubes-4.X-dom0 as / select @qubes-4.X-varlibqubes as /var/lib/qubes => if that fails, only set @qubes-4.X-dom0 as / ignore swap, as Anaconda can't handle swap-files (or I don't know how to) => if anaconda won't install qubes, then start over, reformat the top-level btrfs and install qubes in newly created subvolume or on the top level. finish installation as normal

Make sure that after the install, and before the post-setup, to boot a liveUSB and change: nano @qubes-4.2.0-dom0/root/anaconda-ks.cfg and comment out all lines refering to btrfs subvolumes, like: 

#btrfs none --noformat --useexisting btrfs.329
#btrfs /var/lib/qubes --subvol --name=@qubes-4.2.0-varlibqubes 
#btrfs / --subvol --name=@qubes-4.2.0-dom0

If you forgot this step, and ended up with an empty qubes without any templates/appvms... then change /root/anaconda-ks.cfg now and rerun /usr/libexec/initial-setup/initial-setup-graphical manually

Remember to take plenty of snapshots along the way so you can roll back if something went wrong. btrfs su sn -r @qubes-4.X-dom0 snapshots/@qubes-4.X-dom0-<date|time>-ro # (same for @qubes-4.X-<date|time>-varlibqubes)

[next step was needed for 4.2.0-RC2, but not needed for 4.2.0] If mounting @qubes-4.X-varlibqubes on /var/lib/qubes during anaconda-install failed, then: 

cd @qubes-4.2.0-dom0/var/lib/qubes/
cp -a --reflink=always * /path/to/\@qubes-4.2.0-varlibqubes/ 
rm -Rf *

and mount @qubes-4.2.0-varlibqubes on /var/lib/qubes in /etc/fstab

Go back to https://www.jwillikers.com/btrfs-swapfile and finish off what Anaconda couldn't do automatically to prevent unnecessary disk-writes (important for SSD's), set 'swappiness=1' in dom0: nano /etc/sysctl.d/50-usersettings.conf # and add vm.swappiness = 1 in debian (whonix) based templates: nano /etc/sysctl.conf# and add vm.swappiness = 1

edit /etc/crypttab: 

 #└─nvme0n1p3  259:4    0  3.3T  0 part         aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa
 luks-aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa UUID=aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa none
 #└─nvme1n1p3  259:9    0  1.7T  0 part         bbbbbbbb-0000-0003-0000-bbbbbbbbbbbb
 luks-bbbbbbbb-0000-0003-0000-bbbbbbbbbbbb UUID=bbbbbbbb-0000-0003-0000-bbbbbbbbbbbb none
 #└─sda3       259:0    0  6.6T  0 part         cccccccc-0000-0003-0000-cccccccccccc
 luks-cccccccc-0000-0003-0000-cccccccccccc UUID=cccccccc-0000-0003-0000-cccccccccccc none
edit /etc/fstab 
 #UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /              btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.1.2-dom0             0 0
 #UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.1.2-varlibqubes      0 0
  UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /              btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-dom0             0 0
  UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-varlibqubes      0 0
 #
 #UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /              btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-dom0             0 0
 #UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /var/lib/qubes btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-varlibqubes      0 0
 #
 #UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /              btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-dom0             0 0

    #UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /var/lib/qube  btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache,subvol=@qubes-4.2.1-varlibqubes      0 0

 #
  UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /swap          btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
 #UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /swap          btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
 #UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /swap          btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0

     /swap/swapfile                            none           swap  defaults                                                                         0 0

 #
  UUID=aaaaaaaa-bbbb-0002-bbbb-aaaaaaaaaaaa /media/nvme0-boot     ext2 defaults                           1 2
  UUID=EF1A-AAAA                            /media/nvme0-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
 #UUID=bbbbbbbb-bbbb-0002-bbbb-bbbbbbbbbbbb /media/nvme1-boot     ext2 defaults                           1 2
 #UUID=EF1B-BBBB                            /media/nvme1-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
 #UUID=cccccccc-bbbb-0002-bbbb-cccccccccccc /media/sda-boot  ext2 defaults                           1 2
 #UUID=EF1C-CCCC                            /media/sda-boot/efi   vfat umask=0077,shortname=winnt,discard 0 2
 #
 # /media/nvme0-boot                         /boot                 none defaults,bind                     0 0
 # /media/nvme0-boot/efi                     /boot/efi             none defaults,bind                     0 0
 #
  UUID=aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa /media/aaaaaaaa-bbbb-0003-bbbb-aaaaaaaaaaaa   btrfs   defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache 0 0
  UUID=bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb /media/bbbbbbbb-bbbb-0003-bbbb-bbbbbbbbbbbb   btrfs   defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache 0 0
  UUID=cccccccc-bbbb-0003-bbbb-cccccccccccc /media/cccccccc-bbbb-0003-bbbb-cccccccccccc   btrfs   defaults,x-systemd.device-timeout=0,discard,noatime,ssd,compress=zstd:3,space_cache 0 0
after changing /etc/fstab, run findmnt --verify --verbose

edit /etc/default/grub 

GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-aaaaaaaa-0000-0003-0000-aaaaaaaaaaaa plymouth.ignore-serial-consoles 6.6.2-1.qubes.fc37.x86_64 x86_64 rhgb quiet"
#GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-bbbbbbbb-0000-0033-0000-bbbbbbbbbbbb plymouth.ignore-serial-consoles 6.6.2-1.qubes.fc37.x86_64 x86_64 rhgb quiet"
#GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-cccccccc-0000-0033-0000-cccccccccccc plymouth.ignore-serial-consoles 6.6.2-1.qubes.fc37.x86_64 x86_64 rhgb quiet"
Make sure to make plenty of snapshots along the way. And when dom0 is updated and customized as you want, send your final snapshots to the other disks: 
btrfs su sn -r @qubes4.1.2-dom0 snapshots/@qubes4.1.2-dom0-<date/version>-ro
btrfs send snapshots/@qubes4.1.2-dom0-<date/version>-ro | btrfs receive /path/to/other/SSDs # `https://wiki.archlinux.org/title/Btrfs#Send/receive`
btrfs su sn /path/to/other/SSD/snapshots/@qubes4.1.2-dom0-<date/version>-ro /path/to/other/SSD/@qubes4.1.2-dom0
After moving/copying qubes to a new disk, or after changing the uuid, or subvolume-name, make sure to update the correct uuid/subvol in: - /etc/fstab - /boot/efi/EFI/qubes/grub.cfg => the line 'search --no-floppy --fs-uuid --set=dev XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX' - /etc/default/grub => the line 'GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"' - /etc/grub.d/40_custom and rerun grub2-mkconfig -o /boot/grub2.grub.cfg

test setup by booting in every dom0 on every disk. physically remove one or more disks and test if laptop still boots when one or more disks fails. (make sure to comment out the references to the removd disk in '/etc/fstab)

Keep in mind: this approach does not replace backups!

compress=zstd:3 turns dom0 from 4Gb to 2Gb compsize original installed 4.2.0 

Processed 176186 files, 82608 regular extents (88510 refs), 73123 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       79%      4.0G         5.1G         5.7G       
none       100%      3.7G         3.7G         4.2G       
zstd        23%      341M         1.4G         1.4G       
prealloc   100%       28K          28K          16M
compsize same 4.2.0 transferred to other SSD and after upgrading to 4.2.1 
Processed 176178 files, 88613 regular extents (96716 refs), 83239 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       40%      2.0G         5.1G         5.7G       
none       100%      781M         781M         816M       
zstd        30%      1.3G         4.3G         4.9G
Problems I wasn't able to solve: Template with btrfs as root filesystem: I tried @51lieal suggestion of https://forum.qubes-os.org/t/btrfs-for-template-appvm/12280/2 but I was unable to make it work. It's simply too complicated for my simple mind :-s. Would anyone be willing to create a, preferably, debian-12-xfce template and distribute it through one of the official qubes repositories? Alternatively... mounting a @subvolume inside a VM. => when, for example, cccccccc-bbbb-0003-bbbb-cccccccccccc is not mounted in dom0, then I am able to mount the whole disk (top-level) into any VM. But I am unable to mount a @subvolume of a, in dom0 mounted, btrfs partition. Creating an image file, formatting as btrfs and mounting inside a VM works. But I don't want to allocate disk-space to a file... I would like to mount a specific subvolume inside an appVM. Any tips on how to do this?

In any case, after playing around with LVM and BTRFS for a while, in my opinion, BTRFS is easier to manage, and not having to allocate disk space up front, is a huge advantage. E.g. on my old laptop, one time I didn't have enough room on my root-pool which can be increased, if there's enough disk space left. But since I couldn't decrease the size of the vm-pool..., I ended up having to re-install qubes, restoring all the VM's from backup, which is a problem that doesn't exist with BTRFS.

Does anyone know in which system file the appVM's UUID are stored? That would be handy in case one has to swap over one varlibqube-subvolume to a different dom0-subvolume If I made any errors, or there are better ways to achieve what I did, please let us know. Thanks! Hopefully this is useful to someone.