As requested by @fsflover https://forum.qubes-os.org/t/is-qubes-4-3-rc1-useable/35530/14 I'll share how my migration went here. This howto is for advanced users (and beginners who want to learn/experiment). I am not an expert so please correct me where I went astray. Moderators: feel free to take this text and modify as if it's your own, as I use disposable logins and (probably) won't login as this one. Oh, and if you can activate all links, that would be sweet as I got this: "An error occurred: Sorry, new users can only put 2 links in a post." [and yes, I know I could make a permanent account, we've had this discussion years ago, this forum doesn't like disposibleUsers, and I don't like to be traced (even though I'm linking some of my previous posts to this one)]
I messed around with LVM's before which was often a struggle (e.g.: https://forum.qubes-os.org/t/secondary-storage-issues-for-maximum-redundancy/10864 ), but since I switched over to BTRFS, life has become so much more smooth.
Plan:
- take backups and snapshots
- create new partitions: temp-boot (1gb, ext2) and temp-btrfs (20+Gb, luks, btrfs). EFI partition will be overwritten
- install qubes: dom0 and VMs in separate subvolumes
- When qubes installer requests to reboot, do not let it reboot! Boot in liveUSB (or other system) first
- move dom0 and VMs subvolumes to main partition
- merge previous VMs and edit qubes.xml
- Then boot newly installed qubes and finish the setup.
- Take plenty of snapshots along the way so it's easy to roll back into a previous version if something goes wrong.
My laptop has 3 SSD's, two fast small ones and one large slow SSD.
All SSD's are bootable for maximum redundancy
I use the standard qvm-pool on the first SSD and added 2 qvm-pools (one COW and one noCOW) on the large slow SSD for large VMs and monerod-VM respectively.
- see documentation on secondary storage https://doc.qubes-os.org/en/latest/user/advanced-topics/secondary-storage.html
- BTRFS offers the option to turn copy-on-write (COW) off https://wiki.archlinux.org/title/Btrfs#Disabling_CoW which is handy for large files which grow incrimentally (like blockchans like http://karaparawplotu233ux7adtiaqswdthdk6kiqqovgy4ldp2dseijwnad.onion/wiki/monerod/monerod_basic_qubes_whonix_setup_en/ )
On every SSD I leave 10% empty space between the partition for over-provisioning, so every partition can grow/shrink when needed. (efi 512Mb, boot 1024Mb, rest luks with btrfs)
My current r4.2.4 has:
- [all SSDs] subvolume for swap, with a swapfile as explained here https://www.jwillikers.com/btrfs-swapfile
- [SSD0 & SSD1] subvolumes @dom0 and @varlibqubes
- [SSD2] subvolumes @dom0, @varlibqubes, @varlibqubes-SSD2 (with COW) and @varlibqubes-SSD2-noCOW
subvolumes @dom0 and @varlibqubes are regularly snapshotted and synced over all SSD's using the btrfs send/receive command, see https://wiki.archlinux.org/title/Btrfs#Send/receive
Installing r4.3:
before doing anything: take backups and snapshots (and synced to other drives)
Don't forget to back up /boot too
create subvolume @dom0-4.2.4 by taking a snapshot of @dom0
edit /boot/grub2/grub.cfg and find the section you're currently booted in
copy the whole menuentry {} section and paste it under
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
"rootflags=subvol=" and change the name from @dom0 to whatever subvolume name you've chosen. For me this is changing:
module2 /vmlinuz-6.15.10-1.qubes.fc37.x86_64 placeholder root=UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa ro rootflags=subvol=@dom0 rd.luks.uuid=luks-aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa plymouth.ignore-serial-consoles rhgb quiet rd.qubes.hide_all_usb
to
module2 /vmlinuz-6.15.10-1.qubes.fc37.x86_64 placeholder root=UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa ro rootflags=subvol=@dom0-4.2.4 rd.luks.uuid=luks-aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa plymouth.ignore-serial-consoles rhgb quiet rd.qubes.hide_all_usb
copy the whole menuentry {} section we've just adjusted again and past it in a file so we can add it to @dom0-4.3/etc/grub.d/40_custom later
Make a choice: Do you want to fork the data of all your VMs so changes under 4.3-rc1 won't affect VMs when booting in 4.2.4, or not?
If nothing goes wrong, then using @varlibqubes from r4.2.4 in r4.3-rc1 is the easiest as you don't end up with 2 different versions of your data. But if something goes wrong you'll have to roll-back from an older snapshot and (partially) start over.
to fork @varlibqubes, create snapshots, @varlibqubes-r4.2.4 and @varlibqubes-r4.3, and edit @dom0-4.2.4/etc/fstab.
Change line
UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@varlibqubes 0 0
into
UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@varlibqubes-r4.2.4 0 0
If you don't want to fork your data, no changes are needed.
I dunno if the next step is needed:
Change the names of the standard qubes: default-dvm, sys-firewall, sys-net, sys-usb, sys-whonix, anon-whonix, whonix-workstation-17-dvm, whonix-gateway-17, whonix-workstation-17, personal, vault, work, etc (to avoid conflicts with VM names during install of r4.3)
This step is probably not needed, but changing the names makes it easier later on to know which qubes were created in r4.2.4 and which one are brand new r4.3 ones (as I like to start with a clean slate).
Boot your newly created menuentry {} and test if your old r4.2.4 qubes works as before.
Boot a liveUSB (I chose an ubuntu flavor) and create 2 new partitions (I used gparted)
/dev/nvme0n1p3 1Gb, ext2, label: temp-boot
/dev/nvme0n1p4 50Gb, unformatted, label: temp-btrfs
root@lubuntu:~# cryptsetup luksFormat /dev/nvme0n1p4 root@lubuntu:~# cryptsetup luksOpen /dev/nvme0n1p4 some-mapper-name root@lubuntu:~# mkfs.btrfs -f -L temp-btrfs /dev/mapper/some-mapper-name
boot qubes installer, and when it asks where to install Qubes, select “advanced custom (blivet-gui)”
format /dev/nvme0n1p1 mountpoint=/boot/efi label=SSD0-efi
format /dev/nvme0n1p3 mountpoint=/boot label=temp-boot
unlock /dev/nvme0n1p4
create subvolume @dom0 mountpoint=/
create subvolume @varlibqubes mountpoint=/var/lib/qubes
When installer is finished, it'll ask to reboot. Make sure to boot a liveUSB!
Mount all partitions on first SSD.
(optional) remount main luks drive with compression: sudo mount -o remount,compress=zstd:3 /media/lubuntu/SSD0/
put subvolumes on main drive:
cd /media/lubuntu/temp-btrfs
btrfs su sn -r @dom0 @dom0-4.3-rc-preSetup-ro
btrfs send @dom0-4.3-rc-preSetup-ro | btrfs receive /media/lubuntu/SSD0/snapshots/qubes-4.3-rc1-new-install/
btrfs su sn -r @varlibqubes @varlibqubes-4.3-rc-preSetup-ro
btrfs send @varlibqubes-4.3-rc-preSetup-ro | btrfs receive /media/lubuntu/SSD0/snapshots/qubes-4.3-rc1-new-install/
cd /media/lubuntu/SSD0
btrfs su sn -r snapshots/qubes-4.3-rc1-new-install/@dom0-4.3-rc-preSetup-ro @dom0-4.3
@dom0-4.3 is our new pre-configured Qubes-OS, as the VMs are not created yet,
and @varlibqubes (or @varlibqubes-r4.3 for forked data) contains all VMs and templates as created in r4.2.4,
while @varlibqubes-4.3-rc-preSetup-ro has no templates nor VMs (yet) as it only contains:
-rw-rw---- 1 root qubes 989 @qubes-r4.3-rc-preSetup-ro/qubes.xml
-rw-r--r-- 1 root qubes 1.4G @qubes-r4.3-rc-preSetup-ro/template-packages/qubes-template-debian-13-xfce-4.3.0-202508092358.noarch.rpm
-rw-r--r-- 1 root qubes 1.9G @qubes-r4.3-rc-preSetup-ro/template-packages/qubes-template-fedora-42-xfce-4.3.0-202508092358.noarch.rpm
-rw-r--r-- 1 root qubes 473M @qubes-r4.3-rc-preSetup-ro/template-packages/qubes-template-whonix-gateway-17-4.3.0-202506031922.noarch.rpm
-rw-r--r-- 1 root qubes 791M @qubes-r4.3-rc-preSetup-ro/template-packages/qubes-template-whonix-workstation-17-4.3.0-202506031922.noarch.rpm
-rw-r--r-- 1 root root 139 @qubes-r4.3-rc-preSetup-ro/vm-kernels/6.12.39-1.fc41/default-kernelopts-common.txt
-rw-r--r-- 1 root root 7.2M @qubes-r4.3-rc-preSetup-ro/vm-kernels/6.12.39-1.fc41/initramfs
-rw-r--r-- 1 root root 0 @qubes-r4.3-rc-preSetup-ro/vm-kernels/6.12.39-1.fc41/memory-hotplug-supported
-rw-r--r-- 1 root root 694M @qubes-r4.3-rc-preSetup-ro/vm-kernels/6.12.39-1.fc41/modules.img
next step is to merge @varlibqubes-4.3-rc-preSetup-ro with @varlibqubes (or @varlibqubes-r4.3)
Open @varlibqubes as root with a file manager
copy directories template-packages and vm-kernels from @varlibqubes-4.3-rc-preSetup-ro to @varlibqube
rename @varlibqubes/qubes.xml to @varlibqubes/qubes-original-4.2.4.xml
copy @varlibqubes-4.3-rc-preSetup-ro/qubes.xml to @varlibqubes/qubes-original-4.3-presetup.xml
copy @varlibqubes/qubes-original-4.3-presetup.xml to @varlibqubes/qubes.xml
edit @varlibqubes/qubes.xml and open @varlibqubes/qubes-original-4.2.4.xml in a separate window
if you've created qvm-pools in your old setup, then adjust section:
<pools>
<pool name="varlibqubes" dir_path="/var/lib/qubes" driver="file-reflink" revisions_to_keep="1"/>
<pool name="linux-kernel" dir_path="/var/lib/qubes/vm-kernels" driver="linux-kernel"/>
</pools>
<pools>
<pool name="varlibqubes" dir_path="/var/lib/qubes" driver="file-reflink" revisions_to_keep="1"/>
<pool name="linux-kernel" dir_path="/var/lib/qubes/vm-kernels" driver="linux-kernel"/>
<pool name="varlibqubes_SSD2" dir_path="/var/lib/qubes_SSD2" driver="file-reflink" revisions_to_keep="0"/>
<pool name="varlibqubes_SSD2_noCOW" dir_path="/var/lib/qubes_SSD2_noCOW" driver="file-reflink" revisions_to_keep="0"/>
</pools>
@varlibqubes/qubes.xml we read:
<domains>
<domain id="domain-0" class="AdminVM">
<properties>
<property name="label">black</property>
</properties>
<features/>
<tags/>
</domain>
# <= copy/past all domains (except AdminVM (aka dom0)) from qubes-original-4.2.4.xml here
</domains>
</qubes>
Be carefull not to make any typo's. If we do this step correctly, then on the next reboot r4.3-rc1 will finish it's setup procedure, adding all newly created templates and VMs to our already existing VMs from r4.2.4.
Note: while this worked for the migration from r4.2.4 to r4.3-rc1, there is no guarantee this procedure will work in the future when migrating from, let's say r4.8.9 to r5.0.
Before we can boot, we need to make more changes:
I like to change the UUIDs to make them more human-readable. Useful commands:
mlabel -s -N ef1aaaaa :: -i '/dev/nvme0n1p1'
tune2fs -U aaaaaaaa-0000-0002-0000-aaaaaaaaaaaa /dev/nvme0n1p2
cryptsetup' with '--uuid=aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa
mkfs.btrfs -f -L mylabel -U aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /dev/mapper/ssd0_crypt, or
btrfstune -U aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /dev/mapper/ssd0_crypt
edit @dom0-4.3/etc/crypttab:
#└─nvme0n1p8 aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa
luks-aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa UUID=aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa none
#└─nvme1n1p8 bbbbbbbb-0000-0008-0000-bbbbbbbbbbbb
luks-bbbbbbbb-0000-0008-0000-bbbbbbbbbbbb UUID=bbbbbbbb-0000-0008-0000-bbbbbbbbbbbb none
#└─sda8 cccccccc-0000-0008-0000-cccccccccccc
luks-cccccccc-0000-0008-0000-cccccccccccc UUID=cccccccc-0000-0008-0000-cccccccccccc none
edit @dom0-4.3/etc/fstab
UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa / btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@dom0-4.3 0 0
UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /var/lib/qubes btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@varlibqubes 0 0
UUID=cccccccc-bbbb-0008-bbbb-cccccccccccc /var/lib/qubes_SSD2 btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@varlibqubes-SSD2 0 0
UUID=cccccccc-bbbb-0008-bbbb-cccccccccccc /var/lib/qubes_SSD2_noCOW btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,subvol=@varlibqubes-SSD2-noCOW 0 0
#
#UUID=bbbbbbbb-bbbb-0008-bbbb-bbbbbbbbbbbb / btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@dom0-4.3 0 0
#UUID=bbbbbbbb-bbbb-0008-bbbb-bbbbbbbbbbbb /var/lib/qubes btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@varlibqubes 0 0
#
#UUID=cccccccc-bbbb-0008-bbbb-cccccccccccc / btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@dom0-4.3 0 0
#UUID=cccccccc-bbbb-0008-bbbb-cccccccccccc /var/lib/qubes btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3,subvol=@varlibqubes 0 0
#
UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /swap btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
#UUID=bbbbbbbb-bbbb-0008-bbbb-bbbbbbbbbbbb /swap btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
#UUID=cccccccc-bbbb-0008-bbbb-cccccccccccc /swap btrfs defaults,x-systemd.device-timeout=0,discard,noatime,ssd,space_cache,subvol=@swap 0 0
/swap/swapfile none swap defaults 0 0
#
UUID=aaaaaaaa-bbbb-0002-bbbb-aaaaaaaaaaaa /media/nvme0-boot ext2 defaults 1 2
UUID=EF1A-AAAA /media/nvme0-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
#UUID=bbbbbbbb-bbbb-0002-bbbb-bbbbbbbbbbbb /media/nvme1-boot ext2 defaults 1 2
#UUID=EF1B-BBBB /media/nvme1-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
#UUID=cccccccc-bbbb-0002-bbbb-cccccccccccc /media/sda-boot ext2 defaults 1 2
#UUID=EF1C-CCCC /media/sda-boot/efi vfat umask=0077,shortname=winnt,discard 0 2
#
/media/nvme0-boot /boot none defaults,bind 0 0
/media/nvme0-boot/efi /boot/efi none defaults,bind 0 0
#
UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa /media/aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3 0 0
UUID=bbbbbbbb-bbbb-0008-bbbb-bbbbbbbbbbbb /media/bbbbbbbb-bbbb-0008-bbbb-bbbbbbbbbbbb btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3 0 0
UUID=cccccccc-bbbb-0008-bbbb-cccccccccccc /media/cccccccc-bbbb-0008-bbbb-cccccccccccc btrfs defaults,nofail,x-systemd.device-timeout=0,discard=async,noatime,ssd,compress=zstd:3 0 0
after changing /etc/fstab, run findmnt --verify --verbose
edit @dom0-4.3/etc/default/grub
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa plymouth.ignore-serial-consoles rhgb quiet"
#GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-bbbbbbbb-0000-0008-0000-bbbbbbbbbbbb plymouth.ignore-serial-consoles rhgb quiet"
#GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-cccccccc-0000-0008-0000-cccccccccccc plymouth.ignore-serial-consoles rhgb quiet"
edit /boot/efi/EFI/qubes/grub.cfg
search --no-floppy --fs-uuid --set=dev aaa00004-0000-0002-0000-000000040aaa # boots /dev/nvme0n1p2
#search --no-floppy --fs-uuid --set=dev bbb00002-0000-0002-0000-000000020bbb # uncomment in /dev/nvme1n1p1 to boot /dev/nvme1n1p2
#search --no-floppy --fs-uuid --set=dev ccc00008-0000-0002-0000-000000080ccc # uncomment in /dev/sda1 to boot /dev/sda2
set prefix=($dev)/grub2
export $prefix
configfile $prefix/grub.cfg
edit (/dev/nvme0n1p2)/boot/grub2/grub.cfg and add the working menuentry {} from (/dev/nvme0n1p3)/boot/grub2/grub.cfg and change all UUIDs referring to /dev/nvme0n1p3 and /dev/nvme0n1p4 to /dev/nvme0n1p2 and /dev/nvme0n1p8 (or whatever number your main luks partition is)
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry 'Qubes 4.2.4 ...' {
...
}
menuentry 'Qubes 4.3-rc1, with Xen hypervisor' --class qubes --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa' {
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root aaaaaaaa-0000-0002-0000-aaaaaaaaaaaa
echo 'Loading Xen 4.19.2 ...'
if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
xen_rm_opts=
else
xen_rm_opts="no-real-mode edd=off"
fi
insmod multiboot2
multiboot2 /xen-4.19.2.gz placeholder console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096 ${xen_rm_opts}
echo 'Loading Linux 6.12.39-1.qubes.fc41.x86_64 ...'
module2 /vmlinuz-6.12.39-1.qubes.fc41.x86_64 placeholder root=UUID=aaaaaaaa-bbbb-0008-bbbb-aaaaaaaaaaaa ro rootflags=subvol=@dom0-4.3 rd.luks.uuid=luks-aaaaaaaa-0000-0008-0000-aaaaaaaaaaaa plymouth.ignore-serial-consoles rhgb quiet
echo 'Loading initial ramdisk ...'
insmod multiboot2
module2 --nounzip /initramfs-6.12.39-1.qubes.fc41.x86_64.img
}
/boot/grub2/grub.cfg are only temporarily (until grub updates, overwriting the file), therefore also add both menuentries to @dom0-4.3/etc/grub.d/40_custom :
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry 'Qubes 4.2.4 ...' {
...
}
menuentry 'Qubes 4.3-rc1 ...' {
...
}
That way, if dom0's kernel gets upgaded, or one runs grub2-mkconfig -o /boot/grub2/grub.cfg both versions can be booted.
Copy all files needed by menuentry from (/dev/nvme0n1p3)/boot/ to (/dev/nvme0n1p2)/boot/
note: if on r4.2.4 you've installed the latest kernel (to make some hardware work), then you'll end up with a r4.2.4 kernel with a higher number than the standard r4.3-rc1 kernel:
problem is that grub2's config maker takes the latest kernel number to insert in themy old r4.2.4 has : initramfs-6.15.10-1.qubes.fc37.x86_64.img r4.3-rc1 comes with : initramfs-6.12.39-1.qubes.fc41.x86_64.img after upgrading dom0 : initramfs-6.12.42-1.qubes.fc41.x86_64.imgmenuentry {}, meaning that r4.3-rc1 useskernel 6.15.10-1.qubes.fc37, which makes my machine fail to boot. Therefore I move r4.2.4's boot files in a temporary directory before upgrading@dom0-4.3and put the boot files back afterwards.
If you have multiple disks and want redundancy like I do, use the btrfs send/receive to copy or sync the subvolumes on the different disks, and edit the files mentioned above on every drive
Finishing up:
- remove temporary partitions temp-boot and temp-btrfs, either via blivet-gui in dom0, or gparted in liveUSB. (or scramble with random data if your risk-level is high)
- activate swap, see https://www.jwillikers.com/btrfs-swapfile
- to prevent unnecessary disk-writes (important for SSD’s), set swappiness=1
in dom0, debian-13 and fedora-42: nano /etc/sysctl.d/50-usersettings.conf and add
vm.swappiness = 1
in whonix based templates: nano /etc/sysctl.conf and add
vm.swappiness = 1
comments/improvements are welcome I did this migration a few days ago. I haven't set up swap yet, nor made too many adjustments. So far everything seems to work. Old VMs with old debian-12 templates just work (even though I read somewhere r4.3 doesn't support debian-12 or something). And old VMs converted to debian-13 also work in dom0-r4.2.4 It also feels like writing down this howto took longer than the actual migration. Scrolling through this howto seems daunting... but it's pretty straight forward if you understand what every step does. If something doesn't work, check for typo's 😢 Good luck and have fun! 😊