Simple config for faster storage for disk write intensive building environment via btrfs volumes revisions

Go back to topic: Simple config for faster storage for disk write intensive building environment via btrfs volumes

Revision #3

Edited on: 2023-09-07
Edited by user: deeplow

Revision #2

Edited on: 2022-09-14
Edited by user: logoerthiner

Qubes OS R4.1 by default uses LVM volume with snapshot for AppVM private storage, and by default the filesystem is ext4. A large amount of disk writes (for example extracting a linux kernel source code tarball) can be extremely slow.	BACKGROUND: Qubes OS R4.1 by default uses LVM volume with snapshot for AppVM private storage, and by default the filesystem is ext4. A large amount of disk writes (for example extracting a linux kernel source code tarball) can be extremely slow.
1. remove the snapshot layer. In dom0 the `lvcreate` following command can create a volume that is visible in `qvm-block` and `qvm-device block ls`	METHOD: 1. Create the LVM volume to remove the snapshot layer. In dom0 the `lvcreate` following command can create a volume that is visible in `qvm-block` and `qvm-device block ls`
~~# testvol1 -> volume name (new) lvcreate -V 100G --thinpool vm-pool qubes_dom0 -n testvol1 lvremove /dev/qubes_dom0/testvol1~~	# custom_testvol1 -> volume name (new) lvcreate -V 100G --thinpool vm-pool qubes_dom0 -n custom_testvol1 lvremove /dev/qubes_dom0/custom_testvol1 ``` Better set a common prefix for names of all custom volumes. 2. (Thanks to @rustybird) Edit dom0 config so that dom0 udev does not parse the block device. Qube can be untrusted, and they are free to edit the volume, which is why we should make sure that dom0 does not parse the volume filesystem by default. (Apply for Qubes R4.1) Create a file named `/usr/lib/udev/rules.d/20-skip-custom-lvm-devices.rules` in dom0, and insert the following content: ``` ACTION!="remove", SUBSYSTEM=="block", ENV{DM_UUID}=="LVM-", ENV{DM_LV_NAME}=="custom_" ENV{DM_UDEV_DISABLE_DISK_RULES_FLAG}="1", ENV{UDEV_DISABLE_PERSISTENT_STORAGE_RULES_FLAG}="1" ENV{QUBES_EXPORT_BLOCK_DEVICE}="1"
2. attach the block device onto the builder appvm. I usually uses the GUI approach though `qvm-block attach` is possible; when ro is required, `qvm-block attach --ro` seems to be the only solution. 3. in your appvm format the block device with the filesystem you like: if you only use it to store large files, ext4 is enough; if you need to store a swarm of small files, btrfs or zfs will be your choice. (btrfs is one of preinstalled kernel modules so can be used across various linux appvms - one only need a `mkfs.btrfs` initially; zfs must be installed manually in template vm and is harder to install) 4. mount it and try it out to see the performance improvement. This setup is advantageous mainly because	Note that `custom_*` is the prefix mentioned before, and edit it to fit your use case. 3. attach the block device onto the builder appvm. I usually uses the GUI approach though `qvm-block attach` is possible; when ro is required, `qvm-block attach --ro` seems to be the only solution. 4. format the volume. In your appvm format the block device with the filesystem you like: if you only use it to store large files, ext4 is enough; if you need to store a swarm of small files, btrfs or zfs will be your choice. (btrfs is one of preinstalled kernel modules so can be used across various linux appvms - one only need a `mkfs.btrfs` initially; zfs must be installed manually in template vm and is harder to install) 5. mount it and try it out to see the performance improvement. DISCUSSION: This setup is advantageous mainly because