This guide hopes to help anyone interested in having a try for compression & deduplication feature of btrfs in Qubes OS. Also serve as my personal note.
Make sure you back up everything before experiment and practicing.
# Btrfs, intro
Despite many added feature, btrfs is closer to ext4 rather than lvm - it is a tree of file and directories rooted at /, and can be mounted anywhere mostly.
Compression & Dedup features are enabled runtime rather than on filesystem creation.
Btrfs supports compression, and zstd is the best compression methods. By default it does not compress, and mount flag is required to enable the feature.
Btrfs supports offline deduplication, and this does not require mount flag. Rather you will need a userspace program (`duperemove` here) to find out duplicate blocks of all the files in the filesystem (some smart hash algorithms and data structures to accelerate the finding) and then tell btrfs modules where are the duplicate parts of the file, then btrfs module verify whether they are the same. Btrfs will attempt to dedup after verification and the disk usage reduces then. See relevant documentation for more details.
# Btrfs in Qubes OS, Intro
(Mainly R4.2 as I experiment btrfs only there)
When Qubes OS uses btrfs, the default setup of partition is like:
* EFI partition (/boot/efi, vfat)
* boot partition (/boot, ext4)
* luks'ed swap
* luks'ed btrfs (/)
Dom0 uses the whole btrfs partition.
VM images (root.img, private.img) are regular files under /var/lib/qubes and they are raw. They are probably the largest files in the whole file system. Other than that, the VM image file can be mostly sparse, there are no distinguishment of VM image file and other regular files in dom0. This is different as case in LVM, where dom0 root and VM images are different volumes in LVM.
This makes compression & deduplication mostly no different from other OS using btrfs.
# Recipe
1. Reinstall Qubes OS (R4.2). Of course a proper backup of current VMs are assumed here. It is suggested to do a TRIM on your disk before installing, as in many cases writing on dirty blocks can be slower in some devices.
2. When you get into the OS, do not be hurry to restore the backup. Follow the steps to organize the filesystem first.
3. Edit `/etc/fstab` to add `compress=zstd` or `compress-force=zstd` (I suggest the former, if heuristic disappoint you, you can do `chattr -m file; chattr +c file` any time), and then reboot.
* Note that when you set the compression flag, only new created file are compressed
4. `sudo qubes-dom0-update`, then `sudo qubes-dom0-update compsize duperemove`. These two are essential when we dedup (duperemove) and when we evaluate whether compression and deduplication works as expected (compsize to see the effect)
5. To compress existing file in dom0 by:
| This guide aims to assist anyone interested in exploring the compression and deduplication features of Btrfs within Qubes OS, while also serving as a personal reference.
Please ensure you back up all data before experimenting.
# Introduction to Btrfs
Btrfs, despite its numerous features, is more akin to ext4 than LVM. It organizes files and directories in a tree structure, rooted at /, and can be mounted almost anywhere.
Compression and deduplication in Btrfs can be enabled at runtime, rather than during filesystem creation.
Btrfs supports compression, with zstd being the most efficient method. By default, compression is disabled and must be activated via a mount flag.
For offline deduplication, Btrfs does not require a mount flag. Instead, a userspace tool (such as `duperemove`) is necessary to identify duplicate blocks across files using sophisticated hashing algorithms and data structures. This tool then informs the Btrfs module of any duplicates, which, after verification, *may* be deduplicated, potentially reducing disk usage.
# Btrfs in Qubes OS: An Overview
(Primarily based on my experiences with R4.2)
Qubes OS's default partition setup with Btrfs includes:
* An EFI partition (`/boot/efi`, vfat)
* A boot partition (`/boot`, ext4)
* Encrypted swap (luks)
* An encrypted Btrfs root partition (`/`, luks+btrfs)
Dom0 utilizes the entire Btrfs partition.
VM images (root.img, private.img) are stored as regular, albeit typically large, files under /var/lib/qubes and are in raw format.
Unlike with LVM, where dom0 root and VM images are separate volumes, here they are indistinguishable from other regular files in dom0. This distinction makes compression and deduplication in Qubes OS using Btrfs largely similar to their use in other OSes.
# My Recipe
1. **Reinstall Qubes OS (R4.2).** Make sure you have backed up any important VMs. It's recommended to perform a TRIM on your disk before actual installation, since it may be slower for some disk device to write on a dirty block already written before.
After installation, don't restore the backups immediately. Follow the steps to optimize the filesystem first.
2. **Change mount flags to enable compression.** Amend /etc/fstab to include `compress=zstd` or `compress-force=zstd` (the former is recommended; if unsatisfied, you can later use `chattr -m file; chattr +c file` to force compression on specific files), then reboot.
Note:
* Compression will only apply to files created or modified after this flag is set.
3. **Install btrfs utilities.** Execute `sudo qubes-dom0-update`, followed by `sudo qubes-dom0-update compsize duperemove` to install utilities for deduplication (with duperemove) and to assess the effectiveness of compression and deduplication (with compsize).
4. **Compress existing files in dom0.** Run:
|
| ```
We deliberately excludes VM images for now due to their size.
5. **Recompress the file incorrectly marked incompressible.** Verify compression status with `sudo lsattr <path>`. A file marked with the “m” flag indicates it was deemed incompressible after the initial compression attempt.
* Force compression if necessary using `sudo chattr -m <path> && sudo chattr +c <path>`.
* Recompress them by redo `btrfs filesystem defragment` with specific directory
* Files commonly shown incompressible includes various library files (libLLVMGold.so) and the kernel module disk (/var/lib/qubes/vm-kernels/*/modules.img)
6. **Evaluate your compression efforts.** Run `sudo compsize -x /`, which reveals the actual disk usage, including both compressed and uncompressed portions, and indicates any deduplication achieved.
7. **Shutdown VMs and clean up stale image files.** Prior to further operations, shut down all VMs and clean up temporary VM image files in /var/lib/qubes/{appvms,vm-templates}/*/*{Z,-precache.img}. Usually the only large files there should be “root.img” and “private.img”.
8. **Fill compression of filesystem.** Now you are familiar to the compression steps. Compress the filesystem entirely with:
|
| Here we do not compress VM image now as they can be very big and very time consuming.
After doing these, you can find many large file under /usr (example: libLLVMGold.so) and also the kernel module file at `/var/lib/qubes/vm-kernels/*/modules.img` uncompressed.
* To check out whether they are compressed quickly, use `sudo lsattr <path>`, and see if the file has "m" flag, which disables compression for the file. In brief btrfs compresses the first block of the file and find it hard to compress, so it marks the file as incompressible.
* To compress them by force, `sudo chattr -m <path> && sudo chattr +c <path>`
6. Check out how your compression is working by:
```
sudo compsize -x /
```
This can show the actual total disk usage of one path, along with the size of compressed and uncompressed part, and also it shows the deduplication result somehow.
7. Shutdown all vms. Then, remove all temp vm image files: `/var/lib/qubes/{appvms,vm-templates}/*/*{Z,-precache.img}`. Basically for each vm at most "root.img" and "private.img" are kept (there are some xmls, of course they should not be deleted). Whenever you want to dedup or recompress, doing this step is preferred.
8. Now you are familiar to compression and how to evaluate the filesystem, compress the whole filesystem by
```
| |
| ```
If VM images remain uncompressed, force their compression as step 5 says, and repeat the process.
9. **Deduplication.** Proceed with deduplication:
|
| Similarly, when you find that the VM images (under `/var/log/qubes/*/*/*.img`) are not compressed (as lsattr tells), mark it to be compressed forcefully and redo this step.
9. dedup.
```
| |
```
comments:
* `-A` does not work, do not use it
* `same` is important, `partial` is time consuming but also important as this can really dedup more in general case.
* hashfile is useful as a cache since you may dedup many times later
You can check your dedup effect with `compsize` or `btrfs filesystem df`
10. reboot, and install and update your new templates, and then dedup (compression is automatically done)
* maybe `e4defrag` your template vm root is also important. I will test later
11. restore your backup, and then dedup
| ```
Note:
* Avoid using the `-A` option as it does not work.
* The `same` option is crucial, while `partial` is more time-consuming but can enhance deduplication.
* The hashfile acts as a cache, useful for subsequent deduplications.
10. Review deduplication results with `sudo compsize -x /` or `sudo btrfs filesystem df /`.
11. After rebooting, install and update new templates, then deduplicate again (compression is automatic).
12. Consider defragmenting your template VM roots with `e4defrag`; further testing on this is pending.
13. Finally, restore your backups and perform another round of deduplication.
|
This guide should be updated later with more experiment
| This guide should be updated later with more experiment.
|