Create a Gaming HVM

Original forum link

https://forum.qubes-os.org/t/19000

Original poster

neowutran

Editors

apparatus, deeplow, gonzalo-bulnes, LinuxUser, system, taradiddles

Created at

2023-06-02 05:19:43

Last wiki edit

2025-01-17 09:26:13

Revisions

41 revisions

Posts count

309

Likes count

153

Tags

configuration, dom0, hvm, version-r41

Hardware

To have an 'HVM' for gaming, you must have

• A dedicated GPU. By dedicated, it means: it is a secondary GPU, not the GPU used to display dom0. In 2023, 'Nvidia' and 'Amd' GPU work. Not tested with Intel GPUs. External GPU using thunderbolt work (https://forum.qubes-os.org/t/create-a-gaming-hvm/19000/8)

• A lot of patience. GPU passthrough is not trivial, and you will need to spend time debugging.

• A screen available for the gaming 'HVM'. (It can be a physical monitor or just to have multiple cables connected to the screen and switching between input source) [Optional]

• Dedicated gaming mouse and keyboard [Optional] .

IOMMU Group

Goal

What

The goal of this step if to retrieve the default IOMMU Group (https://www.kernel.org/doc/html/latest/driver-api/vfio.html#groups-devices-and-iommus) of your hardware.

Why

It can help understanding potential issue with your setup (what devices live in the same IOMMU group as your GPU) / finding potential workaround. If you feel lucky, skip this step.

How

You can't see your IOMMU Group when you are using Xen (the information is hidden from dom0).

• Boot a live linux distribution
• In the grub, enable iommu: Add the parameters iommu=1 iommu_amd=on to the linux commandline
• Once you logged in your live linux distribution, you need to retrieve the folder structure of /sys/kernel/iommu_group. You can use the following script to do that:

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
 echo "IOMMU Group ${g##*/}:"
 for d in $g/devices/*; do
  echo -e "\t$(lspci -nns ${d##*/})"
done
done

GRUB modification

You must hide your secondary GPU from dom0. To do that, you have to modify the GRUB. In a dom0 Terminal, type:

qvm-pci

Then find the devices id for your secondary GPU. In my case, it is dom0:0a_00.0. Edit /etc/default/grub, and add the PCI hiding.

GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0 "

then regenerate the grub

grub2-mkconfig -o /boot/grub2/grub.cfg

If you are using UEFI and Qubes OS 4.1 or earlier, the file to override with grub2-mkconfig is /boot/efi/EFI/qubes/grub.cfg.

Note: if after this step when you reboot the computer you get stuck in the QubesOS startup that means you are trying to use the GPU you just hid. Check your BIOS options. Also check the cables, BIOS have some GPU priority based on the type of cable. For example, DisplayPort can be favoured over HDMI.

Once you have rebooted, in dom0, type sudo lspci -vvn, you should see "Kernel driver in use: pciback" for the GPU you just hid.

Configuring the parameter "max-ram-below-4g"

Since the release of this qubes version of xen: 4.17.2-8 (R4.2, 2024-01-03), no additional configuration is required.

Remove any existing "max-ram-below-4g" workaround

mkdir stubroot
cp /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs stubroot/qemu-stubdom-linux-rootfs.gz
cd stubroot
gunzip qemu-stubdom-linux-rootfs.gz
cpio -i -d -H newc --no-absolute-filenames < qemu-stubdom-linux-rootfs
rm qemu-stubdom-linux-rootfs
nano init

# $dm_args and $kernel are separated with \n to allow for spaces in arguments

vm_name=$(xenstore-read "/local/domain/$domid/name")
if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
 max_ram_below_4g=$(echo "$vm_name" | grep -iEo "_([0-9]*(n[0-9]*)?)._" | cut -d '_' -f 2)
 if [[ -z "$max_ram_below_4g" ]];
 then
  max_ram_below_4g="3.5G"
 fi
 max_ram_below_4g=$(echo "$max_ram_below_4g" | sed 's/n/./g')
 dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g='"$max_ram_below_4g"'/g;p}')
fi

find . -print0 | cpio --null -ov \
--format=newc | gzip -9 > ../qemu-stubdom-linux-rootfs
sudo mv ../qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/

#!/bin/bash 

patch_rootfs(){

 filename=${1?Filename is required}

 cd ~/

 sudo rm -R "patched_$filename"
 mkdir "patched_$filename"

 cp /usr/libexec/xen/boot/$filename "patched_$filename/$filename.gz"
 cp /usr/libexec/xen/boot/$filename "$filename.original"

 cd patched_$filename
 gunzip $filename.gz
 cpio -i -d -H newc --no-absolute-filenames < "$filename"
 sudo rm $filename

 grep -i "max-ram-below-4g" init && echo "!!ERROR!! The thing is already patched ! EXITING ! " >&2 && exit

patch_string=$(cat <<'EOF'


vm_name=$(xenstore-read "/local/domain/$domid/name")
if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
 max_ram_below_4g=$(echo "$vm_name" | grep -iEo "_([0-9]*(n[0-9]*)?)._" | cut -d '_' -f 2)
 if [[ -z "$max_ram_below_4g" ]];
 then
  max_ram_below_4g="3.5G"
 fi
 max_ram_below_4g=$(echo "$max_ram_below_4g" | sed 's/n/./g')
 dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\\(-machine\\nxenfv\\)/\\1,max-ram-below-4g='"$max_ram_below_4g"'/g;p}')
fi
\# $dm_args and $kernel

EOF
)

awk -v r="$patch_string" '{gsub(/^# \$dm_args and \$kernel/,r)}1' init > init2
cp init /tmp/init_$filename
mv init2 init
chmod +x init

 find . -print0 | cpio --null -ov \
--format=newc | gzip -9 > ../$filename.patched
 sudo cp ../$filename.patched /usr/libexec/xen/boot/$filename

 cd ~/


}

grep -i "max-ram-below-4g" /usr/share/qubes/templates/libvirt/xen.xml && "!!ERROR!! xen.xml is patched ! EXITING ! " && exit
patch_rootfs "qemu-stubdom-linux-rootfs"
patch_rootfs "qemu-stubdom-linux-full-rootfs"

echo "stubdom have been patched."

<!-- server_ip is the address of stubdomain. It hosts it's own DNS server. -->
<emulator
 {% if vm.features.check_with_template('linux-stubdom', True) %}
 type="stubdom-linux"
 {% else %}
 type="stubdom"
 {% endif %}
 {% if vm.netvm %}
 {% if vm.features.check_with_template('linux-stubdom', True) %}
 cmdline="-qubes-net:client_ip={{ vm.ip -}}
 ,dns_0={{ vm.dns[0] -}}
 ,dns_1={{ vm.dns[1] -}}
 ,gw={{ vm.netvm.gateway -}}
 ,netmask={{ vm.netmask }} -machine xenfv,max-ram-below-4g=3.5G"
 {% else %}
 cmdline="-net lwip,client_ip={{ vm.ip -}}
 ,server_ip={{ vm.dns[1] -}}
 ,dns={{ vm.dns[0] -}}
 ,gw={{ vm.netvm.gateway -}}
 ,netmask={{ vm.netmask }} -machine xenfv,max-ram-below-4g=3.5G"
 {% endif %}
 {% else %}
 cmdline="-machine xenfv,max-ram-below-4g=3.5G"
 {% endif %}

Preparing the guest

As of 2023, I recommend using a Linux guest instead of a window guest.

Windows

Install a window VM, you can use this qvm-create-windows-qube

Linux

Create a new standalone Qube based on the template of your choice.

You must run the kernel provided by the guest distribution, because we will use some non-default kernel module for the GPU driver. Just follow the doc: managing-vm-kernel.

Install the GPU drivers you need.

And automated way to do that is proposed here: https://forum.qubes-os.org/t/create-a-gaming-hvm/19000#p-86347-automating-most-of-that-24

Pass the GPU

In qubes settings for the HVM, go to the 'devices' tab, pass the ID corresponding to your GPU.

qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 --persistent

You may or may not need to add the option "permissive" or "no-strict-reset". You may or may not need to passthrough additional devices depending on the result of the IOMMU script https://forum.qubes-os.org/t/create-a-gaming-hvm/19000#p-86347-iommu-group-2

Some word about the security implication of thoses parameters.

qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 -o permissive=True -o no-strict-reset=True --persistent

Starting the guest

This is where you will have a lot of issues to debug. Prepare for intense suffering

For Linux guests, run 'sudo dmesg' to have all the kernel log indicating you if there is a issue with your GPU driver. For some hardware, the MSI calls won't work. You can work around that using for example pci=nomsi or NVreg_EnableMSI=0 or something else. Check your drivers options. Check if alternative drivers exist (amdgpu, nvidia, nouveau, nvidia-open, using drivers from the official website, ...). Check multiple kernel version.

For nvidia GPU, I recommand using "nvidia-open" drivers instead of "nvidia"

Some links that could help you to debug the issues you will have

• https://forum.qubes-os.org/t/ryzen-7000-serie/

• https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html

For windows guests you will probably have the same issues but it will be harder to debug. I recommend using the drivers from Windows Update instead of the official drivers from the website of the constructor.

Some things that may be useful for debugging:

• Virsh (start, define, ...)

• /etc/libvirt/libxl/

• xl

• /etc/qubes/templates/libvirt/xen/by-name/

• /usr/lib/xen/boot/

• virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/...

Issues with the drivers could be related to 'qubes-vmm-xen-stubdom-linux', 'qubes-vmm-xen', and the Linux kernel you will be using.

[Case: No dedicated screen, mouse or keyboard] VirtualGL and vulkan

In some cases, it will just work out of the box (application will automatically detect the GPU and will be able to use it without modification.

In some cases, an additionnal layer is required: using VirtualGL, mentionned here: https://forum.qubes-os.org/t/seamless-gpu-passthrough-on-qubes-os-with-virtualgl/20265 In that case you may need to specify a new variable LD_LIBRARY_PATH=/usr/lib

You may have issue with how the mouse is handled by the game. AFAIK, no one tried to solve this issue.

[Case: Dedicated Screen, mouse and keyboard] Linux guest --- Integration with QubesOS

Xorg

Now Xorg. From XKCD:

Things you need to install:

• The Xorg input driver to support your mouse and keyboard

• Your favorite Windows Manager

In my case, it is:

archlinux version:

pacman -S xorg i3

debian version:

apt install xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-mouse i3

Then create a XORG configuration file for your GPU and screen. My file named 'xorg.conf':

Section "ServerLayout"
Identifier "Passthrough Layout"
Screen 0 "Passthrough Screen" Absolute 0 0
EndSection

Section "Device"
Identifier  "Passthrough GPU"
# name of the driver to use. Can be "amdgpu", "nvidia", or something else
Driver      "driver"
Option "Coolbits" "4"
# The BusID value will change after each qube reboot. 
BusID       "PCI:0::0"
EndSection

Section "Monitor"
Identifier "Passthrough monitor"
EndSection

Section "Screen"
Identifier "Passthrough Screen"
Device     "Passthrough GPU"
Monitor    "Passthrough Monitor"
EndSection

We can't know what is the correct BusID before the qube is started. And it change after each reboot. So let's write a script --- named "xorgX1.sh" --- that update this configuration file with the correct value, then start a binary on the Xorg X screen n°1.

#!/bin/bash
binary=${1:?binary required}

# Find the correct BusID of the AMD GPU, then set it in the Xorg configuration file
lspci | grep "VGA" | grep -E "NVIDIA" && sed -i 's/^Driver .*/Driver "nvidia"/g' /opt/xorg.conf
lspci | grep "VGA" | grep -E "AMD/ATI" && sed -i 's/^Driver .*/Driver "amdgpu"/g' /opt/xorg.conf
pci=$(lspci | grep "VGA" | grep -E "NVIDIA|AMD/ATI" | cut -d " " -f 1 | cut -d ":" -f 2 | cut -d "." -f 1 | cut -d "0" -f 2)
sed -i 's/"PCI:[^"]*"/"PCI:0:'$pci':0"/g' /opt/xorg.conf

# Start the Xorg server for the X screen number 1.
# The X screen n__0 is already used for QubesOS integration
sudo startx "$binary" -- :1 -config /opt/xorg.conf

Audio

• Create a script to launch your favorite Windows Manager and force it to use a specific pulse server. Example "i3.sh":

#!/bin/bash
sleep 5 && sudo setxkbmap -display :1 fr & 
/bin/sudo -u user PULSE_SERVER=unix:/run/user/1000/pulse/native bash -c 'sudo xhost + local:;/usr/bin/i3'

And launch it:

sudo ./xorgX1.sh ./i3.sh

sudo pacman -Rdd qubes-vm-pulseaudio pulseaudio
sudo pacman -S pipewire-{jack,alsa,pulse} pipewire-qubes

load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1

mkdir /root/.pulse
echo "default-server = 127.0.0.1" > /root/.pulse/client.conf

default-fragments = 60
default-fragment-size-msec = 1
high-priority = no
realtime-scheduling = no
nice-level = 18

load-module module-udev-detect

load-module module-udev-detect tsched=0

sudo ./xorgX1.sh /usr/bin/i3

Automating most of that

I trying to write a script to automate all the previous steps. It is available here: https://git.sr.ht/~yukikoo/gpu_template

Please be carefull and read the code, not much tests have been done

Issues and fixes

In one case in a setup with Intel IGPU + Nvidia DGPU, dom0 xorg crashed. Solved the case by adding a Xorg configuration to explicitly use Intel:

https://forum.qubes-os.org/t/gpu-passthrough-lightdm-service-wont-start/22220/20

References

• Archlinux: PulseAudio

• Archlinux: PulseAudio/Troubleshooting