You need to check what are the things/devices that are in the same IOMMU group as the GPU you want to passthrough. You can't see your IOMMU Group when you are using Xen (the information is hidden from dom0). So, start a live linux distribution, enable iommu in the grub options (iommu=1 iommu_amd=on), and then displayed the folder structure of /sys/kernel/iommu_group

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
 echo "IOMMU Group ${g##*/}:"
 for d in $g/devices/*; do
  echo -e "\t$(lspci -nns ${d##*/})"
done
done

GRUB modification

You must hide your secondary GPU from dom0. To do that, you have to modify the GRUB. In a dom0 Terminal, type:

Then find the devices id for your secondary GPU. In my case, it is dom0:0a_00.0. Edit /etc/default/grub, and add the PCI hiding.

GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0 "

grub2-mkconfig -o /boot/grub2/grub.cfg

If you are using UEFI, the file to override with grub2-mkconfig is /boot/efi/EFI/qubes/grub.cfg.

Note: if after this step when you reboot the computer you get stuck in the QubesOS startup that means you are trying to use the GPU you just hide. Check your BIOS options. Also check the cables, BIOS have some GPU priority based on the type of cable. For example, DisplayPort can be favoured over HDMI.

Once you have rebooted, in dom0, type sudo lspci -vvn, you should see "Kernel driver in use: pciback" for the GPU you just hide.

Configuring the parameter "max-ram-below-4g"

Why do we need to do that ?

This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has. The code to pass a custom max-ram-below-4g value to the qemu command line does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to add this parameter. It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal.

( "max-ram-below-4g" is not related to the amount of ram you can pass to the VM. It is not related to VRAM either. It is related to WHAT is available in the memory space for 32 bits system and WHAT is stored in the memory space for 64 bits system (usable ram is only a part of what need to be mapped in the memory space) )

Finding the correct value for this parameter

Current best guess is to run this command in dom0: lspci -vvs GPU_IDENTIFIER | grep Region , for example: lspci -vvs 0a:00.0 | grep Region. if the max value of [size=XXXX] is 256MB, try 3.5G for max-ram-below-4g. If the max value is bigger, try 2G for max-ram-below-4g.

Patching stubdom-linux-rootfs.gz

I modified the original code to: - make it works with Qubes R4.1/R4.2 - removed one of the original limitations by restricting the modification to VM with a name starting with "gpu_" - Added a way to modify per vm the value for "max-ram-below-4g". Ex, if you want specifically to use 2G for "max-ram-below-4g", name the vm "gpu_2G_YOURNAME", if you want specifically to use 3.5G for "max-ram-below-4g", name the vm "gpu_3n5G_YOURNAME"

mkdir stubroot
cp /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs stubroot/qemu-stubdom-linux-rootfs.gz
cd stubroot
gunzip qemu-stubdom-linux-rootfs.gz
cpio -i -d -H newc --no-absolute-filenames &lt; qemu-stubdom-linux-rootfs
rm qemu-stubdom-linux-rootfs
nano init

# $dm_args and $kernel are separated with \n to allow for spaces in arguments

vm_name=$(xenstore-read "/local/domain/$domid/name")
if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
 max_ram_below_4g=$(echo "$vm_name" | grep -iEo "_([0-9]*(n[0-9]*)?)._" | cut -d '_' -f 2)
 if [[ -z "$max_ram_below_4g" ]];
 then
  max_ram_below_4g="3.5G"
 fi
 max_ram_below_4g=$(echo "$max_ram_below_4g" | sed 's/n/./g')
 dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g='"$max_ram_below_4g"'/g;p}')
fi

find . -print0 | cpio --null -ov \
--format=newc | gzip -9 &gt; ../qemu-stubdom-linux-rootfs
sudo mv ../qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/

Note that this will apply the change to the HVM with a name starting with "gpu_". So you need to name your gaming HVM "gpu_SOMETHING".

Alternatively, the following dom0 script "patch_stubdom.sh" does all the previous steps:

#!/bin/bash 

patch_rootfs(){

 filename=${1?Filename is required}

 cd ~/

 sudo rm -R "patched_$filename"
 mkdir "patched_$filename"

 cp /usr/libexec/xen/boot/$filename "patched_$filename/$filename.gz"
 cp /usr/libexec/xen/boot/$filename "$filename.original"

 cd patched_$filename
 gunzip $filename.gz
 cpio -i -d -H newc --no-absolute-filenames &lt; "$filename"
 sudo rm $filename

 grep -i "max-ram-below-4g" init &amp;&amp; echo "!!ERROR!! The thing is already patched ! EXITING ! " &gt;&amp;2 &amp;&amp; exit

patch_string=$(cat &lt;&lt;'EOF'


vm_name=$(xenstore-read "/local/domain/$domid/name")
if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
 max_ram_below_4g=$(echo "$vm_name" | grep -iEo "_([0-9]*(n[0-9]*)?)._" | cut -d '_' -f 2)
 if [[ -z "$max_ram_below_4g" ]];
 then
  max_ram_below_4g="3.5G"
 fi
 max_ram_below_4g=$(echo "$max_ram_below_4g" | sed 's/n/./g')
 dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\\(-machine\\nxenfv\\)/\\1,max-ram-below-4g='"$max_ram_below_4g"'/g;p}')
fi
\# $dm_args and $kernel

EOF
)

awk -v r="$patch_string" '{gsub(/^# \$dm_args and \$kernel/,r)}1' init &gt; init2
cp init /tmp/init_$filename
mv init2 init
chmod +x init

 find . -print0 | cpio --null -ov \
--format=newc | gzip -9 &gt; ../$filename.patched
 sudo cp ../$filename.patched /usr/libexec/xen/boot/$filename

 cd ~/


}

grep -i "max-ram-below-4g" /usr/share/qubes/templates/libvirt/xen.xml &amp;&amp; "!!ERROR!! xen.xml is patched ! EXITING ! " &amp;&amp; exit
patch_rootfs "qemu-stubdom-linux-rootfs"
patch_rootfs "qemu-stubdom-linux-full-rootfs"

echo "stubdom have been patched."

[details="OUTDATED - DO NOT USE - Other method: Patching xen.xml instead of stubdom"]

Instead of patching stubdom-linux-rootfs, you could inject the command directly inside the configuration template. It is the file "core-admin/ "templates/libvirt/xen.xml" in the "qubes-core-admin" repository. In dom0 this file is in "/usr/share/qubes/templates/libvirt/xen.xml"

See below the part that have been modified to add the needed "max-ram-below-4g" option.

&lt;emulator
 {% if vm.features.check_with_template('linux-stubdom', True) %}
 type="stubdom-linux"
 {% else %}
 type="stubdom"
 {% endif %}
 {% if vm.netvm %}
 {% if vm.features.check_with_template('linux-stubdom', True) %}
 cmdline="-qubes-net:client_ip={{ vm.ip -}}
 ,dns_0={{ vm.dns[0] -}}
 ,dns_1={{ vm.dns[1] -}}
 ,gw={{ vm.netvm.gateway -}}
 ,netmask={{ vm.netmask }} -machine xenfv,max-ram-below-4g=3.5G"
 {% else %}
 cmdline="-net lwip,client_ip={{ vm.ip -}}
 ,server_ip={{ vm.dns[1] -}}
 ,dns={{ vm.dns[0] -}}
 ,gw={{ vm.netvm.gateway -}}
 ,netmask={{ vm.netmask }} -machine xenfv,max-ram-below-4g=3.5G"
 {% endif %}
 {% else %}
 cmdline="-machine xenfv,max-ram-below-4g=3.5G"
 {% endif %}

I haven’t personnally tested this alternative but it should work and some users reported that it work. This method is less tested than patching stubdom-linux-rootfs, so I recommend patching stubdom-linux-rootfs. [/details]

Preparing the guest

Windows

Linux

You must run the kernel provided by the guest distribution, because we will use some non-default kernel module for the GPU driver. Just follow the doc: managing-vm-kernel.

Pass the GPU

In qubes settings for the HVM, go to the 'devices' tab, pass the ID corresponding to your GPU.

qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 -o permissive=True -o no-strict-reset=True

Starting the guest

For Linux guests, run 'sudo dmesg' to have all the kernel log indicating you if there is a issue with your GPU driver. For some hardware, the MSI calls won't work. You can work around that using for example pci=nomsi or NVreg_EnableMSI=0 or something else. Check your drivers options. Check if alternative drivers exist (amdgpu, nvidia, nouveau, nvidia-open, using drivers from the official website, ...). Check multiple kernel version.

For windows guests you will probably have the same issues but it will be harder to debug. I recommend using the drivers from Windows Update instead of the official drivers from the website of the constructor.

Issues with the drivers could be related to 'qubes-vmm-xen-stubdom-linux', 'qubes-vmm-xen', and the Linux kernel you will be using.

Linux guest --- Integration with QubesOS

Xorg

apt install xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-mouse i3

Then create a XORG configuration file for your GPU and screen. My file named 'AOC.conf':

Section "ServerLayout"
Identifier "Gaming"
Screen 0 "AMD AOC" Absolute 0 0
EndSection

Section "Device"
Identifier  "AMD"

# name of the driver to use. Can be "amdgpu", "nvidia", or something else
Driver      "amdgpu"

# The BusID value will change after each qube reboot. 
BusID       "PCI:0:8:0"
EndSection

Section "Monitor"
Identifier "AOC"
VertRefresh 60
# https://arachnoid.com/modelines/ .  IMPORTANT TO GET RIGHT. MUST ADJUST WITH EACH SCREEN. 
Modeline "1920x1080" 172.80 1920 2040 2248 2576 1080 1081 1084 1118
EndSection

Section "Screen"
Identifier "AMD AOC"
Device     "AMD"
Monitor    "AOC"
EndSection

We can't know what is the correct BusID before the qube is started. And it change after each reboot. So let's write a script --- named "xorgX1.sh" --- that update this configuration file with the correct value, then start a binary on the Xorg X screen n°1.

#!/bin/bash

binary=${1:?binary required}

# Find the correct BusID of the AMD GPU, then set it in the Xorg configuration file
pci=$(lspci | grep "VGA" | grep -E "NVIDIA|AMD/ATI" | cut -d " " -f 1 | cut -d ":" -f 2 | cut -d "." -f 1 | cut -d "0" -f 2)
sed -i 's/"PCI:[^"]*"/"PCI:0:'$pci':0"/g' /home/user/AOC.conf

# Start the Xorg server for the X screen number 1.
# The X screen n°0 is already used for QubesOS integration
sudo startx "$binary" -- :1 -config /home/user/AOC.conf

!/bin/bash

Find the correct BusID of the AMD GPU, then set it in the Xorg configuration file

Pulseaudio setup

sudo killall pulseaudio sudo sed -i "s/load-module module-vchan-sink./load-module module-vchan-sink domid=$(qubesdb-read -w /qubes-audio-domain-xid)/" /etc/pulse/qubes-default.pa sudo rm /home/user/.pulse/client.conf start-pulseaudio-with-vchan sleep 5 && sudo chmod -R 777 /root/ & sleep 5 && sudo chmod -R 777 /root/ & sleep 5 && sudo cp /root/.pulse/client.conf /home/user/.pulse/client.conf && sudo chown -R user:user /home/user/.pulse/client.conf &

Start the Xorg server for the X screen number 1.

The X screen n°0 is already used for QubesOS integration

Audio

sudo pacman -Rdd qubes-vm-pulseaudio pulseaudio
sudo pacman -S pipewire-{jack,alsa,pulse} pipewire-qubes

#!/bin/bash
sleep 5 &amp;&amp; sudo setxkbmap -display :1 fr &amp; 
/bin/sudo -u user PULSE_SERVER=unix:/run/user/1000/pulse/native bash -c 'sudo xhost + local:;/usr/bin/i3'

sudo ./xorgX1.sh ./i3.sh

So you need to configure pulseaudio for Xorg multiseat. The archlinux documentation explain that very well: Xorg multiseat Use the option without system-mode deamon and adapt it to qube: Add the following line to /etc/pulse/qubes-default.pa

load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1

mkdir /root/.pulse
echo "default-server = 127.0.0.1" &gt; /root/.pulse/client.conf

The sound was buggy/laggy on my computer. So tried to find a workaround by playing with pulseaudio settings. It was more or less random tries, so I can't really explain it: In /etc/pulse/daemon.conf add the following lines:

default-fragments = 60
default-fragment-size-msec = 1
high-priority = no
realtime-scheduling = no
nice-level = 18

load-module module-udev-detect

load-module module-udev-detect tsched=0

sudo ./xorgX1.sh /usr/bin/i3

References

Things to do

[details="This document was migrated from the qubes-community project"] - Page archive - First commit: 18 Jan 2023. Last commit: 18 Jan 2023. - Applicable Qubes OS releases based on commit dates and supported releases: 4.1 - Original author(s) (GitHub usernames): neowutran - Original author(s) (forum usernames): @neowutran - Document license: CC BY 4.0 [/details] [details="Contributors"] - @neowutran - @deeplow - @xvrhthxn [/details]

Hardware

IOMMU Group