Nvidia GPU Pass-through on Qubes 4.3 (Fedora 43 Template)

Author's Note: This guide addresses specific dependency conflicts involving the grubby-dummy package that break standard Nvidia driver installations in Qubes. It implements a fully automated "native DNF" solution using an improved custom dummy package and DNF5 hooks, eliminating the need for manual wrapper scripts during updates.

⚠️ WARNINGS AND DISCLAIMERS

1. Security: This setup weakens isolation and is not recommended for high-security contexts.
2. Compatibility: This might not work with all hardware configurations.
3. Stability: Updates might beak this setup.

Phase 1: Dom0 Configuration

We need to isolate the GPU from Dom0 so it can be passed to a VM.

1. Identify your GPU PCI IDs: Open a terminal in Dom0 and run:

   lspci -nn | grep -i nvidia
   

   Example Output: > 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation... [10de:1e04] > 05:00.1 Audio device [0403]: NVIDIA Corporation... [10de:10f7] > 05:00.2 USB controller... > 05:00.3 Serial bus controller...

2. Hide the devices from Dom0: Edit /etc/default/grub in Dom0. Find the line GRUB_CMDLINE_LINUX and append the following (replace the IDs with your specific ones found above):

   rd.qubes.hide_pci=05:00.0,05:00.1,05:00.2,05:00.3
   

3. Update Grub and Reboot:

   sudo grub2-mkconfig -o /boot/grub2/grub.cfg
   sudo reboot
   

Phase 2: TemplateVM Preparation

We will create a specialized template for GPU workloads.

1. Clone the Template: Clone your standard Fedora 43 template (e.g., fedora-43-xfce) to fedora-43-xfce-gpu.

2. Configure Template Settings: Open Qube Settings for fedora-43-xfce-gpu:

   • Advanced Tab:
     • Kernel: (provided by qube) (Essential for Nvidia drivers)
     • Virtualization mode: HVM
     • Memory balancing: Uncheck/Disable (PCI pass-through requires fixed memory)
   • Devices Tab:
     • Pass through your Nvidia devices including Audio and USB controllers (if applicable).

3. Install Build Tools: Start the template (or use a temporary Disposable VM) and install the necessary tools:

   sudo dnf install rpm-build rpmrebuild
   

Phase 3: The "Super-Grubby" Fix (Solving Dependency Hell)

The Nvidia driver packages have a strict dependency on /usr/bin/grubby. The standard grubby-dummy package in Qubes does not satisfy this requirement in a way that pleases the DNF5 dependency resolver, causing conflicts with upstream packages (like sdubby or the real grubby).

We solve this by building a "Super-Dummy" package that explicitly provides the binary paths and capabilities required by the driver, preventing DNF from trying to pull in conflicting packages.

1. Create the Spec File: Run this command in the TemplateVM to create the build recipe:

   cat <<EOF > super-grubby.spec
   Name:       grubby-dummy
   Version:    99.0.0
   Release:    2%{?dist}
   Epoch:      1000
   Summary:    Super Dummy for Grubby and Sdubby
   License:    Public Domain
   BuildArch:  noarch
   
   # Claim to provide the packages
   Provides:   grubby = %{version}
   Provides:   sdubby = %{version}
   Provides:   grubby-dummy = %{version}
   
   # Claim to provide the specific binary paths (Virtual Provision)
   Provides:   /usr/bin/grubby
   Provides:   /usr/sbin/grubby
   
   # Block the real packages
   Obsoletes:  grubby < %{version}
   Obsoletes:  sdubby < %{version}
   
   %description
   Dummy package to satisfy Nvidia driver dependencies for /usr/bin/grubby.
   
   %build
   # Nothing to build
   
   %install
   # Create only /usr/bin
   mkdir -p %{buildroot}/usr/bin
   
   # Create the dummy script
   echo '#!/bin/bash' > %{buildroot}/usr/bin/grubby
   echo 'echo "Dummy grubby called - doing nothing."' >> %{buildroot}/usr/bin/grubby
   echo 'exit 0' >> %{buildroot}/usr/bin/grubby
   
   # Make it executable
   chmod +x %{buildroot}/usr/bin/grubby
   
   %files
   /usr/bin/grubby
   
   EOF
   

2. Build the Package:

   rpmbuild -bb super-grubby.spec
   

3. Install the Super-Dummy:

   You might first need to remove old grubby-dummy manually.

   sudo rpm -e --nodeps grubby-dummy
   

   This will replace the existing Qubes dummy and prevent DNF from ever trying to install the conflicting package.

   sudo dnf install ~/rpmbuild/RPMS/noarch/grubby-dummy-99.0.0-2.fc43.noarch.rpm -y
   

Phase 4: Automating Updates (DNF5 Hooks)

We need to automate the Dracut configuration and fix the "Split-Brain" issue (where the headless Template crashes if Nvidia EGL is enabled, but the AppVM needs it enabled).

1. Create the Hook Script: Create /usr/local/bin/qubes-nvidia-hook.sh:

   #!/bin/bash
   set -e
   
   # --- Configuration ---
   NVIDIA_EGL="/usr/share/glvnd/egl_vendor.d/10_nvidia.json"
   NVIDIA_EGL_BACKUP="/usr/share/glvnd/egl_vendor.d/10_nvidia.json.enabled"
   DRACUT_CONF="/usr/lib/dracut/dracut.conf.d/99-nvidia-dracut.conf"
   XORG_CONF="/usr/share/X11/xorg.conf.d/nvidia.conf"
   
   echo ">>> [Nvidia-Hook] Starting post-update configuration..."
   
   # 1. Handle EGL Split-Brain (Template vs AppVM)
   # We save a copy of the config for the AppVM, then disable it for the Template
   if [ -f "$NVIDIA_EGL" ]; then
       # Check if the file is NOT empty (meaning it was just replaced by an update)
       if [ -s "$NVIDIA_EGL" ] && [ "$(cat "$NVIDIA_EGL")" != "{}" ]; then
           echo " -> New Nvidia EGL config detected."
   
           # Snapshot the fresh config to the .enabled file for the AppVM to use
           cp -f "$NVIDIA_EGL" "$NVIDIA_EGL_BACKUP"
           echo " -> Updated AppVM backup ($NVIDIA_EGL_BACKUP)."
   
           # Neuter the active config for the Template (prevents crash on shutdown/boot)
           echo "{}" > "$NVIDIA_EGL"
           echo " -> Disabled EGL for Template (wrote empty JSON)."
       else
           echo " -> EGL config already neutralized."
       fi
   fi
   
   # 2. Fix Dracut Config (omit -> add)
   # The update usually resets this file, so we force-patch it every time.
   if [ -f "$DRACUT_CONF" ]; then
       sed -i 's/omit_drivers/add_drivers/g' "$DRACUT_CONF"
       echo " -> Dracut config patched (omit_drivers -> add_drivers)."
   fi
   
   # 3. Remove conflicting Xorg config (Fixes VM crash/hang on shutdown)
   if [ -f "$XORG_CONF" ]; then
       rm -f "$XORG_CONF"
       echo " -> Conflicting Xorg config removed."
   fi
   
   # 4. Regenerate Initramfs
   # CRITICAL: Target the LATEST installed kernel, not necessarily the running one.
   LATEST_KERNEL=$(ls /lib/modules | sort -V | tail -n 1)
   
   if [ -n "$LATEST_KERNEL" ]; then
       echo " -> Regenerating initramfs for kernel: $LATEST_KERNEL"
       dracut -f --kver "$LATEST_KERNEL"
   else
       echo " -> Warning: Could not detect kernel version. Skipping dracut."
   fi
   
   echo ">>> [Nvidia-Hook] Cleanup complete."
   

2. Make it Executable:

   sudo chmod +x /usr/local/bin/qubes-nvidia-hook.sh
   

3. Register the DNF5 Action: Create /etc/dnf/libdnf5-plugins/actions.d/nvidia-qubes.actions:

   # Trigger the fix script after any transaction involving nvidia packages
   # Syntax: trigger:package_filter:direction:option:command
   post_transaction:*nvidia*:in::/usr/local/bin/qubes-nvidia-hook.sh
   

Phase 5: Install Nvidia Drivers

Now that the infrastructure is in place, installing the drivers is standard.

1. Install Packages:

   sudo dnf install xorg-x11-drv-nvidia xorg-x11-drv-nvidia-cuda akmod-nvidia kernel-devel
   

   Note: Due to the DNF hook, the initramfs regeneration and config patching will happen automatically at the end of this transaction.

2. Shutdown the Template:

   sudo poweroff
   

Phase 6: AppVM Configuration

We need an AppVM that has the physical GPU attached and knows how to restore the EGL config that we disabled in the Template.

1. Create the AppVM: Create gpu-personal based on fedora-43-xfce-gpu.

2. Configure AppVM Settings:

   • Advanced: Kernel (provided by qube), Mode HVM, Memory Balancing Disabled.
   • Devices: Add your Nvidia GPU (VGA, Audio, USB, etc.). Remember to remove card from your template at this point.

3. Enable Nvidia EGL (The Split-Brain Fix):

   The Template has an empty EGL config (to prevent crashes). We need the AppVM to use the valid backup we created.

   Start the AppVM, open a terminal, and edit /rw/config/rc.local:

   sudo nano /rw/config/rc.local
   

   Add this content:

   #!/bin/bash
   # Restore Nvidia EGL config for GPU pass-through
   if [ -f /usr/share/glvnd/egl_vendor.d/10_nvidia.json.enabled ]; then
       mount --bind /usr/share/glvnd/egl_vendor.d/10_nvidia.json.enabled /usr/share/glvnd/egl_vendor.d/10_nvidia.json
   fi
   

4. Reboot the AppVM.

Verification & Troubleshooting

After rebooting the AppVM:

1. Verify Driver Load: Open a terminal in the AppVM:

   nvidia-smi
   

   You should see your GPU model and memory usage.

2. Manual Recovery (If updates fail): If the DNF hook ever fails to fire, you can manually trigger the fix in the TemplateVM:

   sudo /usr/local/bin/qubes-nvidia-hook.sh
   

3. Manual Module Build: If nvidia-smi fails, check if the module was built in the Template:

   rpm -qa | grep kmod-nvidia
   # If missing, force rebuild:
   sudo akmods --rebuild --kernels $(uname -r)