This "guide" aims to explore and give a practical example of leveraging SaltStack to achieve the same goal as NVIDIA GPU passthrough into Linux HVMs for CUDA applications. Salt is a management engine that simplifies configuration, and QubesOS has its own flavour. Want to see some?

This guide assumes that you're done fiddling with your IOMMU groups and modified grub parameters to allow passthrough.

In addition to that, if you haven't set up salt environment yet, complete step 1.1 as described in this guide to get ready.

The basics

Before we even start doing anything, let's discuss the basics. You probably already know that salt configurations are stored in /srv/user_salt/. Here's how it may look:

.
├── nvidia-driver
│   ├── disable-nouveau.sls
│   ├── init.sls
│   └── map.jinja
├── test.sls
└── top.sls

Let's start with the obvious. top.sls is a top file. It describes high state, which is really just a combination of convetional salt formulas. Stray piece of salt configuration can be referred to as formula, although I've seen this word being used in various contexts. test.sls is a state file. It contains a configuration written in yaml. nvidia-driver is also a state, although it is a directory. This is an alternative way to store state for situations when you want to have multiple state (or not only state) files. When a state directory is referenced, salt evaluates init.sls state file inside. State files may or may not be included from init.sls or other state files.

Yaml configuration consists of states. In this context, state refers to a module - piece of code that most often does a pretty specific thing. In a configuration, states behave like commands or functions and methods of a programming language. One valuable thing to note here is that not all modules are state modules. There are a lot of them, and they can do various things, but here we only need the state kind.

In addition to state files, you notice map.jinja. Jinja is a templating engine. What it means is that it helps you to generalize your state files by adding variables, conditions and other cool features. You can easily recognize jinja statement by fancy brackets: {{ }}, {% %}, {# #}. This file in particular stores variable definitions and is used for configuration of the whole state directory thingy (nvidia-driver).

Writing salt configuration

1. Create a standalone

nvidia-driver--create-qube:
  qvm.vm:
    - name: {{ prefs.standalone_name }}
    - present:
      - template: {{ prefs.template_name }}


      - class: StandaloneVM
    - prefs:
      - label: {{ prefs.standalone_label }}
      - mem: {{ prefs.standalone_memory }}
      - vcpus: {{ prefs.standalone_cpus }}
      - pcidevs: {{ devices }}
      - virt_mode: hvm
      - kernel:
      - maxmem: 0
      - class: StandaloneVM
    - features:
      - set:
        - menu-items: qubes-run-terminal.desktop

Here, I use qubes-specific qvm.vm state module (which in reality is a wrapper around other modules, like prefs, features, etc.). Pretty much all values and keys here are the same as you can set and get using qvm-prefs and qvm-features. For nvidia drivers to work, kernel must be provided by the qube - that's why the field is empty. Similarly, to pass GPU we need to set virtualization mode to hvm and maxmem to 0 (it disables memory balancing).

nvidia-driver--create-qube is just a label. As long as you don't cross the syntax writing it, it should be fine. Aside from referencing, plenty of modules can use it to simplify the syntax, and some need it to decide what to do, but you can look it up later if you want.

Now, to the jinja statements. Here, they provide values for keys like label, template, name, etc. Some of them are done this way (as opposed to writing a value by hand) because the value is repeated in the state file multiple times, others are to simplify the process of configuration. In order to figure out why some of them use dot notation whereas other don't, we must check their declaration. In this state file they're imported using the following line:

{% from 'nvidia-driver/map.jinja' import prefs,devices,paths %}

This is pretty much just python in brackets. Notice that you need to specify directory when importing, and use actual path instead of dot notation.

{% set prefs = {
    'standalone_name': 'fedora-40-nvidia',
    'standalone_label': 'yellow',
    'standalone_memory': 4000,
    'standalone_cpus': 4,
    'template_name': 'fedora-40-xfce',
} %}



{% set devices = [
    '01:00.0',
    '01:00.1',
] %}

{% set paths = {
    'nvidia_conf': '/usr/share/X11/xorg.conf.d/nvidia.conf',
    'grub_conf': '/etc/default/grub',
    'grub_out': '/boot/grub2/grub.cfg',
} %}

Uploading salt is forbidden, therefore files are renamed into .yaml. For now, I only have state for fedora 40, but modifying it for debian or fedora without conflicting dependencies is trivial.

Contributions, improvements and fixes are welcome! I call it GPL-3 if you need that for some reason.