As a beginner, Salt seemed daunting to me at first. It took me some efforts to learn but I love it now! I'm writing this guide for beginners who enjoy an hands-on introduction with examples.

1.1 Creating personal state configuration directories

Our journey starts with a file found in the base Salt configuration directory in dom0: /srv/salt/qubes/README.rst (GitHub link). In this file we can read:

> #### qubes.user-dirs > > Install and maintain user salt and pillar directories for personal state configurations: > >

txt
&gt; /srv/user_salt
&gt; /srv/user_pillar
&gt;

> > User defined scripts will not be removed on removal of qubes-mgmt-salt by design nor will they be modified on any updates, other than permissions being enforced.

We can activate qubes.user-dirs to create personal state configuration directories. What is this, and how do we activate it? This is what we call a state configuration. It is a configuration file that tells Salt what to do to reach a particular state.

To activate qubes.user-dirs, we can follow the instructions found in its configuration file, /srv/salt/qubes/user-dirs.sls (GitHub link):

txt
&gt; qubes.user-dirs
&gt; ===============
&gt;
&gt; Install and maintain user salt and pillar directories for personal state
&gt; configurations:
&gt;
&gt;   Includes a simple locale state file
&gt;
&gt;   User defined scripts will not be removed on removal of qubes-mgmt-salt
&gt;   by design nor will they be modified on any updates, other than permissions
&gt;   being enforced.
&gt;
&gt; Execute:
&gt; --------
&gt;   qubesctl state.sls qubes.user-dirs
&gt;

We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tell us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we'll place our state configuration files.

1.2 The top file and running highstate

Running the state qubes.user-dirs will also create the file /srv/user_salt/top.sls. Here is what this file looks like before we modify it (GitHub link):

yaml
&gt; # vim: set syntax=yaml ts=2 sw=2 sts=2 et :
&gt; #
&gt; # 1) Intial Setup: sync any modules, etc
&gt; # --&gt; qubesctl saltutil.sync_all
&gt; #
&gt; # 2) Initial Key Import:
&gt; # --&gt; qubesctl state.sls salt.gnupg
&gt; #
&gt; # 3) Highstate will execute all states
&gt; # --&gt; qubesctl state.highstate
&gt; #
&gt; # 4) Highstate test mode only.  Note note all states seem to conform to test
&gt; #    mode and may apply state anyway.  Needs more testing to confirm or not!
&gt; # --&gt; qubesctl state.highstate test=True
&gt;
&gt; # === User Defined Salt States ================================================
&gt; #user:
&gt; #  '*':
&gt; #    - locale
&gt;

In the future, when we have many state configuration files, it will become quite tedious to run each state one by one with the command sudo qubesctl state.sls my-custom-state. The top file solves that. If we write in this file how to run each state, we get the ability to run all of them with a single command: sudo qubesctl state.highstate. We call this "running highstate".

1.3 Targeting qubes

There are three lines that are commented out at the end of the top file /srv/user_salt/top.sls:

If we were to uncomment those lines and run highstate, Salt would run in all targeted qubes (this is what is meant by the * character) the state locale, for which the state configuration file is either /srv/user_salt/locale.sls or /srv/user_salt/locale/init.sls.

How do we target a qube? By default, the commands qubesctl state.sls my-custom-state and qubesctl state.highstate only target dom0. To make Salt target additional qubes, we can give their names to the --targets argument:

1.4. Creating a qube with Salt

We have a template called fedora-38. We would like Salt to create a purple qube named "salty" based on this template. We write the state configuration file /srv/user_salt/salty.sls as follows:

salty--create-qube:
  qvm.vm:
    - name: salty
    - present:
      - template: fedora-38
      - label: purple
    - prefs:
      - label: purple

That's it! Running sudo qubesctl state.sls salty saltenv=user will make Salt create a purple qube named salty. If salty is already present, Salt will just make sure it's purple but won't do anything else.

[details=Note on "saltenv=user"] Note that we always need to add the extra argument saltenv=user to the command sudo qubesctl state.sls my-custom-state when we run individual states from the user directory /srv/user_salt/. [/details]

To make things easier, we would like to automatically run this state when we run highstate. We add the following to the top file /srv/user_salt/top.sls:

user:
  dom0:
    - salty

Great! Now, the command sudo qubesctl state.highstate will automatically create salty.

1.5 Creating a disconnected qube

We have a template called debian-11. We would like Salt to create a green qube named "disconnected" based on this template, but that has no web browser and no internet access. We write the state configuration file /srv/user_salt/disconnected.sls as follows:

{% set gui_user = salt['cmd.shell']('groupmems -l -g qubes') %}

disconnected--create-qube:
  qvm.vm:
    - name: disconnected
    - present:
      - template: debian-11
      - label: green
    - prefs:
      - label: green
      - netvm: none
    - features:
      - set:
        - menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop

disconnected--update-app-menu:
  cmd.run:
    - name: qvm-appmenus --update disconnected
    - runas: {{ gui_user }}
    - require:
      - qvm: disconnected--create-qube

Perfect! We can now make Salt create this qube with the command sudo qubesctl state.sls disconnected saltenv=user.

[details=Note on the "{}" characters] Note that by default, cmd.run makes Salt run commands as root. The command qvm-appmenus does not work as root, so we have to make Salt run this command as a regular user. To do so, in the first line of the file we use a templating language called Jinja to retrieve our username, we save our username in the gui_user variable, and we use this variable when needed. Salt will always execute all the templating instructions between {} before running a state configuration file. [/details]

To make things easier, we would like to automatically run this state when we run highstate. We add the following lines to the top file /srv/user_salt/top.sls:

user:
  dom0:
    - disconnected

Great! Now, the command sudo qubesctl state.highstate will automatically create our disconnected qube.

[details=Tip: How to make Salt create both "salty" and "disconnected" when we run highstate?] We can write the top file /srv/user_salt/top.sls as follows:

user:
  dom0:
    - salty
    - disconnected

1.6 Useful links

The next part of this guide will be about creating new templates and installing packages in them.

Part 2: Apps and templates

In this part we'll learn how use Salt to make qubes with new software (including apps that are not in the official repositories!), and create new templates.

2.1 Activating pre-installed apps

We have a template called debian-12. We would like Salt to create a "vault" qube based on debian-12 that is never connected to the internet, and that we will only use for the app KeepassXC.

Luckily, KeepassXC comes pre-installed in the template debian-12, so we can simply tell Salt to make it available in the app menu. We write our state configuration file /srv/user_salt/vault.sls as follows:

{% set gui_user = salt['cmd.shell']('groupmems -l -g qubes') %}

vault--create-qube:
  qvm.vm:
    - name: vault
    - present:
      - template: debian-12
      - label: black
    - prefs:
      - label: black
      - netvm: none
    - features:
      - set:
        - menu-items: org.keepassxc.KeePassXC.desktop org.gnome.Terminal.desktop

vault--update-app-menu:
  cmd.run:
    - name: qvm-appmenus --update vault
    - runas: {{ gui_user }}
    - require:
      - qvm: vault--create-qube

As a result, running sudo qubesctl state.sls vault saltenv=user will make Salt create the vault qube if it's not there, and make sure that it has KeePassXC in its app menu.

To make things easier, we would like Salt to automatically take care of the vault when we run highstate. We write the following in the top file /srv/user_salt/top.sls:

user:
  dom0:
    - vault

The command sudo qubesctl state.highstate will now automatically run the "vault" state.

2.2 Installing new apps

We would like to have a "messaging" qube for communicating with our friends through an app called Telegram. However, Telegram is not part of the debian-12 template, so we'll have to install it.

Luckily, Telegram is available in the official repository. We can therefore tell Salt to create the "messaging" qube and make sure that Telegram is installed in the debian-12 template by writing the state configuration file /srv/user_salt/messaging.sls as follows:

{% if grains['id'] == 'dom0' %}

{% set gui_user = salt['cmd.shell']('groupmems -l -g qubes') %}

messaging--create-qube:
  qvm.vm:
    - name: messaging
    - present:
      - template: debian-12
      - label: yellow
    - prefs:
      - label: yellow
    - features:
      - set:
        - menu-items: org.telegram.desktop.desktop org.gnome.Nautilus.desktop

messaging--update-app-menu:
  cmd.run:
    - name: qvm-appmenus --update messaging
    - runas: {{ gui_user }}
    - require:
      - qvm: messaging--create-qube

{% elif grains['id'] == 'debian-12' %}

messaging--install-apps-in-template:
  pkg.installed:
    - pkgs:
      - telegram-desktop

{% endif %}

There you go! As we need to run the install process in the debian-12 template, we have to target debian-12 when we make Salt execute this state: sudo qubesctl --targets=debian-12 --show-output state.sls messaging saltenv=user.

[details=Note on the "{% ... %}" syntax] This state configuration file has two parts. In the first part, we wrote the instructions that Salt has to execute while running in the admin qube dom0, while the second part is about installing Telegram, which must be executed in the template debian-12. To have everything in the same file, but ensure that the right part get executed in the right qube, we decided to use a Jinja "if statement" to modify the state configuration file depending on in what qube Salt is running the instructions for. [/details]

Similarly, we can also have Salt apply this state targeting both dom0 and debian-12 when running highstate. We add the following to the top file /srv/user_salt/top.sls:

user:
  dom0 or debian-12:
    - messaging

This makes the command sudo qubesctl --targets=debian-12 --show-output state.highstate automatically create a messaging qube with Telegram as part of its app menu.

2.3 Creating a "non-free" template

We would like to create a "conferencing" qube with the software Skype to communicate with our family. Skype, however, is not available from the official debian-12 repository because it is distributed under a proprietary software licence: we will have to add an external repository to be able to install it.

As Skype is not from the official repository, we consider that there is a non-zero risk that it compromises the security of the template during its installation process. Because we want to trust our default templates, we decide to create a new "nonfree" template to install this proprietary software.

We start by downloading the cryptographic key that signs the Skype repository. From a trusted qube called "disp2956" that is connected to the internet, we run the command:

curl --output skype.asc https://repo.skype.com/data/SKYPE-GPG-KEY

We check the file's contents with the command cat skype.asc to make sure that the file is not malicious. Only if we are completely sure that it is not malicious, we can copy the file to dom0 by opening a dom0 terminal and running:

[details=Make sure to understand the risks of copying files to dom0 before executing this command.]

qvm-run --pass-io disp2956 'cat skype.asc' &gt; skype.asc

gpg --dearmor --output skype.gpg skype.asc

The keyring is ready to be used by Salt, so we can move it under a new directory at /srv/user_salt/conferencing/skype.gpg.

For practical purposes, we will write our state configuration file under the same directory, at /srv/user_salt/conferencing/init.sls. The state configuration file reads:

{% set gui_user = salt['cmd.shell']('groupmems -l -g qubes') %}

{% if grains['id'] == 'dom0' %}

conferencing--create-nonfree-template:
  qvm.clone:
    - name: nonfree
    - source: debian-12

conferencing--create-app-qube:
  qvm.vm:
    - name: conferencing
    - present:
      - template: nonfree
      - label: yellow
    - prefs:
      - label: yellow
    - features:
      - set:
        - menu-items: skypeforlinux.desktop org.gnome.Nautilus.desktop
    - require:
      - qvm: conferencing--create-nonfree-template

conferencing--update-app-menu:
  cmd.run:
    - name: qvm-appmenus --update conferencing
    - runas: {{ gui_user }}
    - require:
      - qvm: conferencing--create-app-qube

{% elif grains['id'] == 'nonfree' %}

conferencing--add-repository:
  pkgrepo.managed:
    - name: deb [signed-by=/etc/apt/keyrings/skype.gpg] https://repo.skype.com/deb stable main
    - file: /etc/apt/sources.list.d/skype-stable.list
    - key_url: salt://conferencing/skype.gpg
    - aptkey: False
    - require_in:
      - pkg: conferencing--install-apps

conferencing--install-apps:
  pkg.installed:
    - pkgs:
      - skypeforlinux

{% endif %}

Running this state makes Salt create a "conferencing" app qube based on a new template called "nonfree", in which Salt makes sure that Skype is installed through the external repository. To run this state, we target our new "nonfree" template with the command:

sudo qubesctl --targets=nonfree --show-output state.sls messaging saltenv=user

Let's add this state to the top file, so that it is applied automatically when we run highstate! At the end of the top file /srv/user_salt/top.sls, we write:

user:
  dom0 or nonfree:
    - conferencing

sudo qubesctl --targets=nonfree --show-output state.highstate

[details=Tip: Here is what our top file would look like if we would include all the states from this guide so far.]

user:
  dom0:
    - salty
    - disconnected
    - vault
  dom0 or debian-12:
    - messaging
  dom0 or nonfree:
    - conferencing

sudo qubesctl --targets=debian-12,nonfree --show-output state.highstate

2.4 Useful links

I tried to be as concise as I could. Please let me know if you have any further questions! Here are some related links.

In the next part of this guide, we will learn how to make Salt perform automated backups of our qubes with Wyng.

Part 3: Backing up Qubes

One of the biggest advantages of using Salt with Qubes OS is that we would just need to copy and run our state configuration files to completely recreate our system. Saving those files would however not be sufficient for a backup, because that would not contain any our personal data.

We could use the official Qubes Backup tool to create full backups of our qubes, but as of now this tool does not support creating incremental backups, which make the backup process much more performant.

In this part of the guide, we are going to create Salt states that can make fast incremental backups of our qubes.

3.1 Downloading Wyng

Wyng is a backup software that we can use to make incremental backups of the contents of our qubes in a very efficient manner. Its project page gives a detailed explanation of how it works.

It should be noted that a tool called wyng-util-qubes is currently being developed that makes it very easy to use Wyng with Qubes OS. One of the advantages of using this tool is that, on top of backing up the contents of our qubes with Wyng, this tool also backs up their configuration. At the time of writing, however, wyng-util-qubes does not yet support installations of Qubes OS with a BTRFS partitioning scheme. For this reason, and because the configuration of our qubes is already saved in our Salt state configuration files, we are going to use Wyng directly instead of wyng-util-qubes.

To use Wyng, we first have to copy it to dom0. To do so, can we download and extract the main branch of its repository in a trusted qube called "disp8265" with the command:

curl --location https://github.com/tasket/wyng-backup/archive/refs/heads/main.tar.gz | tar --extract --gzip

[details=Note on the version of Wyng used in this guide.] The above method downloads the recommended version of Wyng, which is Wing v0.8beta at the time of writing. It might be that some parts of this guide stop working with a more recent version of Wyng, in which case the guide will have to be updated. [/details]

The entire Wyng program is contained in the file wyng-backup-main/src/wyng, which is in the directory that we just created. We can follow the instructions shown on the project page to verify the authenticity of this file. Once we trust this file, we can copy it to dom0 by opening a dom0 terminal and running:

[details=Make sure to understand the risks of copying files to dom0 before executing this command.]

qvm-run --pass-io disp8265 'cat wyng-backup-main/src/wyng' &gt; wyng

The file wyng is now in our home directory in dom0. We could mark this file as executable with the command chmod +x wyng and run Wyng directly from the command line, but we won't: we are going to use Salt!

3.2 Creating a BTRFS subvolume

When we installed Qubes OS on our machine, we decided to use BTRFS instead of the default partition scheme. To be able to use Wyng with BTRFS, the files that contain the filesystems of our qubes must be located inside of a BTRFS subvolume. This is not the case by default: the filesystems of our qubes are located under /var/lib/qubes, which is not a BTRFS subvolume but a regular directory.

qvm-shutdown --all --wait --force
sudo mv /var/lib/qubes /var/lib/qubes-old
sudo btrfs subvolume create /var/lib/qubes
shopt -s dotglob
sudo mv /var/lib/qubes-old/* /var/lib/qubes
sudo rmdir /var/lib/qubes-old

[details=Tip: There is a longer and riskier method that does not involve moving files across subvolumes.] 1. Shut down all qubes with qvm-shutdown --all --wait --force 2. Rename /var/lib/qubes to /var/lib/qubes-old 3. Create a snapshot of the root subvolume at the location /var/lib/qubes:

sudo btrfs subvolume snapshot / /var/lib/qubes

Once this is done, the output of the command sudo btrfs subvolume list / should contain a line that ends with /var/lib/qubes, and our system should function normally.

3.3 Making local backups

We would like to configure Wyng to create encrypted incremental backups that we will save on our machine in a service qube called "sys-backup". The idea is to back up our qubes regularly and, from time to time, we'll copy the backup archive from sys-backup to some external storage, or upload it to a remote server.

{# This file holds the backup passphrase and general configuration #}

{% set passphrase = 'my-backup-passphrase' %}
{% set qubes = ['vault', 'messaging'] %}
{% set source = '/var/lib/qubes' %}
{% set destination = '/home/user/qubes.backup' %}

# Install Wyng and create the service qube `sys-backup` for storing backups

{% set gui_user = salt['cmd.shell']('groupmems -l -g qubes') %}

backup--install-dependencies:
  pkg.installed:
    - pkgs:
      - python3-pycryptodomex
      - python3-zstd

backup--install-wyng:
  file.managed:
    - names:
      - /usr/local/bin/wyng:
        - source: salt://backup/files/wyng
        - mode: 755
      - /etc/wyng/wyng.ini:
        - source: salt://backup/files/wyng.ini
        - template: jinja
        - makedirs: True
    - require:
      - pkg: backup--install-dependencies

backup--create-service-qube:
  qvm.vm:
    - name: sys-backup
    - present:
      - template: debian-12
      - label: yellow
    - prefs:
      - template: debian-12
      - label: yellow
      - provides-network: True  # "service qube" (see Qubes issue #7298)
    - features:
      - set:
        - menu-items: org.gnome.Terminal.desktop org.gnome.Nautilus.desktop
    - require:
      - file: backup--install-wyng

backup--update-app-menu:
  cmd.run:
    - name: qvm-appmenus --update sys-backup
    - runas: {{ gui_user }}
    - require:
      - qvm: backup--create-service-qube

# Create a backup archive and add the backed-up qubes to its configuration

{% import 'backup/map.jinja' as backup %}

{% set volumes = backup.qubes | map('regex_replace', '(.+)', 'appvms/\\1/private.img') | join(' ') %}

backup.configure--create-archive:
  cmd.run:
    - name: echo '{{ backup.passphrase }}' | wyng arch-init --unattended
    - unless:
      - qvm-run sys-backup test -d {{ backup.destination }}

backup.configure--add-volumes:
  cmd.run:
    - name: echo '{{ backup.passphrase }}' | wyng add {{ volumes }} --unattended
    - require:
      - cmd: backup.configure--create-archive

# Remove the directory /home/user/.cache in each of the backed-up qubes

{% import 'backup/map.jinja' as backup %}

backup.clear-cache--shutdown-app-qubes:
  qvm.shutdown:
    - names: {{ backup.qubes }}

backup.clear-cache--clear-cache:
  qvm.vm:
    - names: {{ backup.qubes }}
    - actions:
      - run
      - shutdown
    - run:
      - cmd: rm --recursive --force /home/user/.cache
    - shutdown: []
    - require:
      - qvm: backup.clear-cache--shutdown-app-qubes

# Create a backup and copy it to the archive in `sys-backup`

{% import 'backup/map.jinja' as backup %}

include:
  - backup.configure
  - backup.clear-cache

backup.send--send-backup:
  cmd.run:
    - name: echo '{{ backup.passphrase }}' | wyng send --all --unattended
    - require:
      - cmd: backup.configure--add-volumes
      - qvm: backup.clear-cache--clear-cache

backup.send--shutdown-service-qube:
  qvm.shutdown:
    - name: sys-backup
    - require:
      - cmd: backup.send--send-backup

# Overwrite current data with the most recent backup from the archive

{% import 'backup/map.jinja' as backup %}

include:
  - backup.configure

backup.receive--shutdown-app-qubes:
  qvm.shutdown:
    - names: {{ backup.qubes }}
    - require:
      - cmd: backup.configure--add-volumes

backup.receive--receive-backup:
  cmd.run:
    - name: echo '{{ backup.passphrase }}' | wyng receive --all --unattended
    - require:
      - qvm: backup.receive--shutdown-app-qubes

backup.receive--shutdown-service-qube:
  qvm.shutdown:
    - name: sys-backup
    - require:
      - cmd: backup.receive--receive-backup

[details=/srv/user_salt/backup/files/wyng] This is the Wyng executable file that we have copied to dom0 in section 3.1. [/details]

{#- This file configures the default options for Wyng commands -#}

{%- import 'backup/map.jinja' as backup -%}

[var-global-default]
local = {{ backup.source }}
dest = qubes://sys-backup{{ backup.destination }}

This gives us a general configuration file map.jinja where we can enter our password, as well as the following Salt states:

sudo qubesctl state.sls backup saltenv=user

sudo qubesctl state.sls backup.configure saltenv=user

sudo qubesctl state.sls backup.clear-cache saltenv=user

sudo qubesctl state.sls backup.send saltenv=user

sudo qubesctl state.sls backup.receive saltenv=user

[details=Tip: For very long operations, running Wyng manually displays the backup progress in real-time.] Making backups manually requires performing more actions than simply applying the send.sls state file, but it is useful because it shows the real-time output of Wyng during the backup. It can be done by running the following command, preceded by sudo qubesctl state.sls backup.configure in case the archive is not yet configured:

sudo qubesctl state.sls backup.clear-cache &amp;&amp; sudo wyng send --all

It is also possible to restore a backup manually by running the following command after shuting down all the backed-up qubes manually:

sudo wyng receive --all

3.4 Making remote backups

3.5 Useful links

It took me quite a while to write this last part of the guide but I'm quite happy with the result. I tried to write it in a way that is easy to adapt to another Qubes instal. Of course, feel free to ask questions if something needs clarification or doesn't work. Have a nice day.

Qubes Salt Beginner's Guide

Part 1: Creating our first qubes