After a 7 year discussion on logging, a Qubes pull request was recently created for new patches to allow the creation of a logging qube.

Both DisposableVM's and AppVM's currently discard all their logs when they are shut down, so this was desperately needed. (note: verify that that situation hasn't changed for AppVM's)

This thread is the start of a guide for how to set up a logging qube that uses those new patches. We can use this thread to collect all the logging qube related information with the hope of having a complete guide by the time the patches are integrated into upstream.

Remember, it is a work in progress, still with many open questions to get feedback on. Please shout out if you have a answer, see a potential security issue that isn't addressed, or have other input.

First: Know your goal Before setting up a logging qube, you should determine your goal. While logs can be great, if your goal is maximum privacy, and your never going to look at the logs anyway, then you might be best off having no logs at all.

However, many people will want logs for one of the following reasons:

1. Debugging. Using logs to figure out why things aren't working so they can fix it.
2. Forensic analysis. Trying to figuring out if a qube was comprimised whenever they suspect it has been. This adds the additional potential complication of trying to figure out if the logging qube itself has been comprimised letting a attacker alter the logs that you are using to try to determine if there was a comprimise.

Now that you know why you want logs, you are ready to make decisions relating to them and we can move on to how to set up a logging qube

Enabling logging in qubes TODO: put instructions on git cloneing the syslog branch of https://github.com/3hhh/qubes-core-agent-linux.git instead of the normal version, then compiling the qubes package, and installing the new qubes package. probably link to build instructions (then remove this section once it's integrated upstream)

Setting up your templateVMs to log to logVM 1. Make sure rsyslog is installed in the template. - it should already be installed in debian based distributions, but if you want to check, you can with: apt-cache policy rsyslog. If it's not there, you can install it in debian with: apt-get install rsyslog - You can install it in fedora based distributions with: dnf install rsyslog - Windows: one option is: choco install nxlog 2. Configure rsyslog by modifying your rsyslog configuration file at /etc/rsyslog.d/qubes_syslog.conf as described at https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67 except in the line, binary="/usr/sbin/runuser -u user -- /usr/bin/qrexec-client-vm work qubes.Syslog" , change work to sys-log so it looks like: binary="/usr/sbin/runuser -u user -- /usr/bin/qrexec-client-vm work qubes.Syslog" (side note: it should be possible to add a copy of this section pointed to sys-log-forward to allow 2 seperate logging systems) 3. Then shut the template down and reboot any qubes you want to use it.

1. TODO: Add instructions specific windows instructions for configuring nxlog?

Configuring the logVM to read the logs 1. Option: Networking: Here we will be talking about creating a sys-log qube with no networking. You also have the option to create a logging qube with networking that can forward the logs to a external logging system (that I recommend calling this qube sys-log-forwarding so you have the option to have both logging qubes) 1. Option - VM type: Decide if you want to use a AppVM that uses a template, or create a standalone VM (Note: if it's using a template you just added the rsyslog configuration to, then you probably need to disable the /etc/rsyslog.d/qubes_syslog.conf file or it could create a feedback loop. :) Note: While rsyslog is not strictly necessary for the logVM, you probably will want to use it. 3. Create a new AppVM service qube called sys-log 4. Check the "provides network" box for the new qube, even though the qube does not provide networking, or even have networking. That checkbox is what determines if the qube is classified as a "service" qube or not. 2. Set the qubes policy in dom0 by creating a file at /etc/qubes-rpc/policy/qubes.Syslog containing:

```
sendingVM receivingVM allow
```

(Other options: If your planning to log all qubes, you could do: @anyvm receivingVM allow, or if your just testing, you could do: @anyvm @anyvm ask)

1. Copy the file to https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog to be /etc/qubes-rpc/qubes.Syslog in the sys-log qube.

2. Test it! Generate some logs in the sending VM by typing logger "hello world" in a appVM that has been rebooted and is using your new modified template. Then notice that the message can be observed in the journalctl -b0 of the sys-log qube!

3. The logs in sys-log will still dissapear after it reboots, so now we make the configuration and logs persistant
4. do: sudo mkdir -p /rw/config/qubes-bind-dirs.d

5. do: sudo nano /rw/config/qubes-bind-dirs.d/50_user.conf

6. add these lines:

   binds+=( '/etc/qubes-rpc/qubes.Syslog' )
   
   binds+=( '/var/log' )
   

OPTIONAL: Configuring a secondary external log outside of dom0 If dom0 is ever compromised, then we should expect sys-log to be compromised. If sys-log is ever compromised, weather via dom0, or via another method, then the attacker can go back in time in the logs to the old logs that showed the compromise when it happened, and change them to cover up the compromise. Having a logging system outside of dom0 gives us a opportunity to have a copy of the logs that have the potential to keep a untained copy of a dom0 comprimise. Note that sys-log reads data from every qube. This seems like a possible attack vector. While they try to midigate that potential in the patches, it seems like a possible attack vector that can't be eliminated while still being functional. Also note that this same data will get forwarded to the external log.

A completely separate reason for having a external log is that if you have more then one qubes system, you can have that one external log aggregating all the logs from all the qubes systems in one place.

Steps: 1. TODO: put instructions for either enabling networking on the logVM, or maybe it's possible to make a separate sys-log-forward qube that has networking that also gets a copy of the logs from the qubes-rpc service, allowing us to leave sys-log with no networking? 2. set up a external computer with a operating system. 2.a Options: 2.a.1 Set up a normal computer with a linux distribution other then Qubes (actually Qubes is more of a xen distribution, but that's beside the point). Recommended distributions. Debian is normal. Would we want to advise a security oriented distribution? Also mention that BSD is fine to install, but most people are probably not familiar with it. 2.a.2 Set up another computer with Qubes on it. (would require pointing to a guide for opening up listening ports to the outside world for a qubes qube) (hardware that can run Qubes 4.0 is valuable enough to a Qubes user that it'd probably be getting used for Qubes. possibly discuss potential issues with using that qubes system for logging and for other things?) 2.a.3 Set up a raspberry pi. (Depending on the traffic generated (which depends on how many qubes computers you have pointed at it), it's possible that not all raspberry pi models can handle the traffic, so maybe some kind of guidance on that?). Guidance on operating systems? Recommended distributions? Standard is raspbian (now renamed to something else), which is a mangled version of debian. Recommend a distribution more security oriented? (Also, if you don't feel Qubesy enough running a raspberry pi note that xen supposedly works fine on raspberry pi nowdays) 2.b install rsyslog on it 2.c TODO: explain how to set up rsyslog to allow receiving from the IP address of the qubes system 2.d TODO: add SSL to explanation

Configuring a queryable database * Depending on if more then one qube can receive a copy of the logs, address if this should be yet another separate logging qube?
* TODO: explain how to set up rsyslog to save the logs in a database * possibly recommend querying interfaces? * address the database both as being a qube, and as the possibility of being a external copy

OTHER: * Maybe mention setting up retention policies of the logs? Maybe just find a external document for that and point to it? * Find guides about how to do debugging under qubes, and point those documents to this? * Find guides about doing forensic analysis when you suspect one or more qubes may have been compromised (I.E. some kind of "what to do if you suspect you may have been hacked" type article), and point those documents to this (with the caution that you need to set up the logging before the incident happens)

Significant open questions: - Should we recommend some specific security oriented operating systems for external logging (either amd64 or raspberry pi operating) instead of a normal one?