Go back to topic: WORKING: Guide for creating a logging qube (I.E. sys-log) - WIP
| - Should we recommend some specific security oriented operating systems for external logging (either amd64 or raspberry pi operating) instead of a normal one? **Alternate version** This is the start of a alternate version that while it wont work with all qubes (like windows qubes), it can work with many qubes. Its a systemd journal First, warning: systemd is excessively complicated, and it takes forever to figure out how to do the simplest things with it. Also, I don't have confirmation that this procedure works yet, as i'm in the middle of writing the procedure. in the sys-log qube, do: ``` sudo apt update; sudo apt install systemd-journal-remote sudo systemctl enable --now systemd-journal-remote.socket sudo systemctl enable systemd-journal-remote.service ``` this should start running the service on port 19532 The man page says you can test if its working by doing: ``` sudo systemctl enable systemd-journal-upload.service ``` But naturally, there is no command, and you'll get: ``` -bash: systemd-journal-remote: command not found ``` But that's just how the systemd user experience goes. :neutral_face: Anyway, moving on... In the templates of all the other qubes do: ``` sudo apt update; sudo apt install systemd-journal-remote sudo systemctl enable systemd-journal-upload.service ``` at this point, you aparently need to create a SSL certificate authority for your qubes system, add its certificate to the trusted certificates of the template of the sys-log qube (and possibly all of the qubes templates?), then go to every template of every qube and create a SSL certificate signing request, and send it to the SSL certificate authority qube, sign them, then send them back to the inidividual qubes and configure them in `/etc/systemd/journal-remote.conf` (not actually attempted because I'm expecting it to probably fail due to something checking the hostname and realizing that the certificate is not for disp2388, it's for tempalte-debian-12, and refusing to work. In this case we'd need to sign certificates for every qube, not just every template. |
| ***NOTE: While this guide is still a work in progress, it is now useful enough that many people should be able to use it to get logging working locally on their system*** | |
| This thread is the start of a guide for how to set up a logging qube that uses those new patches. We can use this thread to collect all the logging qube related information with the hope of having a complete guide by the time the patches are integrated into upstream. Remember, it is a work in progress, still with many open questions to get feedback on. Please shout out if you have a answer, see a potential security issue that isn't addressed, or have other input. | This thread is the start of a guide for how to set up a logging qube that uses those new patches. We can use this thread to collect all the logging qube related information with the hope of having a complete guide by the time the patches are integrated into upstream. Remember, it is still a work in progress, still with many open questions to get feedback on. Please shout out if you have a answer, see a potential security issue that isn't addressed, or have other input. |
| 5. TODO: We haven't mentioned getting logs from dom0 yet, which would be a important one. | |
| *Sending the logging traffic from sys-log:* Sending the logging traffic from sys-log can be as easy as editing the /etc/rsyslog.conf file on sys-log and adding ```*.* @{ip address of your external log collector}``` This is the simplest configuration that will copy all logs from sys-log to that remote system. They will be forwarded unencrypted. (TODO: talk about setting up encryption) *Set up a external computer to recieving the logging traffic coming from sys-log:* 1. Options: 1.a Set up a normal computer with a linux distribution other then Qubes (actually Qubes is more of a xen distribution, but that's beside the point). Recommended distributions. Debian is normal. Would we want to advise a security oriented distribution? Also mention that BSD is fine to install, but most people are probably not familiar with it. 1.b Set up another computer with Qubes on it. (would require pointing to a guide for opening up listening ports to the outside world for a qubes qube) (hardware that can run Qubes 4.0 is valuable enough to a Qubes user that it'd probably be getting used for Qubes. possibly discuss potential issues with using that qubes system for logging *and* for other things?) 1.c Set up a raspberry pi. (Depending on the traffic generated (which depends on *how many* qubes computers you have pointed at it), it's possible that not all raspberry pi models can handle the traffic, so maybe some kind of guidance on that?). Guidance on operating systems? Recommended distributions? Standard is raspbian (now renamed to something else), which is a mangled version of debian. Recommend a distribution more security oriented? (Also, if you don't feel Qubesy enough running a raspberry pi note that xen supposedly works fine on raspberry pi nowdays) 2. install rsyslog on it 3. TODO: explain how to set up rsyslog to allow receiving from the IP address of the qubes system 4. TODO: add SSL to explanation |
| ```qvm-create --template debian-10 --label-purple --property netvm=sys-firewall --property provides_network=True sys-log``` Or Create a sys-log-local qube with no networking: Example (typed from dom0): ```qvm-create --template debian-10 --label-purple --property netvm=none --property provides_network=True sys-log-local``` | ``` qvm-create --template debian-10 --label-purple --property netvm=sys-firewall --property provides_network=True sys-log qvm-tags sys-log add collects-logs ``` Or Create a sys-log-local qube with no networking: Example (typed from dom0): ``` qvm-create --template debian-10 --label-purple --property netvm=none --property provides_network=True sys-log-local qvm-tags sys-log-local add collects-logs ``` |
| sys-log sys-log | sys-log sys-log deny @tag:collects-logs sys-log deny |
| (Other options: If your planning to log all qubes, you could do: `@anyvm | (Other options: If your planning to log all qubes, you could do: ``` sys-log sys-log allow @anyvm sys-log allow ``` or if your just testing, you could do: `@anyvm @anyvm ask`) |
| 4. TODO: Add instructions specific windows instructions for configuring nxlog? **WARNING: Make sure your system will still be able to boot!** If you have added logging to the templates for the sys-net and sys-firewall qubes, then when you boot up, they will generate a significant amount of traffic which will get logged (more then 10000 lines each). The system will automatically try to boot sys-log, so all sounds fine. However during boot time, many other things are going on (for example the sys-net and sys-firewall qubes starting). This can slow down the system. If it slows it down enough so that it takes longer then 60 seconds to boot the sys-log qube, qrexec will kill the sys-log qube, and immediately try launching a new bootup of the sys-log qube, which causes new system load, and starts a infinte loop. Depending on the speed of your system, and how many things you configure in your auto-boot sequence(including possibly more the one sys-log qube), this may be a problem or not. There are 2 precautions to take: 1. Make backups of everything before you complete your logging configurations 2. Change the timeout for all qubes with: ```qubes-prefs default_qrexec_timeout 600``` or change the timeout for individual qubes with `qvm-prefs` |
| ** | **Setting up a sys-log qube to collect all the logs** |
| (Note: you could do both) | (Note: you could do both. A thread has been made for discussing [if there is a benefit to doing both](https://forum.qubes-os.org/t/what-risks-do-we-add-to-a-qube-by-enabling-sys-firewall-networking-instead-of-none/6649https://forum.qubes-os.org/t/what-risks-do-we-add-to-a-qube-by-enabling-sys-firewall-networking-instead-of-none/6649) ) |
| binds+=( '/run/log/journal' ) |
| (side note: it should be possible to add a copy of this section pointed to sys-log- | (side note: it should be possible to add a copy of this section pointed to sys-log-local to allow 2 seperate logging systems) |
| 1. Create | 1. Create a sys-log qube with networking: Example (typed from dom0): |
| Or Create a sys-log-local qube with no networking: Example (typed from dom0): ```qvm-create --template debian-10 --label-purple --property netvm=none --property provides_network=True sys-log-local``` (Note: you could do both) | |
| 1. TODO: put instructions for either enabling networking on the logVM, *or* maybe it's possible to make a separate sys-log-forward qube that has networking that also gets a copy of the logs from the qubes-rpc service, allowing us to leave sys-log with no networking? |
| Now that you know why you want logs, you are ready to make decisions relating to them and we can move on to how to set up a logging qube
| Now that you know why you want logs, you are ready to make decisions relating to them and we can move on to how to set up a logging qube **Install qubes.Syslog file** *(Note: You only need to do this if the patch has not been integrated upstream and distributed out through the normal update mechanisms yet.)* * Start the template that the qube that will collect the logs will use as its template. |
| As stated before, if it has been integrated upstream and you are up to date, then the file wil already be there, and this step is unnecicary. **Setting up your templateVMs to log to the sys-log logVM** | |
| **Configuring the | **Configuring the sys-log qube to collect all the logs** 1. Create the sys-log qube Example (typed from dom0): ```qvm-create --template debian-10 --label-purple --property netvm=sys-firewall --property provides_network=True sys-log``` Alternatives: * you could create it as a standalone qube instead of a AppVM that uses a template Notes: * We are basically checking the "provides network" box for the new qube, even though the qube does not provide networking, (or if creating sys-log-local does not even *have* networking. That checkbox is what determines if the qube is classified as a "service" qube in the menu or not.) Note: While rsyslog is not strictly necessary for the logVM, you probably will want to use it. |
| 3. Create a new AppVM service qube called sys-log 4. Check the "provides network" box for the new qube, even though the qube does not provide networking, or even have networking. That checkbox is what determines if the qube is classified as a "service" qube or not. | |
| 5. The logs in sys-log will still | 5. The logs in sys-log will still disappear after it reboots, so now we make the configuration and logs persistent |
| ``` | ``` 6. Networking options: * If you are not going to forward the logs to a external logging system, then instead of a sys-log qube, you could make a sys-log-local qube that has no networking. * If you are going to forward the logs but are concerned about the logging qube being attacked *from* the network, then you could create both sys-log with networking, and sys-log-local that has no networking. |
| Having a logging system outside of dom0 gives us a opportunity to have a copy of the logs that have the potential to keep a untained copy of a dom0 | Having a logging system outside of dom0 gives us a opportunity to have a copy of the logs that have the potential to keep a untained copy of a dom0 compromise. Note that sys-log reads data from every qube. This seems like a possible attack vector. While they try to mitigate that potential in the patches, it seems like a possible attack vector that can't be eliminated while still being functional. Also note that this same data will get forwarded to the external log. |
| * Turn off that template | * Turn off that template If it has been integrated upstream and you are up to date, then the file wil already be there, and this step is unnecicary. |
| You need to do this for each template that you want all the appVMs that use that template to send logs to sys-log: | |
| 3. Then shut the template down and reboot any qubes you want to | 3. Then shut the template down and reboot any qubes you want to be using it. |
| (Note: if it's using a template you just added the rsyslog configuration to, then you probably need to disable the /etc/rsyslog.d/qubes_syslog.conf file or it could create a feedback loop. :) | |
| 2. If sys-log is using a template you just added the rsyslog configuration to, then you probably need to disable the /etc/rsyslog.d/qubes_syslog.conf file or it could create a feedback loop. :) (we could put here a explanation on how to disable it via /rw bind) |
| ** | **If it has not been integrated upstream** If you read this before the patch gets integrated upstream and distributed out through the normal update mechanisms, then you can install it yourself by doing this: * Start the template that you intend to use for the qube to log to. * Copy the file to https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog to be /etc/qubes-rpc/qubes.Syslog in the template. * Turn off that template |
| (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`)
| (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`) |
| - add | - add this line: |
| binds+=( '/etc/qubes-rpc/qubes.Syslog' ) |
| - it should already be installed in debian based distributions, but if you want to check, you can with: `apt-cache policy rsyslog`. If it's not there, you can install it in debian with: `apt-get install rsyslog` - You can install it in fedora based distributions with: `dnf install rsyslog` - Windows: one option is: `choco install nxlog` | |
| - do: `sudo mkdir -p /rw/config/qubes-bind-dirs.d` - do: `sudo nano /rw/config/qubes-bind-dirs.d/50_user.conf` - add these lines: ``` binds+=( '/etc/qubes-rpc/qubes.Syslog' ) binds+=( '/var/log' ) ``` |
| ``` sendingVM receivingVM allow ``` (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`) | ``` sendingVM receivingVM allow ``` (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`) |
| ``` binds+=( '/etc/qubes-rpc/qubes.Syslog' ) binds+=( '/var/log' ) ``` | ``` binds+=( '/etc/qubes-rpc/qubes.Syslog' ) binds+=( '/var/log' ) ``` |
| 1. | 1. Option: Networking: Here we will be talking about creating a sys-log qube with no networking. You also have the option to create a logging qube with networking that can forward the logs to a external logging system (that I recommend calling this qube sys-log-forwarding so you have the option to have both logging qubes) 1. Option - VM type: Decide if you want to use a AppVM that uses a template, or create a standalone VM (Note: if it's using a template you just added the rsyslog configuration to, then you probably need to disable the /etc/rsyslog.d/qubes_syslog.conf file or it could create a feedback loop. :) Note: While rsyslog is not strictly necessary for the logVM, you probably will want to use it. 3. Create a new AppVM service qube called sys-log 4. Check the "provides network" box for the new qube, even though the qube does not provide networking, or even have networking. That checkbox is what determines if the qube is classified as a "service" qube or not. |
| 4. Copy the file to https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog to be /etc/qubes-rpc/qubes.Syslog in the sys-log qube. | |
| - Can more then one logging qube read full copies of all the logs being fed to the new service? (Update: Looks like the answer to this is yes!) |
| 5. | 5. The logs in sys-log will still dissapear after it reboots, so now we make the configuration and logs persistant 5.a do: `sudo mkdir -p /rw/config/qubes-bind-dirs.d` 5.b do: `sudo nano /rw/config/qubes-bind-dirs.d/50_user.conf` 5.c add these lines: ``` binds+=( '/etc/qubes-rpc/qubes.Syslog' ) binds+=( '/var/log' ) ``` |
| 4. TODO: put here instructions on how to bind-mount the /var/logs directory to file to /rw , so they dont still dissapear after a reboot :) | 4. TODO: put here instructions on how to bind-mount the /var/logs directory to file to /rw , so they dont still dissapear after a reboot :) (Note: link to instructions at https://www.qubes-os.org/doc/bind-dirs/ ?) |
| 1. Create a new AppVM service qube called sys-log. While rsyslog is not strictly necicary for the logVM, you probably will want to be basing the sys-log qube on a debian or fedora template that you just verified that rsyslog is installed on. 2. Set the qubes policy in dom0 by creating a file at `/etc/qubes-rpc/policy/qubes.Syslog` containing: | |
| (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`)
| (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`) 3. Copy the file to https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog to be /etc/qubes-rpc/qubes.Syslog in the sys-log qube. 4. Test it! Generate some logs in the sending VM by typing `logger "hello world"` in a appVM that has been rebooted and is using your new modified template. Then notice that the message can be observed in the `journalctl -b0` of the sys-log qube! 5. TODO: put here instructions on how to bind-mount the new /etc/qubes-rpc/qubes.Syslog file to /rw 4. TODO: put here instructions on how to bind-mount the /var/logs directory to file to /rw , so they dont still dissapear after a reboot :) |
| - Can more then one logging qube read full copies of all the logs being fed to the new service? | - Can more then one logging qube read full copies of all the logs being fed to the new service? (Update: Looks like the answer to this is yes!) |
| 2. Configure rsyslog by modifying your rsyslog configuration file at /etc/rsyslog.d/qubes_syslog.conf as described at [https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67 ](https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67)
| 2. Configure rsyslog by modifying your rsyslog configuration file at /etc/rsyslog.d/qubes_syslog.conf as described at [https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67 ](https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67) except in the line, ```binary="/usr/sbin/runuser -u user -- /usr/bin/qrexec-client-vm work qubes.Syslog"``` , change `work` to `sys-log` so it looks like: ```binary="/usr/sbin/runuser -u user -- /usr/bin/qrexec-client-vm work qubes.Syslog"``` (side note: it should be possible to add a copy of this section pointed to sys-log-forward to allow 2 seperate logging systems) |
| 1. | 1. Make sure rsyslog is installed in the template. 1.a it should already be installed in debian based distributions, but if you want to check, you can with: `apt-cache policy rsyslog`. If it's not there, you can install it in debian with: `apt-get install rsyslog` 1.b You can install it in fedora based distributions with: `dnf install rsyslog` 1.c Windows: one option is: `choco install nxlog` |
| except in the line, `binary="/usr/sbin/runuser -u user -- /usr/bin/qrexec-client-vm work qubes.Syslog"`, change `work` to `sys-log` |
| 2. Configure rsyslog by modifying your rsyslog configuration file at /etc/rsyslog.d/qubes_syslog.conf as described at [https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67 ](https://github.com/QubesOS/qubes-core-agent-linux/blob/009e973b1eb3bbb14df59a25bf52a31d5f5f5043/qubes-rpc/qubes.Syslog#L50-L67) 3. Then shut the template down and reboot any qubes you want to use it. 4. TODO: Add instructions specific windows instructions for configuring nxlog? | |
| * | * Set the qubes policy in dom0 by creating a file at `/etc/qubes-rpc/policy/qubes.Syslog` containing: ``` sendingVM receivingVM allow ``` (Other options: If your planning to log all qubes, you could do: `@anyvm receivingVM allow`, or if your just testing, you could do: `@anyvm @anyvm ask`) |