I recently had to help debug an issue with Xen, and I thought this might be useful to someone else who doesn’t have much experience with Xen.
When Xen crashes, it will generate a crash report with stack dump and backtrace. This is what happens during the “freeze”, the crash report is printed to the terminal, and 5 second later Xen reboots the host.
At this point, dom0 and any domUs are dead, and there is no easy way to get the crash report without having Xen output the report to a serial terminal. This is the most difficult part, because most modern systems don’t have easy accessible serial ports.
Most desktop systems should allow you to fairly easily connect a TTL/UART to USB cable to the motherboard using jumper pins, and the cable can be ordered on eBay or AliExpress for around $5.
It’s a different story with laptops, you can probably also connect a cable to a laptop, but you might not have standard jumper pins, and you would likely need to run the motherboard naked.
To enable the serial terminal, edit /etc/default/grub and add
loglvl=all guest_loglvl=all com1=115200,8n1 console=com1 to the GRUB_CMDLINE_XEN_DEFAULT line.
You are going to need a second device to read the serial console, connect usb cable to and second PC/laptop with picocom installed and run the command
picocom -b 115200 -r -l /dev/ttyUSB0
Check the manual before you try to connect anything to your hardware.
You might need to build Xen with debug options enabled, which works without using the Qubes builder. You need to clone the specific version of Xen used by the version of QubesOS you are running, e.g. 4.2.0 would be Xen 4.17, and you need to use the config from qubes-vmm-xen e.g. https://github.com/QubesOS/qubes-vmm-xen/blob/main/config.
Then you just need to set up the build environment for Xen and compile it, the result being xen.gz.
Now you can copy xen.gz into /boot and configure grub
grub2-mkconfig -o /boot/grub2/grub.cfg
You can now reboot the system and the new Xen should boot as default, if it doesn’t work you can select the previous version in the grub boot menu.