xcp host entering read only state

PGibson

Clean install of XCP-ng 8.3 (multiple attempts, I believe that I'm on the fourth try on a third boot disk). Dell Precision T7910, 2x4GB mirrored HDD (hardware raid, motherboard SAS HBA). Samsung EVO 870 SSD for boot , 32 GB RAM, 1 xenon CPU with 24 virtual CPUs (12 cores@ 2 threads maybe?).

This time, getting XOA installed from XO lite works great, it connects to the host after I put in the name/ IP (static), and I create a debian 12.0.9 VM.

Debian install done, VM is running after a reboot, I can log in to a shell and poke around, and install the XCP guest utilities from the ISO provided on the host. Another reboot, everything seems good, I go to bed.

This morning I'm going to backup the database and do a vm snapshot so that if I need to reinstall (again) I can get back running more quickly. Got into the system console directly on the host (not ssh), start a shell, and cannot make a directory in /mnt (read only filesystem error).

I am about to give up. I had Xen Debian running on this hardware for years with no issues (after I hacked it to get around the UEFI boot which took me a few frustrating hours/ days) which was the basis of my home lab. Xen project seems now to have upgraded and XCP seemed like the right direction to go, but rather than upgrade I backed up all of the important config/ user stuff in my DebianVM and started over. I want to virtualize TrueNAS by passing a PCI HBA directly to it, and have had that working on previous attempts until the XCP system starts to fail. I figure it's a disk error but it is happening now over multiple ssd boot disks. XOA seems to be stable, it is the host system that ends up failing every time. So I think my hardware raid is not the problem as this is the only SR for VMs.

System diagnostics from the BIOS yield no problems (disks, memory, PCI bus all green lights).

Any thoughts before I scrap XCP and look to something else? Or any suggestions for what else to try? Proxmox comes to mind, but I've heard it will auto mount ZFS pools even on passthrough so could be trouble for TrueNAS virtualization.

Danp

Hi @PGibson,

Are you storing ISOs in a local directory on the XCP-ng host? This is one of the major causes for the host having storage issues, where the root partition becomes full due to these ISOs.

If that isn't the problem, the I suggest reviewing the logs to try to identify the cause -- https://docs.xcp-ng.org/troubleshooting/log-files/

Regards, Dan

PGibson

@Danp Thanks, I do have two install images stored in a local iso storage repository. I should have plenty of space, as my boot drive is 500MB and my overall default storage repository is 4GB. I have removed the hardware mirror, and perhaps this will help. I'll also review the logs as you suggest.

Danp

Some of your numbers are confusing to me. For instance --

2x4GB mirrored HDD (hardware raid

I should have plenty of space, as my boot drive is 500MB and my overall default storage repository is 4GB.

The minimum disk size for installing XCP-ng is 46GB as shown here.

Can you show us the output of lsblk and df -h on your host?

PGibson

@Danp oh yes you are right . . . boot is 500GB and SR is 4TB, though I removed the hardware mirror and am just using the 2 4TB drives so I have a SR of close to 8TB. I think my problem has been that the hardware raid was not working well with XCP, and I also was trying to use some older samsung SSDs (also 500GB) both of which did fail a long test in the system diagnostics. So I have removed both of them from the machine and will see how things work now (after yet another reinstall because I was not having luck getting XCP to see the second 4TB HDD after removing the HW mirror).