XCP-ng 8.3 betas and RCs feedback 🚀

jhansen

In the Release Information about 8.3 you can read:
"Users who installed a prerelease of XCP-ng 8.3 must upgrade to the final 8.3.0 version using the installation ISO image. The only exception is for users who installed XCP-ng 8.3 RC2 or have already upgraded to RC2 using the installation ISO image. These users can simply update their system without needing the ISO image."

Tristis Oris

@jhansen ah. maybe only this way.

jhansen

Status quo:
I have now reduced the problem to the NBD.
With an NBD full backup, everything seems to be working.
With a delta backup, NBD leaves snapshots that are blocked by the control domain of the various Xen servers. These can only be disconnected/forgotten and then deleted after a reboot of the hole pool or individual reboot of all Xen servers in the pool. The last option at least gives you the chance of a reboot without shutting down all VMs, if the pool has several Xen servers. But all servers in the pool must be rebooted.
Unfortunately, an xe-tool restart is not enough.
The activation/deactivation of CBT does not matter.
All backups without NBD run without errors.
This error occurs on all Xen servers installed from USB Image 8.3 officially. My lab pool, which was updated from 8.3RC2 to 8.3 Official via yum, doesn't seem to have the problem. 1 full backup and 4 delta backups ran without errors and without blocked snapshots.

As I urgently need a current backup, I deactivated all NBD backups and set the server interface back to NO NBD. The standard delta backups are now running without errors again.

I left my LAB pool on NBD with CBT to see if the error occurs there in the next few days.

I actually wanted to create a new topic for the problem, but I'm not really sure which category I should put it under.

Tristis Oris

i also removed one 8.3 pool from nbd backup, create the usual one. Looks it work fine.

Mathieu

Hello,

Indeed, disabling NBD for the delta backups, they are no more VDI attached to Dom0.

My upgrade path was from 8.2.1 stable to 8.3 using the ISO installation.

jhansen

@Tristis-Oris @Mathieu
Hello, I'm glad I could help you, but it would be nice if the NBD worked. I was able to save a lot of space on the storage with NBD and CBT. I hope that in the not too distant future it will work again like in 8.2.1
My 8.3 LAB pool, updated with yum, made an error-free delta backup with NBD and CBT last night, just like my last 8.2.1 pool.
regards Joerg

olivierlambert

Since you are able to reproduce consistently the issue, it might be interesting to get more logs/debug to find the root cause Pinging @dthenot and all relevant people that could help in here

yann

@Ajmind-0 there may be more details in the log a bit before those last lines, and they would not be visible because of the screen size and volume of logs. You can switch to the console with a shell and have a look into /tmp/install-log, there may be some more useful info in there.

jhansen

@olivierlambert
Unfortunately, I don't have much time in the next few days.
On Sunday I will reset one of the pools with the error to NBD / CBT Delta Backup and then pull the logs and compare them with the one from the working 9.3. Let's see if I can see anything there. If not, I will send you the logs. I will also take a closer look at which processes are stuck there. It looks to me like there are ghost processes running there that don't terminate but seem to continue to block the snapshot. That also seems to be the reason why the snapshot cannot be deleted, except after a reboot.
If I have time, I will also set up a new pool with 9.3 RC2 and then update it to 9.3 officially via yum, just to see if it can then also do error-free NBD deltas.
Regards Joerg

dthenot

@jhansen
Hello,
I created a thread about the NBD issue where VBD are left connected to Dom0.
I added what I already know about the situation on my side.
Anyone observing the error can help us by sharing what they observed in the thread.

https://xcp-ng.org/forum/topic/9864/vdi-staying-connected-to-dom0-when-using-nbd-backups

Thanks

Ajmind 0

@yann

the systems freezes after the last screen and it is diffult to catch the log just before the freeze...

yann

@Ajmind-0 does it still freeze if you add atexit=shell on the linux commandline? In some error situations the installer attempts to reboot the system after a few seconds, this will make it drop into a shell instead.

Finallf

@Ajmind-0 try using the save option at the beginning of the installation.

stormi

@Finallf I think @Ajmind-0 already tried booting the installer in safe mode.

Ajmind 0

I have started the installer again in the BIOS mode and have finished this process successful. It seems to be that the "EFI" Mode from the old intel server is the problem.

As this old equipment is only used as a testing playground I could stay with it as it is. The hardware will never be used as a production system.

yann

@Ajmind-0 this was an install not an upgrade of 8.2.1, right? Was your previous working install of 8.2.1 in UEFI mode?

Ajmind 0

@yann
Yes, it was a fresh install. I have removed any partition layout from the Hardware RAID1 via gparted before I have started from scratch.

The previous 8.2.1 install was also created in BIOS mode. For me it is clear that this old EFI Intel BIOS from 2012 lacks support for this kind of install process.

john.c

@Ajmind-0 said in XCP-ng 8.3 betas and RCs feedback :

@yann
Yes, it was a fresh install. I have removed any partition layout from the Hardware RAID1 via gparted before I have started from scratch.

The previous 8.2.1 install was also created in BIOS mode. For me it is clear that this old EFI Intel BIOS from 2012 lacks support for this kind of install process.

What model of Intel Server is it?

Ajmind 0

@john-c
Intel Server (SR2625/S5520UR / Dual Xeon E5645)

laurentm

@Ajmind-0 Same problem with DELL PowerEdge R820 and DELL PowerEdge R520:

I had to upgrade in BIOS mode and it was the same thing for a fresh install.

With DELL PowerEdge R240, no problem to upgfrade in UEFI.