I saw that Tristis Oris seems to have a similar problem with snapshots.

Something else, when updating from 8.2.1 to 8.3 RC.
My 8.2.1 servers were still running on BIOS boot. When updating to 8.3 RC I got the message that there was still a DOS table on the hard drive and the update from the USB stick was aborted.
A newly installed 8.3 server with UEFI cannot be integrated into the old 8.2 pool.

Here is a small workaround:
2 USB sticks, one with 8.2.1 and one with 8.3 RC.
Take one server out of the pool, not the master.
Reinstall with UEFI on 8.2.1.
Integrate server into the old pool.
make server as new pool master.
The server can now be updated to 8.3 without any problems, the master is on UEFI and all pool information and VM are on 8.3.
Now just take all other servers out of the pool, do a clean 8.3 UEFI installation and put them back into the pool.

Maybe that will help others with the same problem.

regards Joerg

@olivierlambert
Hello,
I created a new SR and installed a new VM and
installed a new XO.
The last one was definitely the answer.
It's working now, so as far as backup via XO goes.
I tested everything extensively, with different VM, SR and different drive fill levels, all ok.
A manual snapshot won't work, but that's what it is.
My old XO installation, unfortunately I didn't write down the version, was from January. Also, when I reinstalled, I switched from stable to latest as you indicated.
Unfortunately, I can't say whether the pure stable update or the change to latest was the solution.
But the main thing is a happy ending.
Greetings Joerg

Hello, here my test of version 8.3 in the last weeks:

I tried to put everything I could think of in or on the machine. Different hardware and different versions of Windows and Linux, backup of XO, with and without Xen tools, etc.
So far no crashes or major problems, even if I drove the machine to the limits of its resilience, everything runs smoothly and for days, for this a thumbs up.

Hardware:
Dell Poweredge 730, CPU 2x E5-2698 V4, 512GB RAM
2 x Intel I350 1Gb adapters
2 x Intel X540-AT2 10Gb adapters
1 x Dell H730P mono Raid Controller (5 x 8TB Disk in Raid5)
2 x SSD in Raid 1 as Boot Drive
1 x PCie NVMe Adapter (4 x 2TB NVMe Disk in Softraid 5)
( Yummi, over 1 GByte/sec write speed in a VM with 2 TB of Data)
1x Nvidia K80 GPU card

This is without a doubt the biggest test machine I've ever had.

I have tested so far:
All standard functions (Copy, Move, Migrate, Snapshot etc.)
Use of GPU (Windows VM)
PCI passthrough (Windows, Linux - NetCard, USB, PCI card)
SR-IOV (see comments below)
Backup with XO
Heavy network load (copy 26TB of data via 10 GB netcard)
Heavy CPU and GPU load (8 VM with CPU at maximum for hours)
Fast copying of large data between the different SRs in the system.

With the exception of the SR-IOV, no problems were encountered and the performance was excellent in all respects.

What made me very happy was the installation of Xen-Tools under Windows 2022. I have often had the experience that after a Windows update the server no longer wanted to start due to a driver update. The problem seems to have disappeared completely, all drivers were installed automatically when the server was installed and have so far survived all updates from Microsoft without any problems. I only had to manually add the management agent.

SR IOV:
The two Intel I350 1Gb adapters no longer show up as SR-IOV adapters and have lost that functionality. They still worked under 8.2.
The Intel X540-AT2 adapters have the SR-IOV function. But when I use it, the adapter port shuts down after a short time. The Xen server still shows the card as connectet, but the network function is gone. The coupled switch shows the port as deactivated. In between, the network function is there again for a short time and the switch also shows this. The second 10 GB port runs error-free all the time. If I switch off the SR-IOV of the port, it works without any problems. Both as a normal Xen-Nic and in PCI passthrough. I copied TByte over the port, no errors.
It must be somehow due to the SR-IOV that apparently no longer works under 8.3.
I would be interested to know if others have experienced something similar or if everything works there.

I would like to test the "VM snapshot with disk exclusion" but somehow I can't do it. Both the snapshot and the XO always back up the entire VM with all disks. I'm sure it's error 50 (50cm in front of the keyboard) . Is there a detailed description of how to set it up somewhere?

Unfortunately I haven't been able to test any pool functions yet, so I have to set up a second machine for that. I can't get a second machine that size. I will probably have to build 2 smaller systems with shared storage and test them there.

If anyone thinks of anything else I could test, let me know

So far everything looks very good, you did a great job.

Greetings Joerg

@stormi
Hello, here the results on my server:

Test server:
Dell Poweredge R730
2 x Xenon E5-2698v4
512GB RAM

./xtf-runner selftest -q --host
Combined test results:
test-hvm32-selftest                      SUCCESS
test-hvm32pae-selftest                   SUCCESS
test-hvm32pse-selftest                   SUCCESS
test-hvm64-selftest                      SUCCESS
test-pv64-selftest                       SUCCESS

./xtf-runner -aqq --host
Combined test results:
test-hvm32-umip                          SKIP
test-hvm64-umip                          SKIP
test-pv64-xsa-167                        SKIP
test-pv64-xsa-182                        SKIP
Echo Result 3

/usr/libexec/xen/bin/test-cpu-policy
CPU Policy unit tests
Testing CPU vendor identification:
Testing CPUID serialise success:
Testing CPUID deserialise failure:
Testing CPUID out-of-range clearing:
Testing MSR serialise success:
Testing MSR deserialise failure:
Testing policy compatibility success:
Testing policy compatibility failure:
Done: all ok
Echo Result 0

@tjkreidl
Hello again,
thanks for your help, you put me on the right path.
I wrote to Olivier what I dit.
Greetings Jörg

Status quo:
I have now reduced the problem to the NBD.
With an NBD full backup, everything seems to be working.
With a delta backup, NBD leaves snapshots that are blocked by the control domain of the various Xen servers. These can only be disconnected/forgotten and then deleted after a reboot of the hole pool or individual reboot of all Xen servers in the pool. The last option at least gives you the chance of a reboot without shutting down all VMs, if the pool has several Xen servers. But all servers in the pool must be rebooted.
Unfortunately, an xe-tool restart is not enough.
The activation/deactivation of CBT does not matter.
All backups without NBD run without errors.
This error occurs on all Xen servers installed from USB Image 8.3 officially. My lab pool, which was updated from 8.3RC2 to 8.3 Official via yum, doesn't seem to have the problem. 1 full backup and 4 delta backups ran without errors and without blocked snapshots.

As I urgently need a current backup, I deactivated all NBD backups and set the server interface back to NO NBD. The standard delta backups are now running without errors again.

I left my LAB pool on NBD with CBT to see if the error occurs there in the next few days.

I actually wanted to create a new topic for the problem, but I'm not really sure which category I should put it under.

@tjkreidl
Yes I believe that too

@stormi
should have been and and not or.
so
and then update with iso 8.3 Of.
Had an old server that kept crashing during the 8.3 Of. installation. It worked with RC1 and then upgraded to 8.3 Of.
Sometime this old things are a bit tricky

Hello,
I also had problems installing Beta 2 on older servers.
Switch from BIOS to UEFI under Boot in the BIOS.
Take a USB stick with the RC1, install and update with yum update
or
just use the new 8.3 Official.

@olivierlambert
Unfortunately, I don't have much time in the next few days.
On Sunday I will reset one of the pools with the error to NBD / CBT Delta Backup and then pull the logs and compare them with the one from the working 9.3. Let's see if I can see anything there. If not, I will send you the logs. I will also take a closer look at which processes are stuck there. It looks to me like there are ghost processes running there that don't terminate but seem to continue to block the snapshot. That also seems to be the reason why the snapshot cannot be deleted, except after a reboot.
If I have time, I will also set up a new pool with 9.3 RC2 and then update it to 9.3 officially via yum, just to see if it can then also do error-free NBD deltas.
Regards Joerg

@Tristis-Oris @Mathieu
Hello, I'm glad I could help you, but it would be nice if the NBD worked. I was able to save a lot of space on the storage with NBD and CBT. I hope that in the not too distant future it will work again like in 8.2.1
My 8.3 LAB pool, updated with yum, made an error-free delta backup with NBD and CBT last night, just like my last 8.2.1 pool.
regards Joerg

Status quo:
I have now reduced the problem to the NBD.
With an NBD full backup, everything seems to be working.
With a delta backup, NBD leaves snapshots that are blocked by the control domain of the various Xen servers. These can only be disconnected/forgotten and then deleted after a reboot of the hole pool or individual reboot of all Xen servers in the pool. The last option at least gives you the chance of a reboot without shutting down all VMs, if the pool has several Xen servers. But all servers in the pool must be rebooted.
Unfortunately, an xe-tool restart is not enough.
The activation/deactivation of CBT does not matter.
All backups without NBD run without errors.
This error occurs on all Xen servers installed from USB Image 8.3 officially. My lab pool, which was updated from 8.3RC2 to 8.3 Official via yum, doesn't seem to have the problem. 1 full backup and 4 delta backups ran without errors and without blocked snapshots.

As I urgently need a current backup, I deactivated all NBD backups and set the server interface back to NO NBD. The standard delta backups are now running without errors again.

I left my LAB pool on NBD with CBT to see if the error occurs there in the next few days.

I actually wanted to create a new topic for the problem, but I'm not really sure which category I should put it under.

@Tristis-Oris Not in your case?

In the Release Information about 8.3 you can read:
"Users who installed a prerelease of XCP-ng 8.3 must upgrade to the final 8.3.0 version using the installation ISO image. The only exception is for users who installed XCP-ng 8.3 RC2 or have already upgraded to RC2 using the installation ISO image. These users can simply update their system without needing the ISO image."

@stormi
Test with removed the NDB and CBT in the backup and removed the NBD interface in the pool.
Fullbackup Okay
Multiple delta backups all Okay no errors.
The error lies in the NBD or CBT.
But why not on the pool updated with yum from 8.3 RC2 ????

Can someone create a new topic?

@stormi
Of course, you shouldn't have a VM on the local storage.
It's also a good idea to write down the IPs, MACs and the number of the NICs beforehand. It can happen during the new installation that the order of the network cards changes.
Final tip: if you use iSCSI storage and use the iSCSI IQN as identification, you should also write this down beforehand and set it on the newly installed server before you put the server in the pool, otherwise you will have problems with the iSCSI storage.

@stormi
Sorry, I was a bit confused this morning when I saw all the errors. Of course it wasn't an update to 8.3 RC but to 8.3.0 Official.
Very strange thing, I checked my test lab server, it's the only pool that doesn't have the error. Full backup and several delta backups, all OK.
The only difference between this and the other pools is that this one was running with 8.3 RC2 and was only updated to 8.3.0 official with yum update.
I have now cleaned a faulty pool, removed all snapshots, removed the NDB and CBT in the backup and removed the NBD interface in the pool. I'm currently doing a full backup on this pool and will then do several deltas to narrow down the error a bit.
Regards, Joerg

I saw that Tristis Oris seems to have a similar problem with snapshots.

Something else, when updating from 8.2.1 to 8.3 RC.
My 8.2.1 servers were still running on BIOS boot. When updating to 8.3 RC I got the message that there was still a DOS table on the hard drive and the update from the USB stick was aborted.
A newly installed 8.3 server with UEFI cannot be integrated into the old 8.2 pool.

Here is a small workaround:
2 USB sticks, one with 8.2.1 and one with 8.3 RC.
Take one server out of the pool, not the master.
Reinstall with UEFI on 8.2.1.
Integrate server into the old pool.
make server as new pool master.
The server can now be updated to 8.3 without any problems, the master is on UEFI and all pool information and VM are on 8.3.
Now just take all other servers out of the pool, do a clean 8.3 UEFI installation and put them back into the pool.

Maybe that will help others with the same problem.

regards Joerg