installed on 2 test machines
Machine 1:
Intel Xeon E-2336
SuperMicro board.
Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC
Both machines installed fine and all VMs came up without issue after. My one test backup job also seemed to run without any issues.
@gduperrey installed on 2 test machines
Machine 1:
Intel Xeon E-2336
SuperMicro board.
Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC
Both machines installed fine and all VMs came up without issue after.
I ran a backup job after to test snapshot coalesce, no issues there.
@stormi Updated a test machine running only couple VMs. Everything installed fine and rebooted without issue.
Machine is:
Intel Xeon E-2336
SuperMicro board.
One VM happens to be windows based with an Nvidia GPU passed though to it running Blue Iris using the MSR fixed found elsewhere on these forums, fix continues to work with this version of Xen. 
@dthenot @olivierlambert thanks guys ill hold off on submitting a ticket for now to keep the conversation centralized here but if you need any more info, would like me to try anything or would like a remote support tunnel opened just let me know! 
@CodeMercenary I am using V4 on the XO-Server to our backup remotes and it seems to work just fine. However using V4 as a storage SR was nothing but problems, as @nikade mentioned we had tons of NFS Server not responding issues which would lock up hosts and VMs causing downtime. Since moving to v3 that hasn't happened.
Checking a host's NFS retransmissions stats after 9 days of uptime i see we have had some retransmissions but they have not caused any downtime or even any timeout messages to appear in dmesg on the host.
[xcpng-prd-02 ~]# nfsstat -rc
Client rpc stats:
calls retrans authrefrsh
268513028 169 268537542
From what a gather from this blog post from redhat (https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat) it seems like that amount of retransmissions is VERY low and not an issue.
@stormi installed on the same test machines i have the other batch of updates installed on. No issues after a reboot.
@gduperrey Installed on the same 2 hosts as the last batch of test updates released in December.
No issues to report so far, ran a backup job after after without issue.
@gduperrey Installed on a 2 host AMD based test pool, as well as our 5 host Intel based production pool without issue using rolling pool update. Everything migrated, updated then migrated back perfectly.
Also installed on my home server without issue.
@bleader Updated a small test server without any issues.
@olivierlambert Makes sense! I would schedule the replication to occur a couple hours after the backup runs are complete to ensure its a replica of all data and not a partial replica!
@olivierlambert Im making progress getting to the bottom of this thanks to some documentation from XenServer about using cbt-util.
You can use the cbt-util utility, which helps establish chain relationship. If the VDI snapshots are not linked by changed block metadata, you get errors like “SR_BACKEND_FAILURE_460”, “Failed to calculate changed blocks for given VDIs”, and “Source and target VDI are unrelated”.
Example usage of cbt-util:
cbt-util get –c –n <name of cbt log file>
The -c option prints the child log file UUID.
I cleared all CBT snapshots from my test VMs and run a full backup on each VM. Then ensured the CBT chain was consistent using cbt-util, the output was:
[14:22 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 867063fc-4d86-420a-9ad2-dfe1749ecbc1.cbtlog
1950d6a3-c6a9-4b0c-b79f-068dd44479cc
After the backup was complete i then migrated the VM to the second host in the pool and ran the same command from both hosts:
[14:26 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 867063fc-4d86-420a-9ad2-dfe1749ecbc1.cbtlog
00000000-0000-0000-0000-000000000000
And from the second host:
[14:26 xcpng-test-02 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 867063fc-4d86-420a-9ad2-dfe1749ecbc1.cbtlog
00000000-0000-0000-0000-000000000000
That clearly is the problem right there, question is, what is causing that to happen?
After running another full the zero'd out cbtlog file is removed and a new one is created which will work fine until the VM is migrated again:
[14:39 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 1eefb7bf-9dc3-4830-8352-441a77412576.cbtlog
1950d6a3-c6a9-4b0c-b79f-068dd44479cc
We use ZFS on our backup target, mounted to XOA via NFS. Works without issue.
@rtjdamen Makes sense, it would be nice is XOA cleaned up these small CBT snapshots though after a migration from one SR to another.
Same issue here, i posted on discord about it but perhaps the forum is a better place for it.
I see the the newer releases of XO from source adds CBT to backups and enables it by default. Anyone else having issues with this feature? All my VMs were stuck "exporting content" in my test environment this morning, also looping tasks consisting of "call: sr.getAllUnhealthyVdiChainsLength" and under the health dashboard all my VDIs were unhealthy needing to coalesce.
What fixed it for me was to roll back to a pre CBT release, and remove all the .cbtlog files from my SRs, i had to delete all my VM snapshots, let garbage collection run, then ran a fresh backup run.
@florent Thats great news! We use both Borg and Veeam so that's where most of these ideas are coming from. Having GFS backups like you mention for weekly, monthly, yearly would also be awesome to have and something we make use of in Veeam currently.
@DustinB Are the any downsides to having two XOA instances pointing at the same pool? Since the config itself is stored at the pool level im guessing theres no downside?
IE: Priimary XOA running in core DC and secondary XOA running at your DR site. Is it just a matter of adding the pool on the secondary XOA and it downloads the existing config or did you need to do a full export / import?
@stormi Updated both of my test hosts. Everything rebooted and came up fine.
No VM stats in XO / XOA i see still. I will be curious if this round of updates fixes my EFI / Windows Server reboot hangs.
@mbven Strange we have a few Windows Server 2016 servers remaining that seem to be working since since the last batch of updates. I even tried to reboot one just to be sure. They are running "Microsoft Windows Server 2016 Datacenter". What CPU are your servers running? Any errors in xl dmesg when the crash occurs?
@stormi Updated without issues however noticed it dumped some errors to the console on reboot.
Checking into it i see the following in xensource.log
/var/log/xensource.log-Sep 25 12:12:29 xcpng-prd-02 xcp-networkd: [ info||131 |server_init D:2740ee30f2da|network_utils] /usr/bin/ovs-vsctl --timeout=20 --bare -f table -- --columns=name find port fake_bridge=true tag=199
/var/log/xensource.log:Sep 25 12:12:29 xcpng-prd-02 xcp-networkd: [error||131 |server_init D:2740ee30f2da|network_utils] Error in read one line of file: /sys/class/net/eth3/features, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/eth3/features")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "ocaml/libs/xapi-stdext/lib/xapi-stdext-unix/unixext.ml", line 92, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "ocaml/libs/xapi-stdext/lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 2-52\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "ocaml/libs/xapi-stdext/lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 160, characters 6-33\x0A
**/var/log/xensource.log-Sep 25 12:12:29 xcpng-prd-02 xcp-networkd: [error||131 |server_init D:2740ee30f2da|network_utils] Caught unix error: No such file or directory [access, /usr/sbin/ovs-vlan-bug-workaround]**
/var/log/xensource.log-Sep 25 12:12:29 xcpng-prd-02 xcp-networkd: [error||131 |server_init D:2740ee30f2da|network_utils] Assuming script /usr/sbin/ovs-vlan-bug-workaround doesn't exist
Looking on hosts i haven't let patched i see this exists on them as well. It doesn't seem to affect anything. All hosts were ISO upgraded to RC2 last week.
@manilx experienced the same thing while upgrading our test pool. Pool master did not ask, other hosts did ask for network config.