@stormi Installed on our 2 production pools, DR and remote sites, 46 hosts total ranging from Dell, Lenovo, HP, and Supermicro servers, no issues to report!

@gduperrey

installed on 2 test machines

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Both machines installed fine and all VMs came up without issue after. My one test backup job also seemed to run without any issues.

@gduperrey installed on 2 test machines

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Both machines installed fine and all VMs came up without issue after.

I ran a backup job after to test snapshot coalesce, no issues there.

@stormi Updated a test machine running only couple VMs. Everything installed fine and rebooted without issue.

Machine is:
Intel Xeon E-2336
SuperMicro board.
One VM happens to be windows based with an Nvidia GPU passed though to it running Blue Iris using the MSR fixed found elsewhere on these forums, fix continues to work with this version of Xen.

@dthenot @olivierlambert thanks guys ill hold off on submitting a ticket for now to keep the conversation centralized here but if you need any more info, would like me to try anything or would like a remote support tunnel opened just let me know!

@CodeMercenary I am using V4 on the XO-Server to our backup remotes and it seems to work just fine. However using V4 as a storage SR was nothing but problems, as @nikade mentioned we had tons of NFS Server not responding issues which would lock up hosts and VMs causing downtime. Since moving to v3 that hasn't happened.

Checking a host's NFS retransmissions stats after 9 days of uptime i see we have had some retransmissions but they have not caused any downtime or even any timeout messages to appear in dmesg on the host.

[xcpng-prd-02 ~]# nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
268513028   169        268537542

From what a gather from this blog post from redhat (https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat) it seems like that amount of retransmissions is VERY low and not an issue.

@stormi

Tested on my usual Intel test servers you've seen me post in this thread in the past as well as a 2 host AMD Epyc pool,(HP DL325 Gen10) rolling pool reboot worked as expected, hosts rebooted without issue, backups and replication appear to function as usual.

Until the XO update is released by Vates you can filter your PV Driver version using the following:

pvDriversVersion:!9.4.1

This will show you all VMs not running that version. If you have Linux VMs they will be shown in this filter. If you have tagged your windows VMs you can combine this with a tag for example:

power_state:running pvDriversVersion:!9.4.1 tags:/^Windows$/ 

Remove the ! to filter for VMs that are running the latest 9.4.1 version.

@stormi Updated both of my test hosts.

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Everything rebooted and came up fine. The MS-01 i test with uses i40e and intel-igc not the e1000 driver. The other machine with the SuperMicro board uses igb so im afraid i'm not much help in testing that driver.

yum commands did seem to work from the small handful i ran.

And yes, stats do indeed work again

I never noticed the issue with Server 2025 and hanging on reboot since the updates from last week. Were you able to see anything in the dump files i sent?

@stormi installed on the same test machines i have the other batch of updates installed on. No issues after a reboot.

Updated around 43 hosts without issue.

@gduperrey im getting the following trying to update:

yum clean metadata && yum check-update
Loaded plugins: fastestmirror
Cleaning repos: xcp-ng-base xcp-ng-updates
4 metadata files removed
3 sqlite files removed
0 metadata files removed
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
xcp-ng-base/signature                                                                                                                                                   |  473 B  00:00:00     
xcp-ng-base/signature                                                                                                                                                   | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/signature                                                                                                                                                |  473 B  00:00:00     
xcp-ng-updates/signature                                                                                                                                                | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/primary_db      FAILED                                          
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/0a94f9d57be162987a7e5162b4471809af2ee0897e97cc24187077fb58804380-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found-:--:-- ETA 
Trying other mirror.
To address this issue please refer to the below wiki article 

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

(1/2): xcp-ng-base/primary_db                                                                                                                                           | 3.9 MB  00:00:02     
xcp-ng-updates/primary_db      FAILED                                          
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/0a94f9d57be162987a7e5162b4471809af2ee0897e97cc24187077fb58804380-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found-:--:-- ETA 
Trying other mirror.
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/0a94f9d57be162987a7e5162b4471809af2ee0897e97cc24187077fb58804380-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.


 One of the configured repositories failed (XCP-ng Updates Repository),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=xcp-ng-updates ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable xcp-ng-updates
        or
            subscription-manager repos --disable=xcp-ng-updates

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=xcp-ng-updates.skip_if_unavailable=true

failure: repodata/0a94f9d57be162987a7e5162b4471809af2ee0897e97cc24187077fb58804380-primary.sqlite.bz2 from xcp-ng-updates: [Errno 256] No more mirrors to try.
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/0a94f9d57be162987a7e5162b4471809af2ee0897e97cc24187077fb58804380-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found

Perhaps mirrors are still syncing?

@stormi

Tested on my usual Intel test servers you've seen me post in this thread in the past as well as a 2 host AMD Epyc pool,(HP DL325 Gen10) rolling pool reboot worked as expected, hosts rebooted without issue, backups and replication appear to function as usual.

@TrapoSAMA Have you tried 3.6.0? The package is named "amsd-3.6.0-1867.26.xenserver8.x86_64.rpm" you should be able to find it on the HPE support site

Edit: i found a link to the RPM: https://downloads.linux.hpe.com/repo/mcp/XenServer/8/x86_64/current/amsd-3.6.0-1867.26.xenserver8.x86_64.rpm

I am currently running "amsd-3.6.0-1867.26.xenserver8.x86_64.rpm" on my DL360 Gen 11 servers. It's been working well!

Its also worth mentioning i had no luck with the newer amsd rpms as they have dependency errors. I am assuming just because the dom0 OS is CentOS 7 based and somewhat dated. I reached out to my HP rep about if the xenserver packages will continue to see updates but never heard back. Having to deal with a daemon like this to keep fan speeds under control will definitely be taking into account with this years server refreshes considering Dell and Lenovo do not have this requirement.

Echoing what @olivierlambert mentioned. I migrated a number of Almalinux VMs and they too would not boot until adding the Xen PV drivers.

@gduperrey Updated both of my test hosts.

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H (e-cores disabled)
32 GB Ram
Using Intel X710 onboard NIC

Everything rebooted and came up fine. Ran a backup job which also completed without issue. Using a mix of ext, XFS and NFS SRs

This seems to be fixed in the latest commit and also does not happen in the just released XOA 5.108.0

@florent i have NBD enabled. Whats strange is i updated builds twice last week and backups continued to run. (Though progress in the tasks view disappeared as pointed out in this thread).

Last night we had a storm that knocked power out and my UPS's ran out of juice taking my home lab down, when everything came back up after backups would no longer run and would show ""invalid HTTP header in response body". I rolled back to a earlier commit (c7167 June 5th) and now not only are backups working again but progress in the tasks view is also displaying properly again.

Edit: Moved to commit 7994f from June 23rd and i still see progress there as well.

Mine was working on the latest commit however tonight due to a power failure my XO VM rebooted and now i too am experiencing the same errors. My hosts also rebooted as a result. Rolled back to a build from earlier this month and all is well again.