flakpyro

flakpyro

@stormi Installed on our 2 production pools, DR and remote sites, 46 hosts total ranging from Dell, Lenovo, HP, and Supermicro servers, no issues to report!

flakpyro

Installed on my usual selection of hosts. (A mixture of AMD and Intel hosts, SuperMicro, Asus, and Minisforum). No issues after a reboot, PCI Passthru, backups, etc continue to work smoothly. Also installed on a HP GL325 Gen 10 with no issues after reboot.

flakpyro

@gduperrey Updated my usual test hosts, (Minisforum and Supermicro X11) as well as an two sets of 2 host AMD pools (one pool of HP DL320 Gen10s and another of Asus Epyc servers of some sort, and lastly a Dell R360 without issue.

flakpyro

@gduperrey Installed on my usual round test hosts. No issues to report so far! With such a small change i wasn't expecting anything to go wrong!

flakpyro

Installed on my usual selection of hosts. (A mixture of AMD and Intel hosts, SuperMicro, Asus, and Minisforum). No issues after a reboot, PCI Passthru, backups, etc continue to work smoothly

flakpyro

One of our pools. (5 hosts, 6 NFS SRs) had this issue when we first deployed it. I engaged with support from Vates and they changed a setting that reduced the frequency of the SR.scan job from 30 seconds to every 2 mins instead. This totally fixed the issue for us going on a year and a half later.

I dug back in our documentation and found the command they gave us

    xe host-param-set other-config:auto-scan-interval=120 uuid=<Host UUID>

Where hosts UUID is your pool master.

flakpyro

@stormi Installed on my usual test hosts (Intel Minisforum MS-01, and Supermicro running a Xeon E-2336 CPU). Also installed onto a 2 host AMD epyc pool. Updates went smooth, backups continue to function as before.

3 windows 11 VMs had secure boot enabled. In XOA i clicked "Copy pool's default UEFI certificates to the VM" after the update was complete. The VMs continued to boot without issue after.

flakpyro

@gduperrey

installed on 2 test machines

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Both machines installed fine and all VMs came up without issue after. My one test backup job also seemed to run without any issues.

flakpyro

@gduperrey installed on 2 test machines

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Both machines installed fine and all VMs came up without issue after.

I ran a backup job after to test snapshot coalesce, no issues there.

flakpyro

@stormi Updated a test machine running only couple VMs. Everything installed fine and rebooted without issue.

Machine is:
Intel Xeon E-2336
SuperMicro board.
One VM happens to be windows based with an Nvidia GPU passed though to it running Blue Iris using the MSR fixed found elsewhere on these forums, fix continues to work with this version of Xen.

flakpyro

@andriy.sultanov

Running it now shows no error! And no output so looks like i have no issues!

flakpyro

@andriy.sultanov

Thanks

I ran that second script on the master and it did indeed find entries:

./snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:The VM d6befe36-fea7-04f3-25f1-feada684702b has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM e013d9ed-63b1-9c9f-a228-d48a6047688c has Ref:89 as its "snapshot_of" value, changing to null.
INFO:root:The VM 060df7ee-8af4-4c93-8265-bfa3024ccacd has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM da7b6ff6-d722-2120-7cc4-a1e0038a216e has Ref:307 as its "snapshot_of" value, changing to null.
INFO:root:The VM 4178b057-efee-427e-bcad-67323419ac7a has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8d624e49-ca99-4af6-b17c-38cad0646194 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM dc72da43-f144-3343-5ee5-a0dbffd209f6 has Ref:251 as its "snapshot_of" value, changing to null.
INFO:root:The VM e8c6376b-49e1-422b-551a-35420f70af2a has Ref:8 as its "snapshot_of" value, changing to null.
INFO:root:The VM 25eb14ab-e943-0d18-2137-c681e1191346 has Ref:11 as its "snapshot_of" value, changing to null.
INFO:root:The VM 6f819166-b008-4a03-a692-fcbd75018be9 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9d1a57e8-3531-4b55-9682-f775215866df has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 0ac88f7c-d79b-281e-cf1f-a2392ba594ed has Ref:29 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1f3541c4-8c81-4959-9cbe-edbec9481220 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...

after which it shows clean now:

./snapshot-fixer.py dry-run
INFO:root:Regenerating database...

However i am still getting the same error when trying to run leaked_vbs:

./leaked_vbds 
Traceback (most recent call last):
  File "./leaked_vbds", line 23, in <module>
    snap = xapi.VM.get_parent(snap)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 317, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 198, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 292, in _parse_result
    raise Failure(result['ErrorDescription'])
XenAPI.Failure: ['HANDLE_INVALID', 'VM', 'Ref:29']

flakpyro

@rzr

Is there something specific needed for the leaked_vbds script?

When i run it i get:

[09:37 xcpng-prd-01 ~]# ./leaked_vbds 
Traceback (most recent call last):
  File "./leaked_vbds", line 23, in <module>
    snap = xapi.VM.get_parent(snap)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 317, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 198, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 292, in _parse_result
    raise Failure(result['ErrorDescription'])
XenAPI.Failure: ['HANDLE_INVALID', 'VM', 'Ref:29']

flakpyro

No issues to report so far.

flakpyro

No issues on the same test hosts i used for the last 2 batches of updates.

flakpyro

Installed on my usual test hosts. No issues so far.

flakpyro

Installed on my usual hosts, one of which has an E810 and used the ICE driver, no issues so far however i am not using LACP bonding on that host.

flakpyro

Installed on my usual batch of test hosts, no issues so far.

flakpyro

Installed on my usual batch of test hosts, so far so good.

flakpyro

@Andrew I have experienced this twice so far as well.

flakpyro

@flakpyro