Dug a little deeper. For a VM where the disks are not shown the following XO API call fails:

/rest/v0/vms/a519e879-3971-9210-51b6-7df14336e7b7/vdis
{
  "error": "no such VDI ac37700d-3157-4df7-b8e8-e1799a994591",
  "data": {
    "id": "ac37700d-3157-4df7-b8e8-e1799a994591",
    "type": [
      "VDI"
    ]
  }
}

Also the VDI cannot be retrieved over the XO API:

/rest/v0/vms/a519e879-3971-9210-51b6-7df14336e7b7
...
 "$VBDs": [
    "4ea8a3cd-0d1b-dc60-4d9c-fd70e060f06c",
    "9f4ca686-9fc2-35a9-c3e9-c871c9f68aba"
  ],
...

/rest/v0/vbds/9f4ca686-9fc2-35a9-c3e9-c871c9f68aba
{
  "type": "VBD",
  "attached": false,
  "bootable": false,
  "device": "xvda",
  "is_cd_drive": false,
  "position": "0",
  "read_only": false,
  "VDI": "ac37700d-3157-4df7-b8e8-e1799a994591",
  "VM": "a519e879-3971-9210-51b6-7df14336e7b7",
  "id": "9f4ca686-9fc2-35a9-c3e9-c871c9f68aba",
  "uuid": "9f4ca686-9fc2-35a9-c3e9-c871c9f68aba",
  "$pool": "93d361b7-f549-53b7-a3aa-c9695bf0abe4",
  "$poolId": "93d361b7-f549-53b7-a3aa-c9695bf0abe4",
  "_xapiRef": "OpaqueRef:1d424d94-f540-2eb4-9e52-2a9b21ec0a19"
}

/rest/v0/vdis/ac37700d-3157-4df7-b8e8-e1799a994591
{
  "error": "no such VDI ac37700d-3157-4df7-b8e8-e1799a994591",
  "data": {
    "id": "ac37700d-3157-4df7-b8e8-e1799a994591",
    "type": "VDI"
  }
}

However the VDI can be listed using the xe cli:

$ xe vm-list uuid=a519e879-3971-9210-51b6-7df14336e7b7
uuid ( RO)           : a519e879-3971-9210-51b6-7df14336e7b7
     name-label ( RW): XXX
    power-state ( RO): halted

$ xe vbd-list vm-uuid=a519e879-3971-9210-51b6-7df14336e7b7
uuid ( RO)             : 4ea8a3cd-0d1b-dc60-4d9c-fd70e060f06c
          vm-uuid ( RO): a519e879-3971-9210-51b6-7df14336e7b7
    vm-name-label ( RO): XXX
         vdi-uuid ( RO): <not in database>
            empty ( RO): true
           device ( RO): xvdd


uuid ( RO)             : 9f4ca686-9fc2-35a9-c3e9-c871c9f68aba
          vm-uuid ( RO): a519e879-3971-9210-51b6-7df14336e7b7
    vm-name-label ( RO): XXX
         vdi-uuid ( RO): ac37700d-3157-4df7-b8e8-e1799a994591
            empty ( RO): false
           device ( RO): xvda

$ xe vdi-list uuid=ac37700d-3157-4df7-b8e8-e1799a994591
uuid ( RO)                : ac37700d-3157-4df7-b8e8-e1799a994591
          name-label ( RW): XXX Disk 0
    name-description ( RW): Created by XO
             sr-uuid ( RO): 977b7e63-bb84-57b2-3e0d-206afea553bf
        virtual-size ( RO): 34359738368
            sharable ( RO): false
           read-only ( RO): false

Seems almost like something changed in the XCP-ng API which XO cannot consume.

Another interesting observation. On a VM where we took a snapshot today (post Windows Update install) and deleted an older snapshot, the disk shows up (both v5 and v6).

Same problem here. But I noticed it only appears with the latest updates from XCP-ng 8.3. The log file shows me the following packages were updated / installed on my host:

May 07 19:44:54 Updated: xen-libs-4.17.6-6.2.xcpng8.3.x86_64
May 07 19:44:54 Updated: gnutls-3.3.29-10.2.xcpng8.3.x86_64
May 07 19:44:54 Updated: 1:net-snmp-libs-5.9.3-8.2.xcpng8.3.x86_64
May 07 19:44:54 Updated: ipmitool-1.8.19-11.2.xcpng8.3.x86_64
May 07 19:44:54 Updated: openssh-9.8p1-1.2.3.xcpng8.3.x86_64
May 07 19:44:54 Updated: openssh-clients-9.8p1-1.2.3.xcpng8.3.x86_64
May 07 19:44:54 Updated: openssh-server-9.8p1-1.2.3.xcpng8.3.x86_64
May 07 19:44:54 Updated: 1:net-snmp-agent-libs-5.9.3-8.2.xcpng8.3.x86_64
May 07 19:44:55 Updated: 1:net-snmp-5.9.3-8.2.xcpng8.3.x86_64
May 07 19:44:55 Updated: blktap-3.55.5-6.7.xcpng8.3.x86_64
May 07 19:44:55 Updated: message-switch-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:55 Updated: xcp-ng-xapi-plugins-1.16.0-1.xcpng8.3.noarch
May 07 19:44:55 Updated: xen-hypervisor-4.17.6-6.2.xcpng8.3.x86_64
May 07 19:44:55 Updated: xen-dom0-libs-4.17.6-6.2.xcpng8.3.x86_64
May 07 19:44:56 Updated: vhd-tool-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:56 Updated: squeezed-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:56 Updated: xen-tools-4.17.6-6.2.xcpng8.3.x86_64
May 07 19:44:56 Updated: xen-dom0-tools-4.17.6-6.2.xcpng8.3.x86_64
May 07 19:44:57 Updated: xcp-rrdd-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:57 Updated: xapi-tests-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:57 Updated: xo-lite-0.20.0-1.xcpng8.3.noarch
May 07 19:44:57 Updated: wsproxy-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:58 Updated: xapi-storage-script-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:58 Updated: xcp-networkd-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:59 Updated: forkexecd-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:59 Updated: sm-cli-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:44:59 Updated: xapi-rrd2csv-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:00 Updated: rrdd-plugins-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:01 Updated: xapi-nbd-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:01 Updated: sm-fairlock-3.2.12-17.8.xcpng8.3.x86_64
May 07 19:45:01 Updated: sm-3.2.12-17.8.xcpng8.3.x86_64
May 07 19:45:01 Updated: xenopsd-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:01 Updated: xenopsd-cli-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:02 Updated: xenopsd-xc-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:05 Updated: xcp-ng-pv-tools-8.3-17.xcpng8.3.noarch
May 07 19:45:05 Installed: 3:traceroute-2.1.5-2.xcpng8.3.x86_64
May 07 19:45:05 Updated: xapi-xe-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:05 Updated: qcow-stream-tool-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:08 Updated: xapi-core-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:08 Updated: xcp-ng-deps-8.3-14.noarch
May 07 19:45:08 Updated: gnutls-utils-3.3.29-10.2.xcpng8.3.x86_64
May 07 19:45:08 Updated: gnutls-devel-3.3.29-10.2.xcpng8.3.x86_64
May 07 19:45:09 Updated: varstored-guard-26.1.3-1.10.xcpng8.3.x86_64
May 07 19:45:12 Updated: kernel-4.19.19-8.0.46.2.xcpng8.3.x86_64

So I guess one of these packages could be the culprit?

@manilx Try installing the .msi from an elevated powershell prompt. That worked for one of our Windows 11 VMs.

@dinhngtu That indeed seems to work. I just realized, that this is a timing issue as well. Sometimes the copy&paste works after a slight delay, where the console seems non-responsive (e.g. cannot move the mouse cursor). After something like ~15 secs things work as expected. For subsequent copy&paste operations the delay seems to be much lower.

We noticed that using the XCP-ng Windows PV tools brakes some functionality of the shared clipboard. On a fully patched XCP-ng 8.3 host using Xen Orchestra 5 (actually the 6.2.2 build, but the v5 interface) allows only to copy the clipboard from the VM to the browser, not the other way around.

@archw I can confirm. That is exactly the behaviour I see with my Windows VMs.

@dinhngtu Thanks for the pointer. Yes, it seems that the root cause also makes routes disappear. Howerver, that the routing information is gone is sadly not mentioned explicitly. May be something to add to your docs as well.

Caution when updating tools: Verify interface IP configuration and routing entries.

Just did a couple more tests. Here are my findings:

• Upgrading the tools from v9.3.3 to v9.4.1 does preserve the routing table.
• Upgrading the tools from v9.2.1 to v9.4.1 does not preserve the routing table.

Here are a couple of powershell commands used for testing:

Get-NetRoute -PolicyStore PersistentStore
Get-NetAdapter
New-NetRoute -DestinationPrefix "10.10.0.0/24" -InterfaceIndex <ifIndex> -NextHop 10.10.0.254

@pdonias Sure thing. I can test it in my test environment.

@DustinB Not IP assignments, I am talking about static routes. See e.g. https://learn.microsoft.com/en-us/powershell/module/nettcpip/get-netroute

Here is another interesting fact: After installing the new tools (v.9.4.1) my static routes in Windows were all gone. Definitively a good way to loose connectivity to your domain controller. And that's why you have good monitoring and store things in Ansible et al. ...

@dinhngtu On the machine where it worked, the option "Manage Citrix PV drivers via Windows Update" was not enabled. Seems that my older BIOS Windows 10 VMs have that option enabled. On all UEFI VMs the options is disabled.

As I wanted to go and check that is present in the templates, I realized that the Windows Templates are gone from Xen Orchestra v5.106.4???

On another Windows 10 host it worked. What was different: I saw the message box "Tools have been installed successfully". May be that makes a difference?

@dinhngtu On a Windows 10 VM rebooting alone did not do the trick. After 5 reboots the script still reports vulnerable devices:

.\Install-XSA468Workaround.ps1 -Scan

Looking for vulnerable XenIface objects
Found vulnerable object XENBUS\VEN_XSC000&DEV_IFACE\_
Found vulnerable object XENBUS\VEN_XSC000&DEV_IFACE\_

Looking for vulnerable XenIface WMI GUIDs
Found vulnerable WMI GUID 1D80EB99-A1D6-4492-B62F-8B4549FF0B5E
Found vulnerable WMI GUID 12138A69-97B2-49DD-B9DE-54749AABC789
Found vulnerable WMI GUID AB8136BF-8EA7-420D-ADAD-89C83E587925

Found XenIface vulnerability, it's recommended to run the script
True

Running .\Install-XSA468Workaround.ps1 works as expected. After another reboot nothing is reported as being vulnerable anymore.

On a Windows 2019 Server I saw the behaviour you described: Installing the tools and a reboot was enough.

@dinhngtu Ok, I will keep that in mind as I go through all the VMs. As I currently cannot update XCP-ng on all hosts (8.2.1 LTS), the VMs where the new tools were installed and mitigations applied show up as "orange".

On a XCP-ng 8.3 test hosts with all updates applied the detection works as advertised.

After upgrading the VM tools the mitigation script still shows vulnerable devices on some hosts. After running the script to apply the mitigations, nothing is reported as being vulnerable anymore.

Thus the question: Is applying the mitigations a necessary action as well? Or does installing the v9.4.1 tools fix the vulnerability?

@stormi I have seen this state clearing for two VM migrations on the same pool. The hardware on all machines is identical and the migration is from one machine to the other, so no cross-pool migration involved.

What we have also observed after the fact, is that in Xen orchestra it states that the VM has been created on the day of the migration, not the day the VM was actually created. So it seems as it was indeed "re-created" after the migration.

For "failed" machines it says:

Created by Unknown
on 2024-08-25 18:11
with template Windows Server 2019 (64-bit)

For machines which were not migration but created at the same instant:

Created by Unknown
on 2022-01-07 10:09
with template Windows Server 2019 (64-bit)

@stormi This did not work on a test system. The command simply errored out.

Thanks for your insights. After a lot of trial and error we were able to get the VM back online with secure boot enabled. The recovery was as follows:

• Disable secure boot in Xen orchestra for the VM
• Boot Windows without secure boot
• Drop into the UEFI firmware settings via shutdown /f /r /o /t 0 and selecting Troubleshoot -> Advanced Options -> UEFI Firmware Settings
• Then select Boot Maintenance Manager -> Boot from File
• Select the right volume and browse to EFI\Microsoft\Boot
• Select SecureBootRecovery.efi and hit enter to start the program, this will re-apply the certificate "Windows UEFI CA 2023" to the secure boot DB