Thanks for letting us know, and I'm happy you have thing working nicely now.
I think to mark this as resolved you need to convert your original post as a question, and it can then be marked as resolved. I actually cannot do it myself, I think only a few people have the permission to do it for others at Vates.

Wow, that's not great indeed.

The console too small, is pretty odd, as this should be at least the standard 80x24 for any kind of VGA.

Have you tried, from the idrac to login on the third terminal (alt+F3) ? It would be good to see if you can login or if the update + failing disk really broke everything. On the 2nd terminal (alt+F2) you should have system messages too, which in the ideal case should be empty…

Can you clarify a few points:

• is the SMB server always the same and the windows server, 11 and debian 12 are just client?
• Is the server a physical host outside of your pool or a VM?
• what is the server a windows, linux, freenas or a physical NAS?

It's just to get a better understanding of the setup.

On the guest tools and PV drivers side I'm not familiar with them myself on windows, maybe @dinhngtu could have some more insight?

Updated my machine at home, no issue so far.

We have 2 packages updated for the first 8.3 security update, a bit late to be part of the final ISO but they will be available at release time. For people willing to test them and provide feedback, see the announcement below.

---

New security update candidates (xen, intel-microcode)

A new XSA was published on September 24th 2024.
Intel published a microcode update on the September 10th 2024.

---

• XSA-462 a malicious HVM or PVH guest can trigger a DoS of the host.

---

SECURITY UPDATES

• xen-*:
      * Fix XSA-462 - x86: Deadlock in vlapic_error(). The handling of x86's APIC (Advanced Programmable Interrupt Controller) allows a guest to configure an illegal vector to handle error interrupts. This causes the vlapic_error() to recurse, this is protected, but the lock used for this protection will try to be taken recursiveley, leading to a deadlock.
• intel-microcode:
      * Latest Intel microcode update, still named IPU 2024.3, including security updates for:
          * INTEL-SA-01103
          * INTEL-SA-01097

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" intel-microcode --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• xen: xen-4.17.5-3.xcpng8.3
• intel_microcode: intel-microcode-20240815-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the update

Until 8.3 release.

Update published: https://xcp-ng.org/blog/2024/09/27/september-2024-security-updates/

Thank you for the tests!

So if the initrd has been regenerated, it's likely the second part that fails, otherwise it is dracut, which is more of an issue.

@Andrew If I understand correctly that is ran after regenerating the initrd, but to be sure, can you check the date of your /boot/initrd-4.19.0+1.img on this host?

New security update candidates (xen, microcode_ctl)

A new XSA was published on September 24th 2024.
Intel published a microcode update on the September 10th 2024.
We also included an updated xcp-ng-release for testing, althrough not related to security.

---

• XSA-462 a malicious HVM or PVH guest can trigger a DoS of the host.

---

SECURITY UPDATES

• xen-*:
      * Fix XSA-462 - x86: Deadlock in vlapic_error(). The handling of x86's APIC (Advanced Programmable Interrupt Controller) allows a guest to configure an illegal vector to handle error interrupts. This causes the vlapic_error() to recurse, this is protected, but the lock used for this protection will try to be taken recursiveley, leading to a deadlock.
• microcode_ctl:
      * Latest Intel microcode update, still named IPU 2024.3, including security updates for:
          * INTEL-SA-01103
          * INTEL-SA-01097

Other updates

• xcp-ng-release:
      * Update the "XOA Quick deploy" feature in the host's web page.
      *  Point at repo.vates.tech for CentOS since mirrorlist.centos.org was cut
      * Add "(EOL)" to repo descriptions for EOL repos
      * Drop unused repos

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" microcode_ctl xcp-ng-release --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• xen: 4.13.5-9.44.1.xcpng8.2
• microcode_ctl: microcode_ctl-2.1-26.xs29.5.xcpng8.2
• xcp-ng-release: xcp-ng-release-8.2.1-13

What to test

Normal use and anything else you want to test.

Test window before official release of the update

~ 1 day because of security updates.

Update published https://xcp-ng.org/blog/2024/08/16/august-security-update/

Thank you all for testing

Thierry is on holidays for now

It is more likely due to the fact the ice driver is either coming from the intel-ice or intel-ice-alt which are both for the main kernel, the -alt one being a more up to date version, and not as the confusing name could imply, for the kernel-alt.

The kernel-alt is in any case a debugging and testing kernel and not meant for production, but your tests confirmed that this seems to fix the fan speed issue, now we can discuss internally about what to do.

New security update candidates (xen, microcode_ctl)

A new XSA was published on August 14th 2024.
Intel published an updated microcode on August 13th 2024.

---

• XSA-460 passing through some PCI devices after guest creation could lead to any kind of security issue (DoS, privilege escalation, information leak,…) the actual risk depends on the systems and the devices passed through.
• XSA-461 some pass-through use cases when on one or more device when they share resources are actually not possible to be made secure. This XSA updates the Xen documentation to reflect that.

---

SECURITY UPDATES

• xen-*:
  • Fix XSA-460 - error handling in x86 IOMMU identity mapping. The handling of errors in the case of mapping Reserved Memory Regions was flawed, potentially keeping the mapping in place, this could allow guests to access memory region they should not be allowed to access. These mapping are typically used when doing pass-through of legacy USB emulation.
  • Document XSA-461 - PCI device pass-through with shared resources Files. According to the Xen Project security team, PCI pass-through of devices that share ressources is not possible to be made safe. This updates the documentation to reflect it. Look at the "MITIGATION" section of the XSA for more information regarding the safe cases.
• microcode_ctl: Security updates from Intel:
  • INTEL-SA-01083
  • INTEL-SA-01118
  • INTEL-SA-01100
  • INTEL-SA-01038
  • INTEL-SA-01046
  • Plus fixes for a lot of functional issues, see the Release Notes for more details.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" microcode_ctl --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• xen: 4.13.5-9.40.3.xcpng8.2
• microcode_ctl: 2.1-26.xs29.3.xcpng8.2

What to test

Normal use and anything else you want to test. Feedback about PCI pass-through is also welcome.

Test window before official release of the update

~ 2 day because of security updates.

Olivier is currently on holidays, so I'll try to answer: it will mostly depend where the fix is implemented by XenServer, they do have some closed source parts, if that fix goes in such a part, we won't be able to get it back. If it is implemented in some of the open source part, we should be able to get it and integrate the change on our side.

Let's hope the fix ends up on the right side

yes you're basically doing an RPU manually.
But it is indeed odd that the process is stuck at 0%, it should be fairly fast to do the install patches, no errors in the logs?

It actually depends if you chose "rolling pool update" or "Install all pool patches", as you're talking about evacuate I assume you went with the first one.

Rolling pool update (RPU) is documented here and Pool updates here.

But to sum it up, "Install all pool patches" button will indeed run the yum update on all server, so similar as doing it manually, while RPU will do hosts one by one, moving VM to other hosts in the pool, install updates and reboot host. Therefore it can take way longer to complete, time will vary based on the number of VMs that have to be migrated around, network speed between hosts, etc…

RPU is the recommanded way as it allows hosts to restart and therefore take hypervisor, microcode and dom0 kernel updates into account right away with no service interruption. But if you don't really mind shutting down some of the VMs to restart hosts, or if there are no low level updates that requires a reboot, you could get away with just the yum update manually. But if the RPU is started already, I would not advise trying to do things manually at the same time.

Update published https://xcp-ng.org/blog/2024/07/18/july-2024-security-updates/

Thank you everyone for your tests!

I indeed forgot to wait for mirrors to sync before posting, my bad, should be good soon

New security update candidate (xen, xapi, xsconsole)

Two new XSAs were published on 16th of July.

---

• XSA-458 guests which have a multi-vector MSI capable device passed through to them can leverage the vulnerability.
• XSA-459 impacts systems running Xapi v3.249.x, which means any up to date XCP-ng 8.2. Note this requires heavy crafting and likely social engineering on the attacker side, see the XSA's "VULNERABLE SYSTEMS" section for more details.

---

SECURITY UPDATES

• xen-*:
  • Fix XSA-458 - double unlock in x86 guest IRQ handling. When passing through a multi-vector MSI capable device to a guest, an attacker could use an error handling path that could lead to the issue, no exploitations results have been ruled out: Denial of Service (DoS), crashes, information leaks, or elevation of privilege could all be possible.
• xapi, xsconsole:
  • Fix XSA-459 - Xapi: Metadata injection attack against backup/restore functionality. A malicious guest can manipulate its disk to appear to be a metadata
    backup, then having about a 50% chance of appearing ahead of a legitimate metadata backup. The more disks the guest has, the higher the chances of this happening are.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" "xapi-*" xsconsole --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• xen: xen-4.13.5-9.40.2.xcpng8.2
• xapi: xapi-1.249.36-1.2.xcpng8.2
• xsconsole: xsconsole-10.1.13-1.2.xcpng8.2

What to test

Normal use and anything else you want to test.

Test window before official release of the update

~ 1 day because of security updates.

We're still actively working on it, we're still not a 100% sure what the root cause is unfortunately.

It does seem to affect all Zen generations, from what we could gather, sligthly differently: it seems to be a bit better on zen3 and 4, but still always leading to underwhelming network performance for such machines.

---

To provide some status/context to you guys: I worked on this internally for a while, then as I had to attend other tasks we hired external help, which gave us some insight but no solution, and now we have @andSmv working on it (but not this week as he's at the Xen Summit).

From the contractors we had, we found that grant table and event channels have more occurences than on an intel xeon, looking like we're having more packet processed at first, but then they took way more time.

What Andrei found most recently is that PV & PVH (which we do not support officially), are getting about twice the performance of HVM and PVHVM. Also, having both dom0 and a guest pinned to a single physical core is also having better results. It seems to indicate it may come from the handling of cache coherency and could be related to guest memory settings that differs between intel and amd. That's what is under investigation right now, but we're unsure there will be any possibilty to change that.

---

I hope this helps make things a bit clearer to you guys, and shows we do invest a lot of time and money digging into this.

Do you use wireguard on your OPNsense? We did have an issues with FreeBSD based systems and wireguard, but if I remember properly it is supposed to be fixed, still worth asking