@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output. [ 108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE [ 108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202 [ 108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset [ 108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2 [ 108.547789] amdgpu 0000:00:09.0: Failed to evict process queues [ 108.547803] amdgpu: Failed to quiesce KFD [ 108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source: 3 [ 109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0 [ 109.657324] amdgpu 0000:00:09.0: Dumping IP State [ 109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed [ 112.047695] amdgpu 0000:00:09.0: MODE1 reset [ 112.047797] amdgpu 0000:00:09.0: GPU mode1 reset [ 112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset [ 113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume [ 113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000). [ 113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created [ 113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data [ 113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset! [ 113.092921] amdgpu 0000:00:09.0: PSP is resuming... [ 114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed [ 115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available [ 115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available [ 115.091134] amdgpu 0000:00:09.0: SMU is resuming... [ 115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0) [ 115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully! [ 115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000 [ 115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000 [ 115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800 As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...

@poddingue These are VMs that have been doing full backups fine for a very long time. I have run into the too-much-free-space issue in the past with a different brand new VM but these have been around since I migrated them from VMware during the Broadcom fiasco years ago. It is also totally inconsistent which VMs fail with the body timeout. One day it'll be a single one, the next day it'll be three or four and won't include that one, then it'll be a couple other ones unrelated to any of the earlier ones, then I'll have no failures. It's all over the place. I'm just glad that over the course of several days I get good backups of everything. Oh, a maybe useful data point is that my delta backups that all succeed are handled by a different instance of XO that runs on a different one of the hosts. I might be overly paranoid, but I run full backups and delta backups of the same VMs using different XO instances to different target remotes, at different non-overlapping times. This is part of the reason that I'm not hair-on-fire worried about the failing full backups. A while back I stupidly let one of my Storage Repositories run out of space and it was the one hosting my main XO instance that handled all the backups. It took a while to get it back going again and I realized I wanted redundancy in what handled backups and moved the deltas over to another XO instance. Too many eggs in that one basket. I also wasn't able to restore the backup of that XO instance because that XO instance is what handled the backups. Thankfully it all worked out in the end, plus a number of extra gray hairs it gave me.

@christopher-petzel Thanks!

@TrapoSAMA said: @john.c I have this iso but how to know what is de package? Thx It’s going to be an msi installer package or on the guest tools iso, which is on the XCP-ng or Xen Server your running likely in Local Storage SR or Shared Storage SR (if moved). Named the same as the file linked to above, likely. Anyway did you know that the UEFI SecureBoot certificates (at least one of them from 2011) issued by Microsoft are expiring fully during October 2026. Around that time likely at sometime after then those certificates will be placed into dbx. This means the Windows Server 2012 R2 if operating as a UEFI SecureBoot VM, will if restarted (or reboot following a crash), fail to boot. So your client will go from having a working app that they can use, but not upgrade at the moment to one which won’t work and can’t be used! They really need to upgrade or migrate from Windows to Linux (released during 2025 or 2026) along with the app if needed. Alternatively upgrade to Windows Server 2022 or 2025 and the app!

@poddingue I should already have all the logs, including kernel[image: 1783960418266-2ce97f35-954d-4f1b-88a5-ae0899e75d99-image.jpeg] Nothing has happened since. I am also increasing dom0 ram for all hosts.

@kagbasi-wgsdac said: @poddingue Bug report filed as requested — https://github.com/xcp-ng/xcp/issues/825 — and tagging @Team-Storage per your suggestion. Full evidence bundle is attached to the issue (versions, sweep output, vhd-util vs xe comparison, SMlog). Summary of what I found: One correction to the mechanism, and I think it matters. The recap describes is-a-snapshot being flipped to true. On my system that isn't what's happening — is-a-snapshot is false on every affected VDI. The field being wrongly written is snapshot-of, which is getting populated on base disks that aren't snapshots at all. XO's disappearing-disks symptom is consistent with either (it filters on a non-empty snapshot-of), but if the storage team is hunting for a bad is-a-snapshot write, that may be the wrong field. Every affected VDI here looks like: is-a-snapshot: false <-- correct snapshot-of: <populated with an unrelated VDI's UUID> <-- wrong A VDI that is a snapshot of itself. The clearest single artifact: uuid: 806f7f42-083f-4a40-b3f1-0700d00bab5a name-label: WinSrv2022SHB_Disk1_Data is-a-snapshot: false snapshot-of: 806f7f42-083f-4a40-b3f1-0700d00bab5a <-- itself snapshot-time: 20260709T11:19:15Z sm-config: vhd-parent: c86e3247-... <-- bears no relation to the snapshot-of value No valid code path produces snapshot-of = self. Whatever writes this field isn't validating the target. It's still actively corrupting new VDIs — this is not just legacy damage. That self-referential VDI was created 2026-07-09, a week after my patch + reboot. Sweeps 9 days apart went from ~180 → 191 affected VDIs on one SR, and a fourth anchor UUID appeared that didn't exist in the first sweep. Newly created VHDs keep landing in the affected set. So "stop it happening again" is the urgent half of the two-part fix, at least in my case. The bogus targets cluster onto a tiny anchor set, and the anchors point at each other: Count Anchor 97 937c3945 (→ a893fdb4) 50 a893fdb4 (→ ea150883) 37 ea150883 7 806f7f42 (→ itself, new since Jul 9) That looks less like corrupted lineage and more like the field being filled from an incorrect/uninitialised source. On-disk VHDs are completely healthy. vhd-util check says valid, parent locators are consistent, GC reports no work. The two VDIs the DB calls parent/child are, on disk, siblings under a common parent. The corruption is purely in the XAPI database — which is good news for recoverability. The VDI_IN_USE is not a real lock. current-operations is empty, xe task-list is empty, no tapdisk holds it. VM.start fails because it's walking a snapshot relationship that doesn't exist on disk. Reproduces from xe on the pool master with XO entirely out of the path — which is why I filed against xcp-ng/xcp rather than the XO tracker. Versions: XCP-ng 8.3.0, xapi 26.1.11 (xapi-core-26.1.11-1.2), sm-3.2.12-17.9, sm-fairlock-3.2.12-17.9, blktap 3.55.5-9.1, build 20260618. I have not attempted to bulk-clear the fields — on-disk data is intact and I'd rather not do a mass write against the XAPI DB on a live SR without guidance. Backing store snapshotted as a safety net. Happy to run whatever diagnostics would help. And +1 to the hand-grenade feeling — the affected set growing on its own is the part that worries me. Has this issue been validated on a storage server built around Debian 13, LVM and ext4 or just TrueNAS when connected to XCP-ng version 8.3.0. As part of the XAPI DB corruption. Can anyone answer this please or give a clue?

@poddingue I'm in agreement with all of this. I can clearly see the Intel card starting at slot 81 and ending at 84. Unfortunately, in XCP-ng 8.3, only 83 (the "gpu" itself) is visible as an assignable PCI pass-through device. When I pass through just slot 83, I get a dysfunctional result. Again...I'm a little out of my expertise here, but my assumption here is that in order to actually get the full GPU to work, I need to get slots 81-84 passed over to the VM. Problem...In XCP-ng 8.2, you could mask off PCI devices with the pciback.hide function: /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:04:01.0)(0000:00:19.0)" Or in my case: /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)" Then you would assign those same PCI devices to the guest VM. In 8.3 you do this with an xe command: xe pci-list and xe pci-disable-dom0-access uuid=<pci uuid> The bridge devices for the Intel card aren't being included in the pci-list of XCP-ng and are therefore not assignable to guests. For example, running "xe pci-list" on a host shows me this single entry relevant to the Intel card: uuid ( RO) : f3e6842b-ad1b-9e93-dbea-214dde28618a vendor-name ( RO): Intel Corporation device-name ( RO): Device e212 pci-id ( RO): 0000:83:00.0 With the new 8.3 method of assigning PCI resources to guest VM's, I don't see how I can fully pass-through the Intel GPU to a guest. I won't necessarily call this a "bug" but it is an unexpected and perhaps edge case behavior that Vates didn't expect.

@poddingue Hi, The backup mirror job is still running, and I'm waiting for the larger VMs to complete, but so far it looks good - one backup, larger than 50GB was successfully mirrored without specifying minPartSize. [image: 1783936023727-a5158a60-6607-4679-ac2d-ee9f84c02e4e-image.jpeg]

I converted the topic to a question, then marked it solved. Thanks!

Your write-up is the kind of thing that saves the next person a weekend. One thing worth knowing before you commit to the whole-controller route. The passthrough page has a "Passing through Keyboards and Mice" section further down, and it says XCP-ng ships /etc/xensource/usb-policy.conf with DENY rules for mice and keyboards by default. You edit those to ALLOW, then refresh with /opt/xensource/libexec/usb_scan.py -d followed by xe pusb-scan host-uuid=<host_uuid>. It's at https://docs.xcp-ng.org/compute/ under USB Passthrough. I have no idea whether that covers your USB-to-serial adapter, which is a different device class, and passing the whole controller may still be the cleaner setup for two discrete workstations anyway. Might be worth a mention to @Team-Hypervisor-Kernel on the display question, because that one still puzzles me.

I had a look at your screenshots. The Topology dropdown is greyed out with a tooltip saying Requires admin permissions, so this looks deliberate rather than broken: the field seems gated on being a full XO admin, not on being admin of your own resource set. I couldn't find an existing report asking for it to respect resource-set admin instead, so feedback.vates.tech is probably the right place to raise it. It would carry more weight coming from you, with those screenshots, than from me. I don't know whether the ACL rework changes any of this, so I wouldn't count on it until someone who works on it says so. Might be worth a mention to @Team-XO-Backend, since where that permission gate lives is really their call.

You may have worked it out yourself already. A consistency check reporting inconsistent parity on Virtual Disk 1, plus Buffer I/O error on several dm- devices, is the storage layer underneath XCP-ng telling you something is wrong down there. The VDI_IO_ERROR is mostly XCP-ng saying it could not read the disk, not the cause itself. I would be careful about anything that writes to that array until someone who knows hardware RAID recovery better than I do has looked at it. I honestly don't know whether a rebuild helps or makes things worse from this state, and I'd rather say that than guess with your data. Might be worth a mention to @Team-Storage.

I don't fully follow the mac-seed side of this, but a couple of things in the thread stand out. Tristis Oris's workaround looks like the practical unblock for now: removing the halted CR copy on the target host lets the migration go through, presumably because that replica VM is what collides on the mac-seed. Since you, KPS and Tristis Oris are all hitting the same DUPLICATE_MAC_SEED migrating into a replica target, this feels like something worth a GitHub issue on xen-orchestra with your XO commit, the exact steps, and whether a halted CR copy is present each time. It might also be worth a mention to @Team-XAPI-Network, since they'd know whether a CR replica is supposed to share its source's mac-seed. I could be wrong on the mechanism, so take that with a pinch of salt.

@poddingue Hello, sorry for the delayed response. We had some issues with the Supermicro IPMI and were waiting for a resolution. Here are the two log files: dmesg_xcp-ng-8320250606-aacraid-legacy.log dmesg_xcp-ng-8320250606-aacraid-uefi.log The alternative kernel did not work with your xcp-ng-8.3.0-20250606-aacraid.iso. I am also providing the log after installing the standard xcp-ng-8.3.0-20250606.iso: The files can be found at https://drive.google.com/drive/folders/1H-0FEruGK26oclD2WInDSOBJhVW2k8Xz?usp=sharing, as I wasn't able to upload the log files directly to the forum. I am also ready to participate in any further testing of new images, up to and including XCP-ng 9.0.

@jedimarcus solution work, thanks, We had server working 100% since last two weeks but error didn't disappear, after clean logs was ok thank you.

Hi @simonp. The problem is gone with commit 44e5b!!! Up-to-date XCP-NG host and XenOrchestra VM were restarted after updating XenOrchestra. I just initiated a full backup and all VMs - including my problematic machine - were backed up successfully without error. Awesome! I will wait with closing until the scheduled delta backup is completed successfully, too. Edit: Scheduled backup ran without issue. The problem seems to have been resolved! Thanks for all your work and assistance Regards, Marc

So if i set VMs to Best-effort or disabled for HA i get not enough memory. @pkgw Do you have Twinstore configured and/or HA enabled? If no I will continue my issues at the twinstore post. If i set to restart i get this.... host.setMaintenanceMode { "id": "35233210-4e37-4703-9bf6-9e8a9c24df9f", "maintenance": true } { "code": "HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN", "params": [], "call": { "duration": 5, "method": "host.disable", "params": [ "* session id *", "OpaqueRef:d9bd976f-445c-8244-1b56-879f446efc12" ] }, "message": "HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN()", "name": "XapiError", "stack": "XapiError: HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN() at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12) at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21 at runNextTicks (node:internal/process/task_queues:65:5) at processImmediate (node:internal/timers:472:9)" }

OK so that resource was a pain and did not want to give up, so I restarted the host. Migration is running good now.

Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates