Hi @Danp , I've updated XO to the latest commit and the error is still happening.

@pierrebrunet said: Another thing to check is the exact time between your XO and your host. I installed the host with NTP from "static" DHCP (OK according to https://docs.xcp-ng.org/installation/requirements/) not made any manual changes/configs After I did a RTC adjustment in BIOS 3 days ago, the time now is ~1s ahead This is the output from timedatectl [22:04 x2 ~]# timedatectl status Local time: tor 2026-07-16 22:06:45 CEST Universal time: tor 2026-07-16 20:06:45 UTC RTC time: tor 2026-07-16 20:06:42 Time zone: Europe/Stockholm (CEST, +0200) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: yes Last DST change: DST began at sön 2026-03-29 01:59:59 CET sön 2026-03-29 03:00:00 CEST Next DST change: DST ends (the clock jumps one hour backwards) at sön 2026-10-25 02:59:59 CEST sön 2026-10-25 02:00:00 CET NTP synchronized: no I thought that the time would be adjusted automatically at reboot The first NTP sync should allow a bigger step than when it's already synced. @pierrebrunet said: do you make these backup regularly? They run once a day @ 03:04 before the normal backup that starts @ 03:10 and they have all run OK since July 8, the failed one reported 8 days ago ( post ~85 ) Maybe you should resync your clock NTP is enabled but not synced, I don't know the correct way to sync it In xsconsole/ I tried Use DHCP NTP Servers It changed the RTC clock to the same as UTC time but time was still 1s ahead and NTP not synced. I rebooted the host and I got this: [23:36 x2 ~]# timedatectl status Local time: tor 2026-07-16 23:37:17 CEST Universal time: tor 2026-07-16 21:37:17 UTC RTC time: tor 2026-07-16 21:37:16 Time zone: Europe/Stockholm (CEST, +0200) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: yes Last DST change: DST began at sön 2026-03-29 01:59:59 CET sön 2026-03-29 03:00:00 CEST Next DST change: DST ends (the clock jumps one hour backwards) at sön 2026-10-25 02:59:59 CEST sön 2026-10-25 02:00:00 CET NTP still not synced, Local time 2s ahead, RTC 1s ahead, I don't know, If not someone can show me the proper way to manually resync, maybe I should try to reinstall the host

@pierrebrunet Hello! I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens? I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ? REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -

@simonp , thank you very much and all of your team ! ! ! Keep up the outstanding work. Regards, Marc

Hi @acebmxer, thanks for reporting, we identified this bug internally already, we're going to fix it once root cause is found.

Hi Everyone, If I may advertise what I did you have a dedicated thread about my project, this is an easy to setup XCP-ng and XO from the source solution. Sorry sharing the link to the topic as is couldn't find better way to do that : XCP-ng HL — an ISO for home labs, deploy any XOA image from XO Lite directly Essentially you deploy XCP-ng ISO and then in XO Lite deploy XOA section you can deploy your prebuilt or XOA VM directly. Hope that helps Home labers to get in to XCP-ng easily. Same here this is still work in progress, also I'm personally already using this version in my lab. Any feedback is welcome ! With regards.

@acebmxer I can confirm that, we'll fix it, thanks

Just FYI, we had some issues with some Debian VMs for instance, with old kernel where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive. It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore. Newer kernel don't have this issue and are able to wake up. I'm quoting Debian here but it might very well happen on other distro

Ping @Team-Storage

Hi @dvdwx, RBAC is available only at the REST API level, not through the XO6 UI. UI compatibility with RBAC is currently under development. For the topology, this is something that has been here for a long time, so I may be wrong since I don't have the full history. However, if I look at the code, creating the topology selector requires knowing how many CPUs are available in the pool. Since a self-service user doesn't have visibility into the pool, they can't access this kind of information.

Also a question for @Team-XAPI-Network

@MajorP93 said: What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer. Yes, that's it - sorry - I should have been more explicit, even if 'lumbers along' wasn't a hint //EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool. Pedantic, but true I technically meant 'not hosts in the source pool' I guess.

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes. Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 8,700 log lines around the panic suggests dom0 was under real pressure right then. If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side. I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful. Fingers crossed it stays boring from here.

@michael.manley Would it be possible to backport the XSA-498 fixes? I've tried updating the SDK as a whole but it resulted in a lot of churn and caused errors elsewhere: https://github.com/xcp-ng/xenadmin/pull/271 The fixes to the SDK have been posted here https://github.com/xapi-project/xen-api/pull/7176 and here: https://xenbits.xen.org/xsa/advisory-498.html

@rvreugde I believe this is already being reviewed internally.

@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output. [ 108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE [ 108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202 [ 108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset [ 108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2 [ 108.547789] amdgpu 0000:00:09.0: Failed to evict process queues [ 108.547803] amdgpu: Failed to quiesce KFD [ 108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source: 3 [ 109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0 [ 109.657324] amdgpu 0000:00:09.0: Dumping IP State [ 109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed [ 112.047695] amdgpu 0000:00:09.0: MODE1 reset [ 112.047797] amdgpu 0000:00:09.0: GPU mode1 reset [ 112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset [ 113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume [ 113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000). [ 113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created [ 113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data [ 113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset! [ 113.092921] amdgpu 0000:00:09.0: PSP is resuming... [ 114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed [ 115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available [ 115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available [ 115.091134] amdgpu 0000:00:09.0: SMU is resuming... [ 115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0) [ 115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully! [ 115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000 [ 115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000 [ 115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800 As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...

@christopher-petzel Thanks!