Just got a failure today but with a different pool. "message": "backup task failed with undefined error", "name": "Error", "stack": "Error: backup task failed with undefined error\n at forwardResult (file:///usr/local/lib/node_modules/xo-server/src/_handleBackupLog.mjs:37:25)\n at handleBackupLog (file:///usr/local/lib/node_modules/xo-server/src/_handleBackupLog.mjs:68:12)\n at onTaskUpdate (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/metadata-backups.mjs:117:13)\n at onTaskUpdate (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/mixins/Tasks.mjs:205:23)\n at onLogFct (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/combineEvents.js:61:5)\n at metadataBackup._executor (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/metadata-backups.mjs:122:11)\n at Jobs.runJob (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n at file:///usr/local/lib/node_modules/xo-server/src/api/schedule.mjs:84:9\n at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:204:22)\n at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:188:20)\n at Xo.runSequence (file:///usr/local/lib/node_modules/xo-server/src/api/schedule.mjs:73:3)\n at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:204:22)\n at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:188:20)\n at Api.#callApiMethod (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/api.mjs:475:18)" } } [image: 1785806777819-screenshot_20260803_212536.png]

All hosts are restarted with 9.2.18 loaded now. Will see what happens next. [15:56 ovbh-pprod-xen05 ~]# drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36 DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION_CODE=0x090212 DRBD_KERNEL_VERSION=9.2.18 DRBDADM_VERSION_CODE=0x092100 DRBDADM_VERSION=9.33.0

@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output. [ 108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE [ 108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202 [ 108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset [ 108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2 [ 108.547789] amdgpu 0000:00:09.0: Failed to evict process queues [ 108.547803] amdgpu: Failed to quiesce KFD [ 108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source: 3 [ 109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0 [ 109.657324] amdgpu 0000:00:09.0: Dumping IP State [ 109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed [ 112.047695] amdgpu 0000:00:09.0: MODE1 reset [ 112.047797] amdgpu 0000:00:09.0: GPU mode1 reset [ 112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset [ 113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume [ 113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000). [ 113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created [ 113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data [ 113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset! [ 113.092921] amdgpu 0000:00:09.0: PSP is resuming... [ 114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed [ 115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available [ 115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available [ 115.091134] amdgpu 0000:00:09.0: SMU is resuming... [ 115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0) [ 115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully! [ 115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000 [ 115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000 [ 115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800 As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...

@msupport said: @acebmxer You need two Disk with 300GB If this is a reply to my initial post. I have already got it installed. Where do you see this 300GB requirement? https://helpcenter.veeam.com/docs/vbr/em/system_requirements.html?ver=13 Disk Selection Logic: When selecting a disk for system and adjacent functions, Veeam Software Appliance automatically chooses an SSD over an HDD, and selects the smaller disk of the two available. The sizing recommendations for Disk 1 below remain valid for the system disk. Disk 1: 240 GB1 minimum. This disk hosts Veeam JeOS, Enterprise Manager Veeam Backup & Replication software, configuration database and instant recovery cache. Recommended sizing depends on the number of protected workloads. Sizing also accounts for configuration database growth as you add more workloads to protect: 480 GB1 SSD for small environments (up to a few hundred workloads). 960 GB1 SSD for medium-sized environments (up to a few thousand workloads). Multi-TB1 SSD for large environments. Larger capacity increases the disk space available to instant recovery cache, allowing for running more machines for longer time. Note: After Veeam Software Appliance deployment, adding new storage devices or resizing existing ones is not supported. Plan your disk capacity carefully before you start the deployment. Disk 2: 240 GB1 minimum. This disk hosts guest file system catalogs and backups, therefore recommended sizing depends on your backup storage needs. Any additional disks found in the system during Veeam Software Appliance deployment will be automatically joined with Disk 2 into the single Logical Volume Manager (LVM) spanned volume. This is only backing a few vms around 10 vms, and actual location is to nfs storage.

@pierrebrunet said: Hi, just to be sure, are you saying it is working ok apart one task 8 days ago? Yes, after I adjusted the time it's working

@TeddyAstie From what I've tried...It won't work at all in Linux. It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least. I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs. Side tangent... I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere.

@acebmxer all good and thank you again for sharing your discovery on the forums

Good news that it sorted itself out. Both changes went in at once and the next run came back clean on the ADDS VMs too, so I don't think there's anything left to pin down here, and I'd rather say that than invent an explanation after the fact. Your question about the snapshot modes was the useful thing to come out of this thread. I went looking, they genuinely aren't documented anywhere, and that's written up on our side now If the fall back to a full ever comes back on the ADDS VMs specifically, that would deserve its own thread with the SMlog output @tjkreidl asked for, since nobody got to look at that part.

@jeremie1977 Ah, oui effectivement je n'ai pas testé avec plusieurs disques ! Et j'ai du typo dans la doc, je vais faire une PR pour modifier Je pense que c'est du a la façon d'insérer les disques et les DVD coté XCP-ng. Je vais investiguer !

@olivierlambert Happy to hear that they are providing results to be looked at. If there is anything else I can provide, please don't hesitate to ask. And I'll do my very best to be helpfull

Excellent! It's really cool to see how XO Backup and Replication can help you to migrate easily

So if i set VMs to Best-effort or disabled for HA i get not enough memory. @pkgw Do you have Twinstore configured and/or HA enabled? If no I will continue my issues at the twinstore post. If i set to restart i get this.... host.setMaintenanceMode { "id": "35233210-4e37-4703-9bf6-9e8a9c24df9f", "maintenance": true } { "code": "HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN", "params": [], "call": { "duration": 5, "method": "host.disable", "params": [ "* session id *", "OpaqueRef:d9bd976f-445c-8244-1b56-879f446efc12" ] }, "message": "HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN()", "name": "XapiError", "stack": "XapiError: HA_OPERATION_WOULD_BREAK_FAILOVER_PLAN() at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12) at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21 at runNextTicks (node:internal/process/task_queues:65:5) at processImmediate (node:internal/timers:472:9)" }

@pedro_udifar Hi, You can try again with minPartSize=100000000 indeed. But I would advise you to contact B2 directly. They should have a clearer error to show instead of a 500. Thanks for your patience and your help

@MajorP93 said: What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer. Yes, that's it - sorry - I should have been more explicit, even if 'lumbers along' wasn't a hint //EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool. Pedantic, but true I technically meant 'not hosts in the source pool' I guess.

Got this issue since 2026-07-12.

@acebmxer Thank you for your quick feedback.

Hello @anthoineb, We captured another occurrence on the same VM, os-ott-data-3-2 (172.30.52.193), on the same hypervisor and QCOW2 VDI as the incident reported on 2026-07-25. This occurrence is useful because xen_blkfront.max_ring_page_order=3 was definitely active: xenstore showed ring-page-order=3 with eight ring references; the guest queue depth was 256; 255 write requests remained in flight; all 256 blk-mq scheduler tags were busy; wbt_lat_usec=0; I/O PSI full was approximately 98%; completed xvdb counters did not advance between samples. The backend state was nevertheless the same as in the previous capture: tapdisk was sleeping in scheduler_wait_for_events(); tap-ctl reported reqs_outstanding=0; tap request counters were 0/0; req_prod=0 and rsp_prod=0; no tap, image, VBD, map or xenbus errors were reported. One detail may be relevant: GDB still printed n_reqs=32 and n_reqs_free=32, although the frontend had negotiated an order-3 ring and the guest exposed 256 tags. Is this expected because n_reqs represents a separate fixed tapdisk request pool, or could it indicate that tapdisk is not observing the complete multi-page ring? The controller captured GDB and the guest/hypervisor state before reboot. After the ten-minute diagnostic window, the same signature was confirmed three more times and the VM was rebooted. It rejoined the cluster successfully. I have attached: the complete GDB/tap-ctl capture; the guest blk-mq, diskstats, PSI, D-state stack and kernel-journal snapshot; the corresponding tapdisk/xenstore, daemon.log and SMlog excerpt; a README and SHA-256 checksums. The important new result appears to be that increasing max_ring_page_order from 0 to 3 did not prevent the stall; the same mismatch recurred with 255/256 frontend requests/tags occupied while the backend ring and tapdisk remained empty. 20260730-005220-gdb-172.30.52.193.txt 20260730-pre-reboot-guest-172.30.52.193.txt 20260730-pre-reboot-hypervisor-172.30.50.191.txt README.txt SHA256SUMS.txt

Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates

@yannsionneau , may this link helps if you not anyway know it https://medium.com/@timemaster5/getting-amd-phoenix-igpu-passthrough-working-on-xen-xcp-ng-8856593f9c0d I will give it a try this weekend