@TrapoSAMA said: @john.c I have this iso but how to know what is de package? Thx It’s going to be an msi installer package or on the guest tools iso, which is on the XCP-ng or Xen Server your running likely in Local Storage SR or Shared Storage SR (if moved). Named the same as the file linked to above, likely. Anyway did you know that the UEFI SecureBoot certificates (at least one of them from 2011) issued by Microsoft are expiring fully during October 2026. Around that time likely at sometime after then those certificates will be placed into dbx. This means the Windows Server 2012 R2 if operating as a UEFI SecureBoot VM, will if restarted (or reboot following a crash), fail to boot. So your client will go from having a working app that they can use, but not upgrade at the moment to one which won’t work and can’t be used! They really need to upgrade or migrate from Windows to Linux (released during 2025 or 2026) along with the app if needed. Alternatively upgrade to Windows Server 2022 or 2025 and the app!

@rvreugde I believe this is already being reviewed internally.

@christopher-petzel Thanks!

@michael.manley Would it be possible to backport the XSA-498 fixes? I've tried updating the SDK as a whole but it resulted in a lot of churn and caused errors elsewhere: https://github.com/xcp-ng/xenadmin/pull/271 The fixes to the SDK have been posted here https://github.com/xapi-project/xen-api/pull/7176 and here: https://xenbits.xen.org/xsa/advisory-498.html last-genius opened this pull request in xcp-ng/xenadmin draft Update the SDK to 26.16.0 #271 alexbrett opened this pull request in xapi-project/xen-api open [SDK] Corrected certificate validation for HTTP calls #7176

I have some good news for you: our Rocky8 VMs have been updated to 4.18.0-553.134.1.el8_10.x86_64 (and some even to the latest 4.18.0-553.141.1.el8_10.x86_64) kernel on 24th June (3 weeks ago). Since then the problem disappeared completely and today we continued our V2V migration from VMware to XCP-ng. Thanks for all your support!

@poddingue Hi! Unfortunately, another error came up on the larger vms. The job also seems to be painfully slow. Slower than usual I mean. Also, it seems that job changes are not being commited. I've changed the job so it wouldn't retry in case of failure and the log shows it still trying 3 times (that was the old setting, and probably explains the job taking longer than usual). [image: 1784017759119-a917295e-ea92-4173-bc5a-7c9652a8d2ad-image.jpeg] { "id": "0mrjahr8e-ssawlbveq5", "start": 1783951215134, "status": "failure", "tasks": [ { "id": "0mrjahs3e-dbfaqajeze", "start": 1783951216250, "status": "failure", "tasks": [ { "id": "0mrjahs8s-2yb6i3fny6f", "start": 1783951216444, "status": "failure", "end": 1783959266484, "result": { "$fault": "server", "$metadata": { "httpStatusCode": 500, "requestId": "c0a9f30190109f99", "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP", "attempts": 3, "totalRetryDelay": 197 }, "name": "InternalError", "Code": "InternalError", "message": "internal incident", "stack": "InternalError: internal incident\n at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20" }, "message": "transfer", "data": { "progress": 40 } } ], "end": 1783959266485, "result": { "$fault": "server", "$metadata": { "httpStatusCode": 500, "requestId": "c0a9f30190109f99", "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP", "attempts": 3, "totalRetryDelay": 197 }, "name": "InternalError", "Code": "InternalError", "message": "internal incident", "stack": "InternalError: internal incident\n at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20" }, "message": "export", "data": { "id": "14b59f44-4517-411e-9e39-4eb7136928d0", "type": "remote", "isFull": true } }, { "id": "0mrjfabp3-c8mf2pnq8yf", "start": 1783959266487, "status": "success", "warnings": [ { "data": { "path": "xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/cache.json.gz", "actual": 0, "expected": 1 }, "message": "unexpected number of entries in backup cache" } ], "end": 1783959266989, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1783959266989, "result": { "$fault": "server", "$metadata": { "httpStatusCode": 500, "requestId": "c0a9f30190109f99", "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP", "attempts": 3, "totalRetryDelay": 197 }, "name": "InternalError", "Code": "InternalError", "message": "internal incident", "stack": "InternalError: internal incident\n at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20" }, "message": "backup VM", "data": { "id": "60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee", "type": "VM", "progress": 0 } },

@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output. [ 108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE [ 108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202 [ 108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset [ 108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2 [ 108.547789] amdgpu 0000:00:09.0: Failed to evict process queues [ 108.547803] amdgpu: Failed to quiesce KFD [ 108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source: 3 [ 109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0 [ 109.657324] amdgpu 0000:00:09.0: Dumping IP State [ 109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed [ 112.047695] amdgpu 0000:00:09.0: MODE1 reset [ 112.047797] amdgpu 0000:00:09.0: GPU mode1 reset [ 112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset [ 113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume [ 113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000). [ 113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created [ 113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data [ 113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset! [ 113.092921] amdgpu 0000:00:09.0: PSP is resuming... [ 114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed [ 115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available [ 115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available [ 115.091134] amdgpu 0000:00:09.0: SMU is resuming... [ 115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0) [ 115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully! [ 115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000 [ 115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000 [ 115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800 As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...

@poddingue These are VMs that have been doing full backups fine for a very long time. I have run into the too-much-free-space issue in the past with a different brand new VM but these have been around since I migrated them from VMware during the Broadcom fiasco years ago. It is also totally inconsistent which VMs fail with the body timeout. One day it'll be a single one, the next day it'll be three or four and won't include that one, then it'll be a couple other ones unrelated to any of the earlier ones, then I'll have no failures. It's all over the place. I'm just glad that over the course of several days I get good backups of everything. Oh, a maybe useful data point is that my delta backups that all succeed are handled by a different instance of XO that runs on a different one of the hosts. I might be overly paranoid, but I run full backups and delta backups of the same VMs using different XO instances to different target remotes, at different non-overlapping times. This is part of the reason that I'm not hair-on-fire worried about the failing full backups. A while back I stupidly let one of my Storage Repositories run out of space and it was the one hosting my main XO instance that handled all the backups. It took a while to get it back going again and I realized I wanted redundancy in what handled backups and moved the deltas over to another XO instance. Too many eggs in that one basket. I also wasn't able to restore the backup of that XO instance because that XO instance is what handled the backups. Thankfully it all worked out in the end, plus a number of extra gray hairs it gave me.

@poddingue I should already have all the logs, including kernel[image: 1783960418266-2ce97f35-954d-4f1b-88a5-ae0899e75d99-image.jpeg] Nothing has happened since. I am also increasing dom0 ram for all hosts.