@MajorP93 said: What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer. Yes, that's it - sorry - I should have been more explicit, even if 'lumbers along' wasn't a hint //EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool. Pedantic, but true I technically meant 'not hosts in the source pool' I guess.

That's great! Thanks for your feedback @laszlobortel

@rvreugde I believe this is already being reviewed internally.

@john.c my XO is compiled from sources... on future i will try config the v.6 to use the RBAC ACL for this case. @poddingue Thanks for your response, i will look this in a near future... My Cluster here is used only by the Tech/NOC team for tests/homologation purposes , beside that we are studying other virtualization systems because we will decrease our dependency on VMware on next year. Soon i will try change a few configurations and overview the v.6 to try fix this issue.

Also a question for @Team-XAPI-Network

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes. Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 8,700 log lines around the panic suggests dom0 was under real pressure right then. If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side. I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful. Fingers crossed it stays boring from here.

@michael.manley Would it be possible to backport the XSA-498 fixes? I've tried updating the SDK as a whole but it resulted in a lot of churn and caused errors elsewhere: https://github.com/xcp-ng/xenadmin/pull/271 The fixes to the SDK have been posted here https://github.com/xapi-project/xen-api/pull/7176 and here: https://xenbits.xen.org/xsa/advisory-498.html last-genius opened this pull request in xcp-ng/xenadmin draft Update the SDK to 26.16.0 #271 alexbrett opened this pull request in xapi-project/xen-api open [SDK] Corrected certificate validation for HTTP calls #7176

@poddingue Hi! Unfortunately, another error came up on the larger vms. The job also seems to be painfully slow. Slower than usual I mean. Also, it seems that job changes are not being commited. I've changed the job so it wouldn't retry in case of failure and the log shows it still trying 3 times (that was the old setting, and probably explains the job taking longer than usual). [image: 1784017759119-a917295e-ea92-4173-bc5a-7c9652a8d2ad-image.jpeg] { "id": "0mrjahr8e-ssawlbveq5", "start": 1783951215134, "status": "failure", "tasks": [ { "id": "0mrjahs3e-dbfaqajeze", "start": 1783951216250, "status": "failure", "tasks": [ { "id": "0mrjahs8s-2yb6i3fny6f", "start": 1783951216444, "status": "failure", "end": 1783959266484, "result": { "$fault": "server", "$metadata": { "httpStatusCode": 500, "requestId": "c0a9f30190109f99", "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP", "attempts": 3, "totalRetryDelay": 197 }, "name": "InternalError", "Code": "InternalError", "message": "internal incident", "stack": "InternalError: internal incident\n at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20" }, "message": "transfer", "data": { "progress": 40 } } ], "end": 1783959266485, "result": { "$fault": "server", "$metadata": { "httpStatusCode": 500, "requestId": "c0a9f30190109f99", "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP", "attempts": 3, "totalRetryDelay": 197 }, "name": "InternalError", "Code": "InternalError", "message": "internal incident", "stack": "InternalError: internal incident\n at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20" }, "message": "export", "data": { "id": "14b59f44-4517-411e-9e39-4eb7136928d0", "type": "remote", "isFull": true } }, { "id": "0mrjfabp3-c8mf2pnq8yf", "start": 1783959266487, "status": "success", "warnings": [ { "data": { "path": "xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/cache.json.gz", "actual": 0, "expected": 1 }, "message": "unexpected number of entries in backup cache" } ], "end": 1783959266989, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1783959266989, "result": { "$fault": "server", "$metadata": { "httpStatusCode": 500, "requestId": "c0a9f30190109f99", "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP", "attempts": 3, "totalRetryDelay": 197 }, "name": "InternalError", "Code": "InternalError", "message": "internal incident", "stack": "InternalError: internal incident\n at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20" }, "message": "backup VM", "data": { "id": "60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee", "type": "VM", "progress": 0 } },