Perfect, thanks for your feedback!

@tmk I was able to reproduce the problem, so it seems the modifications we brought a couple of weeks ago didn't fix it. I'm working on a fix at the moment.

Hello! So I created a new job and a small test vm to gather info. I did 3 replications runs and the last 2 failed. Backups are working thoug. Since this would be a large post, I created a pdf with the test cycle. ... Only to find out that those are not allowed on this forum! So I captured the pdf pages... [image: 1784297275075-535e6b06-678a-4228-b109-e6fad5ec97e5-image.jpeg] [image: 1784297294513-9c5c5933-3dc8-4223-983e-7a9c3d884a72-image.jpeg] [image: 1784297307874-e199e8fa-4335-412a-924b-67f482b8bdd2-image.jpeg] [image: 1784297320970-7e8a9103-1a5f-4b7c-b886-2c9364cdd3af-image.jpeg] [image: 1784297331437-18fedc6b-3f83-4165-a6e3-f1586b97c127-image.jpeg] [image: 1784297343300-7c917ab7-31f7-441a-bca8-397fd8653a29-image.jpeg] [image: 1784297353287-921b2af6-5276-4272-818b-6f7a025d6c59-image.jpeg] And the logs from the jobs: { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml" }, "id": "1784293017099", "jobId": "10d0331c-694e-4681-8860-33fef3790552", "jobName": "TESTCRANDBK", "message": "backup", "scheduleId": "bc7b29ce-25b2-4c66-9186-b1d4c314b979", "start": 1784293017099, "status": "success", "tasks": [ { "id": "0mroxztk5-9yuipq34src", "start": 1784293020005, "status": "success", "tasks": [ { "id": "0mroxztkj-izqz7v0ufn", "start": 1784293020019, "status": "success", "end": 1784293020027, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mroxztz1-v8ycdmtgo9k", "start": 1784293020541, "status": "success", "end": 1784293036537, "result": "20683621-7819-7661-91a2-93c377a7345a", "message": "snapshot" }, { "id": "0mroy06bf-9frdqsgnpym", "start": 1784293036539, "status": "success", "tasks": [ { "id": "0mroy0aun-bscsuoyzrpu", "start": 1784293042415, "status": "success", "tasks": [ { "id": "0mroy3r14-2sk7dgyqrbl", "start": 1784293203352, "status": "success", "end": 1784293207572, "message": "target snapshot" } ], "end": 1784293207573, "result": { "size": 9254731776 }, "message": "transfer" } ], "end": 1784293207664, "message": "export", "data": { "id": "a729e916-4cd8-2208-7dce-329e6b4af892", "isFull": true, "name_label": "xxxxx", "type": "SR" } }, { "id": "0mroy06bg-eyd2s4tidgg", "start": 1784293036540, "status": "success", "end": 1784293207664, "message": "export", "data": { "id": "b0b77301-4625-a13b-7022-9512f927944a", "isFull": true, "name_label": "xxxxxx", "type": "SR" } }, { "id": "0mroy06bh-ra3dxpusxr8", "start": 1784293036541, "status": "success", "end": 1784293207664, "message": "export", "data": { "id": "7054c407-d620-2c9f-5810-1e192a04ffe2", "isFull": true, "name_label": "xxxxxxxxx", "type": "SR" } }, { "id": "0mroy06bi-tdyzypsg58", "start": 1784293036542, "status": "success", "end": 1784293207664, "message": "export", "data": { "id": "621c9b87-4f58-987b-2c07-e8dee67da73a", "isFull": true, "name_label": "xxxxxxxxxx", "type": "SR" } }, { "id": "0mroy06be-u3p9nky3uwd", "start": 1784293036538, "status": "success", "tasks": [ { "id": "0mroy0aus-ax44kkpim1", "start": 1784293042420, "status": "success", "end": 1784293200916, "result": { "size": 9257013760 }, "message": "transfer" }, { "id": "0mroy434q-md07n6vujw", "start": 1784293219034, "status": "success", "end": 1784293219099, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1784293219102, "message": "export", "data": { "id": "c1fdab1d-5535-42e4-b282-1a0282df1081", "isFull": true, "type": "remote" } } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:080d2018-c2a3-aa6f-89a1-b3eaf72c3de5" }, "message": "Snapshot data has been deleted" } ], "end": 1784293219102, "message": "backup VM", "data": { "id": "d771731a-f132-7365-be50-71f19424f96e", "type": "VM", "name_label": "TESTREPL", "progress": 0 } } ], "end": 1784293219104, "infos": [ { "data": { "vms": [ "d771731a-f132-7365-be50-71f19424f96e" ] }, "message": "vms" } ] } { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml" }, "id": "1784293990343", "jobId": "10d0331c-694e-4681-8860-33fef3790552", "jobName": "TESTCRANDBK", "message": "backup", "scheduleId": "bc7b29ce-25b2-4c66-9186-b1d4c314b979", "start": 1784293990343, "status": "success", "tasks": [ { "id": "0mroykonr-nktgebhtb4k", "start": 1784293993431, "status": "success", "tasks": [ { "id": "0mroykoo5-171n48gwq3r", "start": 1784293993445, "status": "success", "end": 1784293993493, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mroykp31-4r5wrf9blru", "start": 1784293993981, "status": "success", "end": 1784294007573, "result": "9ff54ec5-0247-4692-945b-c183206ae6dd", "message": "snapshot" }, { "id": "0mroykzkm-ykzk53um8l", "start": 1784294007574, "status": "success", "tasks": [ { "id": "0mroyl3qn-vzompxsw79o", "start": 1784294012975, "status": "success", "end": 1784294018591, "result": { "size": 396481024 }, "message": "transfer" }, { "id": "0mroyljgo-nlch68rkbil", "start": 1784294033352, "status": "success", "end": 1784294033444, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1784294033446, "message": "export", "data": { "id": "c1fdab1d-5535-42e4-b282-1a0282df1081", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "the writer AggregatedIncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'get'). It won't be used anymore in this job execution." } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:77605638-5e58-4a10-c2de-bc6e77f1ad61" }, "message": "Snapshot data has been deleted" } ], "end": 1784294033447, "message": "backup VM", "data": { "id": "d771731a-f132-7365-be50-71f19424f96e", "type": "VM", "name_label": "TESTREPL", "progress": 0 } } ], "end": 1784294033449, "infos": [ { "data": { "vms": [ "d771731a-f132-7365-be50-71f19424f96e" ] }, "message": "vms" } ] } { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml" }, "id": "1784295416670", "jobId": "10d0331c-694e-4681-8860-33fef3790552", "jobName": "TESTCRANDBK", "message": "backup", "scheduleId": "bc7b29ce-25b2-4c66-9186-b1d4c314b979", "start": 1784295416670, "status": "success", "tasks": [ { "id": "0mrozf8yr-lix8jxbm6pd", "start": 1784295419428, "status": "success", "tasks": [ { "id": "0mrozf8zg-zddbxm901mc", "start": 1784295419452, "status": "success", "end": 1784295419537, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mrozf9dv-7nj082i92kp", "start": 1784295419971, "status": "success", "end": 1784295426890, "result": "18312e82-adb6-2dc5-b13e-2fd9a4b2bf77", "message": "snapshot" }, { "id": "0mrozfeq3-w3y5xrwtqse", "start": 1784295426891, "status": "success", "tasks": [ { "id": "0mrozfrvy-4g1gefr14hx", "start": 1784295443950, "status": "success", "end": 1784295445177, "result": { "size": 92319744 }, "message": "transfer" }, { "id": "0mrozfw5f-mx0xt5f7wjm", "start": 1784295449475, "status": "success", "end": 1784295449563, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1784295449565, "message": "export", "data": { "id": "c1fdab1d-5535-42e4-b282-1a0282df1081", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "the writer AggregatedIncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'get'). It won't be used anymore in this job execution." } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:6a833d28-2d2c-5fe6-858b-02d2fd41b919" }, "message": "Snapshot data has been deleted" } ], "end": 1784295449566, "message": "backup VM", "data": { "id": "d771731a-f132-7365-be50-71f19424f96e", "type": "VM", "name_label": "TESTREPL", "progress": 0 } } ], "end": 1784295449567, "infos": [ { "data": { "vms": [ "d771731a-f132-7365-be50-71f19424f96e" ] }, "message": "vms" } ] } Hope it helps

@pierrebrunet said: Hi, just to be sure, are you saying it is working ok apart one task 8 days ago? Yes, after I adjusted the time it's working

@pierrebrunet Hello! I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens? I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ? REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -

@simonp , thank you very much and all of your team ! ! ! Keep up the outstanding work. Regards, Marc

Hi @acebmxer, thanks for reporting, we identified this bug internally already, we're going to fix it once root cause is found.

Hi Everyone, If I may advertise what I did you have a dedicated thread about my project, this is an easy to setup XCP-ng and XO from the source solution. Sorry sharing the link to the topic as is couldn't find better way to do that : XCP-ng HL — an ISO for home labs, deploy any XOA image from XO Lite directly Essentially you deploy XCP-ng ISO and then in XO Lite deploy XOA section you can deploy your prebuilt or XOA VM directly. Hope that helps Home labers to get in to XCP-ng easily. Same here this is still work in progress, also I'm personally already using this version in my lab. Any feedback is welcome ! With regards.

@acebmxer I can confirm that, we'll fix it, thanks

Just FYI, we had some issues with some Debian VMs for instance, with old kernel where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive. It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore. Newer kernel don't have this issue and are able to wake up. I'm quoting Debian here but it might very well happen on other distro

Ping @Team-Storage

Hi @dvdwx, RBAC is available only at the REST API level, not through the XO6 UI. UI compatibility with RBAC is currently under development. For the topology, this is something that has been here for a long time, so I may be wrong since I don't have the full history. However, if I look at the code, creating the topology selector requires knowing how many CPUs are available in the pool. Since a self-service user doesn't have visibility into the pool, they can't access this kind of information.

Also a question for @Team-XAPI-Network

@MajorP93 said: What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer. Yes, that's it - sorry - I should have been more explicit, even if 'lumbers along' wasn't a hint //EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool. Pedantic, but true I technically meant 'not hosts in the source pool' I guess.

That it's stayed quiet since you throttled Velero is a good sign, and it lines up with the concurrency theory rather than anything about the specific volumes. Bumping dom0 RAM makes sense too, since linstor-satellite spiking to roughly 8,700 log lines around the panic suggests dom0 was under real pressure right then. If it does come back, a xen-bugtool --yestoall bundle from the host that panicked, plus the Loki window you already have, would give @Team-Storage something concrete to line up against the DRBD side. I'm not sure whether the real fix sits in Velero's pacing or in how LINSTOR handles concurrent create/delete, but the reproduction you've narrowed down is genuinely useful. Fingers crossed it stays boring from here.

@michael.manley Would it be possible to backport the XSA-498 fixes? I've tried updating the SDK as a whole but it resulted in a lot of churn and caused errors elsewhere: https://github.com/xcp-ng/xenadmin/pull/271 The fixes to the SDK have been posted here https://github.com/xapi-project/xen-api/pull/7176 and here: https://xenbits.xen.org/xsa/advisory-498.html

@rvreugde I believe this is already being reviewed internally.