Hi @Andrew. This PR should fix your issue https://github.com/vatesfr/xen-orchestra/pull/8854 MathieuRA opened this pull request in vatesfr/xen-orchestra closed fix(xo-server): fix incorrect state when remove server #8854

@stormi Unless new Dom0 is Kernel 6.15 or newer (with updates), this will be an ongoing problem to support the newest hardware as they seem to keep updating the chips (you can't stop progress). The current r8125 driver in 8.2/8.3 supports the 8125 and some 8126 chips, but not 8127. Other than supporting the new hardware versions of the chips, I have not seen any real complaints posted about the current r8125 driver (may be there are some Vates tickets?) While the risk on the new driver is low, it is different (not just an update). I agree that it would be best to have r8125-module-alt for the new code and then r8126-module and r8127-module. Note, that the new r8126-module driver should not be installed without the r8125-module-alt as the existing r8125-module would conflict with the new r8126-module driver. The work is already done (compiled and running)... I'm just waiting for Realtek to fix some issues with the new code before it's worth fully testing the drivers (issues already acknowledged by Realtek).

@MathieuRA For this issue, fix-server-connect works for me. Thanks.

I will create a new thread from your message

I mean I agree with the idea that sounds good, I need to be reviewed again to see if we could do better than we did

Hello @Andrew, thanks for your question! First off, I haven't tested Deno or Node with Xen-Orchestra yet. Changing the Node version can sometimes introduce bugs—some of which may be subtle. For that reason, I’d prefer not to venture into this unless it truly proves to be worth the effort. Secondly, in my experience, benchmarks don’t always translate into meaningful results in complex applications. That said, we are keeping an eye on alternative engines, both in terms of performance and the new features they offer. If we determine that the benefits are significant, we’d definitely consider migrating. Additionally, if any users want to experiment with Deno or Bun, they’re welcome to start dedicated threads on the forum to collaborate. We’d be happy to review and merge compatibility PRs as long as they don’t break Node and aren’t overly complex.

Worth creating a GH XO issue with most details as possible

We are implementing tests in our CI to detect this issue and protect against regressions, then we'll investigate the cause.

Got it. We might ship that in a patch release if it's bundled in our tomorrow's latest XOA release.

@stormi Thanks @stormi. I have to get used to XO. I always use XCP-ng Admin on Windows because old habits die hard, and there I cannot even select UEFI with all the Linux templates. The magic that XO does is that it allows to use the templates, but with UEFI. I have a VM that I've deployed with the "Other install media" option, so I just dettached the disk and reimported on the new one created by XO, with the correct template. Everything seems fine. So my question about inheriting the template is not relevant if we use XO. Hope my understand is correct. Thanks.

@florent I see it has been merged into master. Yes, transfer size seems to be back to before. I have not checked for damage. At worst I can just force a full backup.

We see occasional VM backup failures that match the above pattern (ENOENT on a "path" in a S3 bucket remote destination during a VM delta backup). We are running a backup of a large VM (200GB disk) to S3 bucket in Ceph Object Gateway. Our current XO is this commit https://github.com/vatesfr/xen-orchestra/commit/da51b1649c65d7d78a4eb25cc46c488ce2552800 from 2024-03-04. The failure is intermittent. Example failure log: { "data": { "mode": "delta", "reportWhen": "always" }, "id": "1712008868496", "jobId": "49c2634a-f281-46b7-9192-aea8c786f3d4", "jobName": "ZABBIX VM to ceph02 Delta Backup", "message": "backup", "scheduleId": "6b1280f0-21e7-475c-9af3-330726108568", "start": 1712008868496, "status": "failure", "infos": [ { "data": { "vms": [ "c020890b-c1a7-79cf-0b55-aa340ba9226b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c020890b-c1a7-79cf-0b55-aa340ba9226b", "name_label": "zabbix" }, "id": "1712008870266", "message": "backup VM", "start": 1712008870266, "status": "failure", "tasks": [ { "id": "1712008870283", "message": "clean-vm", "start": 1712008870283, "status": "success", "end": 1712008872364, "result": { "merge": false } }, { "id": "1712008872791", "message": "snapshot", "start": 1712008872791, "status": "success", "end": 1712008873901, "result": "902d65f0-8c9b-2460-dad7-d041ceefebdc" }, { "data": { "id": "d8d6a743-8c15-4b9a-baf4-df5d7d379b6e", "isFull": false, "type": "remote" }, "id": "1712008873919", "message": "export", "start": 1712008873919, "status": "failure", "tasks": [ { "id": "1712008875215", "message": "transfer", "start": 1712008875215, "status": "success", "end": 1712009333076, "result": { "size": 35343973376 } }, { "id": "1712009335152", "message": "clean-vm", "start": 1712009335152, "status": "failure", "tasks": [ { "id": "1712009336153", "message": "merge", "start": 1712009336153, "status": "failure", "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585031 } ], "end": 1712009585032 } ], "end": 1712009585032 } I know I should be on the current XO to report a problem. Not asking for a fix now, just trying to document that this defect still exists. 0 julien-f committed to vatesfr/xen-orchestra fix(xo-server/robots.txt): must not require authentication Introduced by a2c36c083 This file is intended for search engines, which, by definition are not signed in.

Thanks everyone for your feedback!

@florent Updated XO source to latest master and it's all green again. Commit 3f316fcaea11775fc1393737cc6e09d8138da62d fixed the issue for me. Thanks!

@julien-f Thanks. I updated to 0444cf0 and CR times are normal again (as expected).

@julien-f I just wanted to update and let you know that whatever changes you made fixed the issue for me permanently! I have been able to routinely update since that time and have had no further issues. Thanks for all of your hard work, Julien-f!

@julien-f The new update seems to be working correctly now. Thanks for the quick update fix. The backup cache fixes/cleanup made a huge difference (improvement). I was trying to figure out what was going on with my backups as they were running but were missing from the restore list. After the cache cleanup, it showed 3-4 times the number of restore points! All the missing stuff and more. It took some time to clean up all the extra stuff, but backups seem to be back on track again.

@Darkbeldin I totally missed that.... I know I have used it before. It must be a long week already. Thanks.

@Andrew I didn't say "Xen" I said XenServer.