Topics created by Andrew

Hello @Andrew, thanks for your question! 😊

First off, I haven't tested Deno or Node with Xen-Orchestra yet. Changing the Node version can sometimes introduce bugs—some of which may be subtle. For that reason, I’d prefer not to venture into this unless it truly proves to be worth the effort.

Secondly, in my experience, benchmarks don’t always translate into meaningful results in complex applications.

That said, we are keeping an eye on alternative engines, both in terms of performance and the new features they offer. If we determine that the benefits are significant, we’d definitely consider migrating.

Additionally, if any users want to experiment with Deno or Bun, they’re welcome to start dedicated threads on the forum to collaborate. We’d be happy to review and merge compatibility PRs as long as they don’t break Node and aren’t overly complex. 😊

Worth creating a GH XO issue with most details as possible 🙂

@stormi I'm using 8.2.1 (updated July 2024).

If you read Debian's docs about UEFI boot it explains why they do something different than everyone else and how to fix it so it works. I use UEFI boot for Debian and use the removable media path install it boots every time on XCP.

I think there are several issues, each can cause the same failure to boot the VM.

If you upgrade XOA to UEFI with Debian then you need to install the extra boot files so it always boots on XCP.

Got it. We might ship that in a patch release if it's bundled in our tomorrow's latest XOA release.

@stormi Thanks @stormi.

I have to get used to XO. I always use XCP-ng Admin on Windows because old habits die hard, and there I cannot even select UEFI with all the Linux templates.

The magic that XO does is that it allows to use the templates, but with UEFI.

I have a VM that I've deployed with the "Other install media" option, so I just dettached the disk and reimported on the new one created by XO, with the correct template. Everything seems fine.

So my question about inheriting the template is not relevant if we use XO. Hope my understand is correct.

Thanks.

@florent I see it has been merged into master. Yes, transfer size seems to be back to before. I have not checked for damage. At worst I can just force a full backup.

We see occasional VM backup failures that match the above pattern (ENOENT on a "path" in a S3 bucket remote destination during a VM delta backup). We are running a backup of a large VM (200GB disk) to S3 bucket in Ceph Object Gateway. Our current XO is this commit https://github.com/vatesfr/xen-orchestra/commit/da51b1649c65d7d78a4eb25cc46c488ce2552800 from 2024-03-04.

The failure is intermittent.

Example failure log:

{ "data": { "mode": "delta", "reportWhen": "always" }, "id": "1712008868496", "jobId": "49c2634a-f281-46b7-9192-aea8c786f3d4", "jobName": "ZABBIX VM to ceph02 Delta Backup", "message": "backup", "scheduleId": "6b1280f0-21e7-475c-9af3-330726108568", "start": 1712008868496, "status": "failure", "infos": [ { "data": { "vms": [ "c020890b-c1a7-79cf-0b55-aa340ba9226b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c020890b-c1a7-79cf-0b55-aa340ba9226b", "name_label": "zabbix" }, "id": "1712008870266", "message": "backup VM", "start": 1712008870266, "status": "failure", "tasks": [ { "id": "1712008870283", "message": "clean-vm", "start": 1712008870283, "status": "success", "end": 1712008872364, "result": { "merge": false } }, { "id": "1712008872791", "message": "snapshot", "start": 1712008872791, "status": "success", "end": 1712008873901, "result": "902d65f0-8c9b-2460-dad7-d041ceefebdc" }, { "data": { "id": "d8d6a743-8c15-4b9a-baf4-df5d7d379b6e", "isFull": false, "type": "remote" }, "id": "1712008873919", "message": "export", "start": 1712008873919, "status": "failure", "tasks": [ { "id": "1712008875215", "message": "transfer", "start": 1712008875215, "status": "success", "end": 1712009333076, "result": { "size": 35343973376 } }, { "id": "1712009335152", "message": "clean-vm", "start": 1712009335152, "status": "failure", "tasks": [ { "id": "1712009336153", "message": "merge", "start": 1712009336153, "status": "failure", "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585031 } ], "end": 1712009585032 } ], "end": 1712009585032 }

I know I should be on the current XO to report a problem. Not asking for a fix now, just trying to document that this defect still exists.

0 julien-f committed to vatesfr/xen-orchestra fix(xo-server/robots.txt): must not require authentication Introduced by a2c36c083 This file is intended for search engines, which, by definition are not signed in.

Thanks everyone for your feedback!

@florent Updated XO source to latest master and it's all green again.

Commit 3f316fcaea11775fc1393737cc6e09d8138da62d fixed the issue for me.

Thanks!

@julien-f Thanks. I updated to 0444cf0 and CR times are normal again (as expected).

@julien-f I just wanted to update and let you know that whatever changes you made fixed the issue for me permanently! I have been able to routinely update since that time and have had no further issues.

Thanks for all of your hard work, Julien-f!

@julien-f The new update seems to be working correctly now. Thanks for the quick update fix.

The backup cache fixes/cleanup made a huge difference (improvement). I was trying to figure out what was going on with my backups as they were running but were missing from the restore list.

After the cache cleanup, it showed 3-4 times the number of restore points! All the missing stuff and more. It took some time to clean up all the extra stuff, but backups seem to be back on track again.

@Darkbeldin I totally missed that.... I know I have used it before. It must be a long week already. Thanks.

@Andrew I didn't say "Xen" I said XenServer.

Fixed on master (commit d8721). Thanks @florent !

@florent No, I did not see that in the logs. I did see this problem is bigger than I thought.

It happens more often than just causing a VM backup failure. It happens during the merge or other checks which causes the backup process to destroy (remove) parts of other VM backups.

Clean VM directory parent VHD is missing parent VHD is missing parent VHD is missing some VHDs linked to the backup are missing some VHDs linked to the backup are missing some VHDs linked to the backup are missing some VHDs linked to the backup are missing

and

Clean VM directory VHD check error some VHDs linked to the backup are missing

@florent Limiting concurrency did not fix my S3 backup problem, but it's working again after updating the build. So I guess it's resolved.

@Andrew Hi Andrew,

Metadata backup use the report plugin settings to send mail so you can define the email address there.