@julien-f 0k, is there a specific rule on the format of this identifier? When I create a backup, I set an email address for reporting but that address or notification setting) is not present in the JSON, why and how can I set it ? For rolling snapshots, the content is 0k (same identifier in both settings and schedule): { "name": "rolling snapshots", "mode": "full", "schedules": { "gbkk1mbemhg": { "cron": "15 12 * * *", "enabled": true, "name": "daily rolling snapshots", "timezone": "Europe/Rome" } }, "settings": { "": { "concurrency": 2, "nRetriesVmBackupFailures": 3, "timeout": 7200000, "offlineSnapshot": false, "checkpointSnapshot": true }, "gbkk1mbemhg": { "healthCheckSr": "ac595bf5-ca14-586e-6f89-9e4e884043eb", "healthCheckVmsWithTags": [], "snapshotRetention": 2 } }, "vms": { "id": { "__or": [ "64adb2e9-e0f3-9e70-b08f-7c8653415053", "aaf9e807-3ee9-37ff-05ec-cf782ba74d56", "7cd276ae-b7df-06d8-3da8-48262750051d", "9ccfcd0c-a35e-7f45-a0d2-19db678291f1", "42afaaea-ada4-fb7f-dc55-f62a51a6997a", "54b97ba5-9858-a563-e02f-89b0a6c450a1", "9524834a-3577-4445-54c6-1c786775749d" ] } } } Ok for the VM backup using: { "name": "vm backups", "mode": "full", "compression": "zstd", "schedules": { "fdi73l44ewf": { "cron": "40 12 * * 6", "enabled": true, "name": "weekly vm backups", "timezone": "Europe/Rome" } }, "settings": { "": { "concurrency": 2, "nRetriesVmBackupFailures": 3, "timeout": 7200000, "offlineSnapshot": false, "checkpointSnapshot": true }, "fdi73l44ewf": { "exportRetention": 1, "healthCheckSr": "ac595bf5-ca14-586e-6f89-9e4e884043eb", "healthCheckVmsWithTags": [] } }, "remotes": { "id": "40e5ecde-f12f-43c7-befd-c1f0a76e8a25" }, "vms": { "id": { "__or": [ "64adb2e9-e0f3-9e70-b08f-7c8653415053", "aaf9e807-3ee9-37ff-05ec-cf782ba74d56", "7cd276ae-b7df-06d8-3da8-48262750051d", "9ccfcd0c-a35e-7f45-a0d2-19db678291f1", "42afaaea-ada4-fb7f-dc55-f62a51a6997a", "54b97ba5-9858-a563-e02f-89b0a6c450a1", "9524834a-3577-4445-54c6-1c786775749d" ] } } }

We'll keep XOA standalone for XenServer, the pricing will likely change for those (per host and only with Premium). In your case, you'll have 2 options: going for XOA standalone for XenServer per host (dunno yet the price per host we'll have) migrating to XCP-ng to get an Essential bundle

This helped me https://www.youtube.com/watch?v=H5PJ_tHQlZk

@olivierlambert Thank you.

@olivierlambert

Hi and welcome! Have you registered your XOA already? Enable the free trial to get all the plugins

We updated the image recently without updating the md5 (doing the modification now). If your sum is c3dd6102bef4d9aa2b7c5fc2c38c2bda that's fine

We see occasional VM backup failures that match the above pattern (ENOENT on a "path" in a S3 bucket remote destination during a VM delta backup). We are running a backup of a large VM (200GB disk) to S3 bucket in Ceph Object Gateway. Our current XO is this commit https://github.com/vatesfr/xen-orchestra/commit/da51b1649c65d7d78a4eb25cc46c488ce2552800 from 2024-03-04. The failure is intermittent. Example failure log: { "data": { "mode": "delta", "reportWhen": "always" }, "id": "1712008868496", "jobId": "49c2634a-f281-46b7-9192-aea8c786f3d4", "jobName": "ZABBIX VM to ceph02 Delta Backup", "message": "backup", "scheduleId": "6b1280f0-21e7-475c-9af3-330726108568", "start": 1712008868496, "status": "failure", "infos": [ { "data": { "vms": [ "c020890b-c1a7-79cf-0b55-aa340ba9226b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c020890b-c1a7-79cf-0b55-aa340ba9226b", "name_label": "zabbix" }, "id": "1712008870266", "message": "backup VM", "start": 1712008870266, "status": "failure", "tasks": [ { "id": "1712008870283", "message": "clean-vm", "start": 1712008870283, "status": "success", "end": 1712008872364, "result": { "merge": false } }, { "id": "1712008872791", "message": "snapshot", "start": 1712008872791, "status": "success", "end": 1712008873901, "result": "902d65f0-8c9b-2460-dad7-d041ceefebdc" }, { "data": { "id": "d8d6a743-8c15-4b9a-baf4-df5d7d379b6e", "isFull": false, "type": "remote" }, "id": "1712008873919", "message": "export", "start": 1712008873919, "status": "failure", "tasks": [ { "id": "1712008875215", "message": "transfer", "start": 1712008875215, "status": "success", "end": 1712009333076, "result": { "size": 35343973376 } }, { "id": "1712009335152", "message": "clean-vm", "start": 1712009335152, "status": "failure", "tasks": [ { "id": "1712009336153", "message": "merge", "start": 1712009336153, "status": "failure", "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585031 } ], "end": 1712009585032 } ], "end": 1712009585032 } I know I should be on the current XO to report a problem. Not asking for a fix now, just trying to document that this defect still exists.

See the documentation for how to reset the SSH password -- https://xen-orchestra.com/docs/xoa.html#first-console-connection

@Squall124 While being a bug and I understand it can be a pita but then there's a workaround, I still think you're making a storm in a waterglass by saying "really scary when we think about using this solution in production". Nobody is forcing you to this. We ourselves are more than happy, to say the least, to have made the switch from Vmware (with 2yrs of paid contract in place) to xcp-ng last year! The level of support and interaction you get, you can only dream about it getting it on the black side (even before the Broadcom debacle), puts up with this smaller nuisances by the ton. Just my 2 cents.

@olivierlambert Thank you, this solve the problem

Then you should open a support ticket, please

Yes, you can create a group and provide ACLs on the group (IIRC). So all users in the group will have the same permissions.

@waveguide This is a good solution, glad it's working.

Message for @julien-f who both broke and fixed it

It's not that I'm not happy, I'm just explaining why it makes more sense to post issues on the GitHub repo that you used to install XO in the first place What if a 3rd party installed is installing XO in the incorrect way? We have already many messages to deal with, adding potential 3rd party installer would make it even harder for us regarding our community support

Solved by 583c728

And I confirm this, the problem has disappeared. Thank you

This can happen since using the sources means also experiment some bugs, while on XOA, you should be safer because we do QA before each release. If you want stability and a production grade solution, go for XOA Anyway, the bug is now fixed, get the latest commit on master and try again, it should work!

@paulorrockgit thanks paul for the bug report and exploration I propose this bugfix : https://github.com/vatesfr/xen-orchestra/pull/7504 and can deploy it on your XOA