We see occasional VM backup failures that match the above pattern (ENOENT on a "path" in a S3 bucket remote destination during a VM delta backup). We are running a backup of a large VM (200GB disk) to S3 bucket in Ceph Object Gateway. Our current XO is this commit https://github.com/vatesfr/xen-orchestra/commit/da51b1649c65d7d78a4eb25cc46c488ce2552800 from 2024-03-04. The failure is intermittent. Example failure log: { "data": { "mode": "delta", "reportWhen": "always" }, "id": "1712008868496", "jobId": "49c2634a-f281-46b7-9192-aea8c786f3d4", "jobName": "ZABBIX VM to ceph02 Delta Backup", "message": "backup", "scheduleId": "6b1280f0-21e7-475c-9af3-330726108568", "start": 1712008868496, "status": "failure", "infos": [ { "data": { "vms": [ "c020890b-c1a7-79cf-0b55-aa340ba9226b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c020890b-c1a7-79cf-0b55-aa340ba9226b", "name_label": "zabbix" }, "id": "1712008870266", "message": "backup VM", "start": 1712008870266, "status": "failure", "tasks": [ { "id": "1712008870283", "message": "clean-vm", "start": 1712008870283, "status": "success", "end": 1712008872364, "result": { "merge": false } }, { "id": "1712008872791", "message": "snapshot", "start": 1712008872791, "status": "success", "end": 1712008873901, "result": "902d65f0-8c9b-2460-dad7-d041ceefebdc" }, { "data": { "id": "d8d6a743-8c15-4b9a-baf4-df5d7d379b6e", "isFull": false, "type": "remote" }, "id": "1712008873919", "message": "export", "start": 1712008873919, "status": "failure", "tasks": [ { "id": "1712008875215", "message": "transfer", "start": 1712008875215, "status": "success", "end": 1712009333076, "result": { "size": 35343973376 } }, { "id": "1712009335152", "message": "clean-vm", "start": 1712009335152, "status": "failure", "tasks": [ { "id": "1712009336153", "message": "merge", "start": 1712009336153, "status": "failure", "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585031 } ], "end": 1712009585032 } ], "end": 1712009585032 } I know I should be on the current XO to report a problem. Not asking for a fix now, just trying to document that this defect still exists. 0 julien-f committed to vatesfr/xen-orchestra fix(xo-server/robots.txt): must not require authentication Introduced by a2c36c083 This file is intended for search engines, which, by definition are not signed in.

See the documentation for how to reset the SSH password -- https://xen-orchestra.com/docs/xoa.html#first-console-connection

@Squall124 While being a bug and I understand it can be a pita but then there's a workaround, I still think you're making a storm in a waterglass by saying "really scary when we think about using this solution in production". Nobody is forcing you to this. We ourselves are more than happy, to say the least, to have made the switch from Vmware (with 2yrs of paid contract in place) to xcp-ng last year! The level of support and interaction you get, you can only dream about it getting it on the black side (even before the Broadcom debacle), puts up with this smaller nuisances by the ton. Just my 2 cents.

@olivierlambert Thank you, this solve the problem

Then you should open a support ticket, please

Yes, you can create a group and provide ACLs on the group (IIRC). So all users in the group will have the same permissions.

@waveguide This is a good solution, glad it's working.

Message for @julien-f who both broke and fixed it

It's not that I'm not happy, I'm just explaining why it makes more sense to post issues on the GitHub repo that you used to install XO in the first place What if a 3rd party installed is installing XO in the incorrect way? We have already many messages to deal with, adding potential 3rd party installer would make it even harder for us regarding our community support

Solved by 583c728

And I confirm this, the problem has disappeared. Thank you

This can happen since using the sources means also experiment some bugs, while on XOA, you should be safer because we do QA before each release. If you want stability and a production grade solution, go for XOA Anyway, the bug is now fixed, get the latest commit on master and try again, it should work!

@paulorrockgit thanks paul for the bug report and exploration I propose this bugfix : https://github.com/vatesfr/xen-orchestra/pull/7504 and can deploy it on your XOA fbeauchamp opened this pull request in vatesfr/xen-orchestra closed fix(vwmare-explorer): handle raw cdrom #7504

@olivierlambert I was able to get the import working by importing into vmware workstation, exporting to ovf and then converting to ova with vmwares command line tool.

@Tristis-Oris said in SMB ISO share can't upload: @mohammadm looks like Truenas related issue. Check the account permissions. if it stop working after XO update, try to rollback previous commit. User – xcp-ng Read | Write | Execute For now I upload the ISO's through Windows SMB. Because it is accessible from XOA. Only uploading is not working.

I'm using ZeroTier and access works, except for the console view.

@ckargitest You need to run: fs0: cd \efi\boot and then run the program which is in this folder. It took me a while as well to figure it out.

I'm also getting the "self-signed certificate" error sinds the last update. I get it while trying to import/upload ISO. From source, commit e52b1 8.3 beta

I added a big warning in step 3 which should never be done outside a test host that you are ready to reinstall afterwards. @codycrypto's comment is another workaround which replaces it without any impact on the host, but weakens the security by accepting weaker key types.