Hi @Andrew, Thanks for the report. I'm seeing the same on my end, so I'll pass this along to the developers. Dan

@snowstruk The stock XCP 8.3 r8125 driver works for my 8125 and 8126 devices. I have not tried it a 5G, so it's possible the 8126 is not working at 5G (I don't see it advertised) with the current driver. I have run the current dedicated realtek 8125 driver and it works but I have not tried the 8126 driver. I don't know how many people are using the 8126 chipset (I guess not many). If your device is not working at >1G then it sees to be a hardware incompatibility issue. You could try loading the new 8126 firmware to see if that helps. Here's the output about my test 8126 card (attached to a 2.5G switch): # lspci -v -s 04:00.0 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller (rev 01) Subsystem: Realtek Semiconductor Co., Ltd. Device 0123 Flags: bus master, fast devsel, latency 0, IRQ 18 I/O ports at 3000 [size=256] Memory at e0d00000 (64-bit, non-prefetchable) [size=64K] Memory at e0d10000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at e0d20000 [disabled] [size=64K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] Express Endpoint, MSI 01 Capabilities: [b0] MSI-X: Enable+ Count=32 Masked- Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [170] Device Serial Number 01-00-00-00-b4-b3-c9-88 Capabilities: [180] #19 Capabilities: [190] Transaction Processing Hints Capabilities: [21c] Latency Tolerance Reporting Capabilities: [224] L1 PM Substates Capabilities: [234] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Kernel driver in use: r8125 Kernel modules: r8125 # ethtool -i eth1 driver: r8125 version: 9.012.04-NAPI-PTP-RSS firmware-version: rtl8126a-2_0.0.2 02/01/24 expansion-rom-version: bus-info: 0000:04:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: no # ethtool eth1 Settings for eth1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseX/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseX/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseX/Full Link partner advertised pause frame use: Symmetric Receive-only Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 2500Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000033 (51) drv probe ifdown ifup Link detected: yes

@MathieuRA For this issue, fix-server-connect works for me. Thanks.

@Andrew said in XOA fails after update to 5.106.0: @acebmxer Did you activate your License? Click on XOA then Licenses (at the top). Make sure you see This license is active on this XOA Yes in 5.105.0 (fresh install) it worked. When i upgrated to 5.108.1 it stoped. I tried v5.109 and failed. same as @Lhoust Using XO from sources for the moment.

I mean I agree with the idea that sounds good, I need to be reviewed again to see if we could do better than we did

Hello @Andrew, thanks for your question! First off, I haven't tested Deno or Node with Xen-Orchestra yet. Changing the Node version can sometimes introduce bugs—some of which may be subtle. For that reason, I’d prefer not to venture into this unless it truly proves to be worth the effort. Secondly, in my experience, benchmarks don’t always translate into meaningful results in complex applications. That said, we are keeping an eye on alternative engines, both in terms of performance and the new features they offer. If we determine that the benefits are significant, we’d definitely consider migrating. Additionally, if any users want to experiment with Deno or Bun, they’re welcome to start dedicated threads on the forum to collaborate. We’d be happy to review and merge compatibility PRs as long as they don’t break Node and aren’t overly complex.

Worth creating a GH XO issue with most details as possible

We are implementing tests in our CI to detect this issue and protect against regressions, then we'll investigate the cause.

Got it. We might ship that in a patch release if it's bundled in our tomorrow's latest XOA release.

@stormi Thanks @stormi. I have to get used to XO. I always use XCP-ng Admin on Windows because old habits die hard, and there I cannot even select UEFI with all the Linux templates. The magic that XO does is that it allows to use the templates, but with UEFI. I have a VM that I've deployed with the "Other install media" option, so I just dettached the disk and reimported on the new one created by XO, with the correct template. Everything seems fine. So my question about inheriting the template is not relevant if we use XO. Hope my understand is correct. Thanks.

@florent I see it has been merged into master. Yes, transfer size seems to be back to before. I have not checked for damage. At worst I can just force a full backup.

We see occasional VM backup failures that match the above pattern (ENOENT on a "path" in a S3 bucket remote destination during a VM delta backup). We are running a backup of a large VM (200GB disk) to S3 bucket in Ceph Object Gateway. Our current XO is this commit https://github.com/vatesfr/xen-orchestra/commit/da51b1649c65d7d78a4eb25cc46c488ce2552800 from 2024-03-04. The failure is intermittent. Example failure log: { "data": { "mode": "delta", "reportWhen": "always" }, "id": "1712008868496", "jobId": "49c2634a-f281-46b7-9192-aea8c786f3d4", "jobName": "ZABBIX VM to ceph02 Delta Backup", "message": "backup", "scheduleId": "6b1280f0-21e7-475c-9af3-330726108568", "start": 1712008868496, "status": "failure", "infos": [ { "data": { "vms": [ "c020890b-c1a7-79cf-0b55-aa340ba9226b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c020890b-c1a7-79cf-0b55-aa340ba9226b", "name_label": "zabbix" }, "id": "1712008870266", "message": "backup VM", "start": 1712008870266, "status": "failure", "tasks": [ { "id": "1712008870283", "message": "clean-vm", "start": 1712008870283, "status": "success", "end": 1712008872364, "result": { "merge": false } }, { "id": "1712008872791", "message": "snapshot", "start": 1712008872791, "status": "success", "end": 1712008873901, "result": "902d65f0-8c9b-2460-dad7-d041ceefebdc" }, { "data": { "id": "d8d6a743-8c15-4b9a-baf4-df5d7d379b6e", "isFull": false, "type": "remote" }, "id": "1712008873919", "message": "export", "start": 1712008873919, "status": "failure", "tasks": [ { "id": "1712008875215", "message": "transfer", "start": 1712008875215, "status": "success", "end": 1712009333076, "result": { "size": 35343973376 } }, { "id": "1712009335152", "message": "clean-vm", "start": 1712009335152, "status": "failure", "tasks": [ { "id": "1712009336153", "message": "merge", "start": 1712009336153, "status": "failure", "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585029, "result": { "cause": { "name": "NoSuchKey", "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "attempts": 1, "totalRetryDelay": 0 }, "Code": "NoSuchKey", "BucketName": "xcp-ng-mama-pool-vm-backup", "RequestId": "tx00000c27bed21222dc684-00660b3170-302aa3c-s3", "HostId": "302aa3c-s3-s3", "message": "UnknownError" }, "code": "ENOENT", "path": "/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52", "message": "ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'", "name": "Error", "stack": "Error: ENOENT: no such file or directory '/xo-vm-backups/c020890b-c1a7-79cf-0b55-aa340ba9226b/vdis/49c2634a-f281-46b7-9192-aea8c786f3d4/378753c5-9072-47b7-b879-0488a86e85ba/data/b70bc690-28c7-440a-8147-bf6d61c179de.vhd/blocks/10/52'\n at S3Handler._copy (/opt/xo/xo-builds/xen-orchestra-202403041050/@xen-orchestra/fs/dist/s3.js:157:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1712009585031 } ], "end": 1712009585032 } ], "end": 1712009585032 } I know I should be on the current XO to report a problem. Not asking for a fix now, just trying to document that this defect still exists. 0 julien-f committed to vatesfr/xen-orchestra fix(xo-server/robots.txt): must not require authentication Introduced by a2c36c083 This file is intended for search engines, which, by definition are not signed in.

Thanks everyone for your feedback!

@florent Updated XO source to latest master and it's all green again. Commit 3f316fcaea11775fc1393737cc6e09d8138da62d fixed the issue for me. Thanks!

@julien-f Thanks. I updated to 0444cf0 and CR times are normal again (as expected).

@julien-f I just wanted to update and let you know that whatever changes you made fixed the issue for me permanently! I have been able to routinely update since that time and have had no further issues. Thanks for all of your hard work, Julien-f!

@julien-f The new update seems to be working correctly now. Thanks for the quick update fix. The backup cache fixes/cleanup made a huge difference (improvement). I was trying to figure out what was going on with my backups as they were running but were missing from the restore list. After the cache cleanup, it showed 3-4 times the number of restore points! All the missing stuff and more. It took some time to clean up all the extra stuff, but backups seem to be back on track again.

@Darkbeldin I totally missed that.... I know I have used it before. It must be a long week already. Thanks.

@Andrew I didn't say "Xen" I said XenServer.