@olivierlambert thanks for confirming!
Posts made by Forza
-
RE: Xen 4.17 on XCP-ng 8.3!
Would be nice to see if Xen 4.17 fixes the slow network on EPYC servers.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
@florent hi, for the full backup test above, I used normal mode with zstd enabled. There were no snapshots of the source VM and it was stored on local ssd storage on the pool master.
-
RE: VMβs Slow booting
@bassopt said in VMβs Slow booting:
@olivierlambert hi, again!
Nah, as told before itβs on initramdisk. After loading grub. A Ubuntu test install I did freezes for almost 20 seconds there.
I do have some warnings when I boot xcp-ng regarding x-273 and x-297 .... would that cripple performance that much. ( consumer hardware )
Sounds like you have the wrong template. If you disable "quiet" option in grub (press e, then go to the kernel command line) you may see it is error out on enabling some hardware feature. I have seen it happen when trying to enumerate the CPUs. Have a look. You might be able to see in
dmesg
orjournalctl
too. -
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
I did a 3 x backup of a single VM hosted on an SSD drive on the same host as XOA is running, which is also pool master.
This is with
nconnect=1
:
This is with
nconnect=16
:
The transfer speed according to XOA is slightly less, but looking at the bandwidth graph, it looks like the LACP bonded network on the storage server is reaching a higher max throughput.
I will test some more with incremental backups and see if there's a difference with them.
If we ignore the
nconnect
for a second, and just look at the graphs, it seems we have a lot of possibilities to improve the backup performance if we could make the transfer more even. What is causing this type of pattern? I do not believe any coalesce was happening during this test. -
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
Ah, that's a shame, but reasonable for a point release. Maybe next major release?
This was an interesting read regarding nconnect on Azure https://learn.microsoft.com/en-us/azure/storage/files/nfs-performance
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
Changing nconnect requires restart of XOA (seems remount doesn't take a new value for nconnect), and I am heading home now, so I will try some additional benchmarks later in the week.
Btw, does xcp-ng 8.3 support nconnect? 8.2 that I am using does not.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
That would explain why the transfer rate is uneven, as some bandwidth and IO is used by the coalescing processes.
Overall the transfer is somewhat faster. about 1.5 hours for 253GiB on 1Gbit/s connection. Before it was maybe 2ish hours.
I lost the backup logs in XOA since I switched to the new version, but looking back in the Netdata stats from the backup server we can see the following pattern for the previous run of the same backup job. Just looking at the timestamps it looks pretty similar, while the bandwidth used looks less. Maybe some rounding errors in Netdata graphs?
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
@olivierlambert
Yes, I am checking on the XOA VM itself.Here is a view over the XOA VM. You can see that the full backup was started around 3:10 pm:
This is the NFS stats from the backup server (Remote) for the same period:
Why is the server doing reads? Maybe coalesce? But is there coalesce on full backups?
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
Testing full backups. Seems at least XOA is saturating the link for larger VMs. It does have quite a bit of delay outside of the transfer itself, so the overall BW that is logged in the web interface is lower.
This is
bmon
running inside XOA.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
I usually read the blog posts, but I must have missed it that time.
As for performance it is a little difficult to test i a structured manner, but the initial incremental backup seems to run very well. Normally I have about 60-90MB/s as max before, but also often around 20-30MB/s. This time the backup took 6 minutes, and normal is around 15 minutes. This is over a 1 Gbit/s network.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
I will do.
It would be nice to have an notice that a new XOA is available, in addition to the normal updates. Perhaps something on the https://xoa/#/xoa/support page?
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
@olivierlambert said in Updated XOA with kernel >5.3 to support nconnect nfs option:
XOA is based on Debian 12. I'm not against doing a custom kernel if it's really a big bonus in terms of performance
Debian 12 should use kernel 6.1, so that would be ok. Maybe I am having an old XOA as my kernel is 4.19. This is what shows when I login:
login as: xoa xoa@<redacted>'s password: Linux <redacted> 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 __ __ ____ _ _ \ \ / / / __ \ | | | | \ V / ___ _ __ | | | |_ __ ___| |__ ___ ___| |_ _ __ __ _ > < / _ \ '_ \ | | | | '__/ __| '_ \ / _ \/ __| __| '__/ _` | / . \ __/ | | | | |__| | | | (__| | | | __/\__ \ |_| | | (_| | /_/ \_\___|_| |_| \____/|_| \___|_| |_|\___||___/\__|_| \__,_| Welcome to XOA Unified Edition, with Pro Support. * Restart XO: sudo systemctl restart xo-server.service * Display status: sudo systemctl status xo-server.service * Display logs: sudo journalctl -u xo-server.service * Register your XOA: sudo xoa-updater --register * Update your XOA: sudo xoa-updater --upgrade OFFICIAL XOA DOCUMENTATION HERE: https://xen-orchestra.com/docs/xoa.html Support available at https://xen-orchestra.com/#!/member/support In case of issues, use `xoa check` for a quick health check. Build number: 21.01.02 Based on Debian GNU/Linux 10 (Stable) 64bits in PVHVM mode
-
Updated XOA with kernel >5.3 to support nconnect nfs option
Hi,
Are there any plans to release a new XOA based on kernel >=5.3 so that we can use thenconnect
nfs mount option. The nconnect option increases NFS performance by utilising multiple connections to the same remote.https://medium.com/@emilypotyraj/use-nconnect-to-effortlessly-increase-nfs-performance-4ceb46c64089
-
RE: Epyc VM to VM networking slow
@olivierlambert said in Epyc VM to VM networking slow:
If I wouldn't be convinced to fix it, I wouldn't throw money & time to solve the problem
I think everyone knows this. Nevertheless, it is frustrating anyone if it becomes a bottleneck.
I am curious, do we know if this happens on Xen systems, or if it happens on xcp-ng systems where Open vSwitch is not used?
-
RE: I/O errors on file restore
@olivierlambert said in I/O errors on file restore:
I/O error on a sector is usually due to something between your XOA and the BR. It could be many things though.
I did further testing and found that I can reproduce the issue. While the file-restore dialogue is open, it is possible to access the files via XOA console in /tmp/<xxxx mountpoint>/, but as soon as you press "OK" the access to the disk disappears and all processes trying the mount point freeze. I reported the exact details in the ticket.
-
RE: I/O errors on file restore
You mean errors on the backup server? I have no error there and have been able to read all files (cat >/dev/null) without issue. The server is using Btrfs with RAID1c3 so it verifies checksums on all reads.
-
RE: I/O errors on file restore
@olivierlambert Ticket created.
It could be a problem of the backup file. I am attempting a normal restore now to see if the full disk can be restored. Could the issue be related to NBD and multiple blocks transfer?
-
RE: I/O errors on file restore
@olivierlambert said in I/O errors on file restore:
Hi,
If you have a support subscription (you said XOA Premium, right?), it's better to create a ticket
Yes I do. I could open the ticket there instead. It was not "urgent" so I thought the advise from the community instead.
-
I/O errors on file restore
Hi, I wanted to do a file restore via XOA (Premium) but had some issues with it.
- First I notice that file restore only lists incremental backups, not full backups.
- I chose an incremental backup and chose tar.gz restore of one directory (about 3GiB uncompressed), but it failed (the download stopped and I got an empty file)
- I tried the export again as a zip file which worked good.
The VM guest OS is Windows 10 with NTFS.
NBD is enabled, but I think it is only used for incremental backups.I had a look in XOA and dmesg has a few errors that look very much like the ones in this old thread https://xcp-ng.org/forum/topic/7519/file-restore-on-large-backups-ends-in-print_req_error-i-o-error-s
[ +4.563750] loop: module loaded [Apr 8 09:44] print_req_error: I/O error, dev loop0, sector 82242200 [Apr 8 09:45] print_req_error: I/O error, dev loop0, sector 0 [ +15.000803] print_req_error: I/O error, dev loop0, sector 82242200 [ +0.001002] Buffer I/O error on dev loop0, logical block 10280275, async page read [ +15.002638] print_req_error: I/O error, dev loop0, sector 0 [Apr 8 09:46] print_req_error: I/O error, dev loop0, sector 60079896 [ +15.002314] print_req_error: I/O error, dev loop0, sector 0 [ +15.002160] print_req_error: I/O error, dev loop0, sector 60079896 [ +0.001001] Buffer I/O error on dev loop0, logical block 7509987, async page read [Apr 8 09:47] print_req_error: I/O error, dev loop0, sector 0 [ +30.002121] print_req_error: I/O error, dev loop0, sector 41011136 [ +15.002544] print_req_error: I/O error, dev loop0, sector 0 [Apr 8 09:48] print_req_error: I/O error, dev loop0, sector 41011136 [ +0.000987] Buffer I/O error on dev loop0, logical block 5126392, async page read [ +15.002323] print_req_error: I/O error, dev loop0, sector 0 [ +30.001776] print_req_error: I/O error, dev loop0, sector 37880256 [ +0.001019] Buffer I/O error on dev loop0, logical block 4735032, async page read [Apr 8 09:49] print_req_error: I/O error, dev loop0, sector 82242200 [ +0.001025] Buffer I/O error on dev loop0, logical block 10280275, async page read [ +15.002031] print_req_error: I/O error, dev loop0, sector 60079896 [ +0.001019] Buffer I/O error on dev loop0, logical block 7509987, async page read [ +15.001864] print_req_error: I/O error, dev loop0, sector 41011136 [ +0.001374] Buffer I/O error on dev loop0, logical block 5126392, async page read [ +15.001471] print_req_error: I/O error, dev loop0, sector 0 [Apr 8 09:50] print_req_error: I/O error, dev loop0, sector 37880256 [ +0.001082] Buffer I/O error on dev loop0, logical block 4735032, async page read