Posts made by nikade

@gskger said in Backup reports on Microsoft Teams:

@olivierlambert Thanks - that is what I like about this forum and XCP-ng: learn and share

We need more ppl like you, this is great information. I am trying this at work tomorrow!

I've got a NVIDIA card in my lab, but I probably need to install that driver you're talking about.
Are there any instructions on the wiki?

01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [NVS 510] (rev a1)

@jedimarcus said in Adding NFS as remote:

https://github.com/xcp-ng/xcp/wiki/FreeNAS-as-Storage-Repository-(SR)

Did you: Check the All dirs option

This is probably it, pretty common mistake.

@olivierlambert said in 10GB backup with XOA (XCP to windows NFS):

First thing on the calendar will be to use the XO proxy code in threads directly in XOA for "normal" backups. It will allow more XOA usage and we'll see if it helps (at least it will help to scale).

Sounds like a step in the right direction, any ETA?

@olivierlambert said in 10GB backup with XOA (XCP to windows NFS):

We'll get rid of potential stream buffer limitation in a relatively short time frame, but again, I'm pretty sure the export speed on the host is the real bottleneck.

As we are also working on XCP-ng storage performance in parallel, I'm confident we'll be able to improve that

I think the storage performance-improvements will contribute to increasing the export-storage as well as working on the XAPI export/import performance.
I know the XCP team is aware of this issue and I am also confident that it will be a priority

We see about the same speed, just below 1Gbit/s no matter how many concurrent backups and what not. The bursting is probably due to buffering in the node-process.

I think this is a limitation in XAPI just like @olivierlambert wrote earlier and nothing much you can do in XO to speed it up.

We've been trying to speed things up for a long time, we ended up using the Delta-backup and Continious Replication which are both incremental. This allows us to backup all our VM's over night to our NFS-machines.

Did anyone try the performance of VM export/import yet?
I am really excited to see the increase since this was one of the improvements mentioned by Citrix.

Did anyone try VM export/import to see how much the performance has been improved?
More interesting would be if someone tried it with 10G or making backups with XOA.

@akurzawa said in Extrlemy slow backup speec = about few MB/s:

@nikade said in Extrlemy slow backup speec = about few MB/s:

Maybe you saw the same thing I did - I also had major issues with backups. Then in the summer of 2019 I upgraded our XOA (Yes, we have the paid version) and everything was going 3-4x times faster. Must have been some major improvement in the XOA code.

So maybe some official info that if you have XO from source Your backup speeds will be as low as 3-4 MB/s? Or what?

Sorry if I was unclear - XOA is built from the same sources so there is no difference in that aspect.
All tho, there was a big performance issue regarding backups which was resolved.

I am using XO from sources in my "private" infrastructure and I can assure you there are no "limits" on purpose.

@synologic said in Extrlemy slow backup speec = about few MB/s:

Ok, upgrading to the latest version worked. Thank you.

I see some improvements in the backup speed like:

• from xcp 8.1 - no compression - about 33 MiB/s
• from xcp 8.1 - compression (zstd) about 15 MiB/s

both are up from about 4 MiB/s

However, i am also seeing vast improvements when backing up from XCP 8.0.0 with transfer speeds (as reported by iptraf-ng) going up to 700 Mbps, which was not happening before.

Viorel

Maybe you saw the same thing I did - I also had major issues with backups. Then in the summer of 2019 I upgraded our XOA (Yes, we have the paid version) and everything was going 3-4x times faster. Must have been some major improvement in the XOA code.

Hi,

I've been struggling with this for many years and I havent found any solution yet.
Our setup is not exactly like yours, but it is simular.

We have 10st XenServer's with 2x10Gb each in 2 different pools, the SR's are located on a NFS share over 4x10Gb and benchmarks from within a VM gives 450-600MB/s in both read and write so I know that is not the limiting factor.
XOA runs in Pool A, so the physical host that it runs on has 2x10Gb NIC.

The backup NFS are 2 different FreeNAS machines with 2x1Gbit nic in each, running a VM from there and running a disk-benchmark gives me around 100-110MB/s per NFS share, in total around ~200MB/s which is fine.

When we backup we backup about 100 VM's to the 2 NFS shares, Pool A backups to NFS Backup A and Pool B backups to NFS Backup B and no matter what we do the network seems to be limited to 800Mbit/s.
If we backup a single VM it goes up to 800Mbit/s, if we backup 4 at the same time it is still limited to 800Mbit/s, since all traffic passes through XOA I am under the understanding that this must be a limitation in XOA or the XAPI-mechanism used by XOA to export the VM's.

Are you using XO from sources?
If yes, this XO will run inside a regular VM and therefor I'd say yes, you can put this script into crontab.

If you are using the XOA appliance I wouldnt recommend it just due to the fact that they could blame it incase anything goes wrong

We have been using CR at work for over a year now and it is working great. There is no point of using the DR-feature since it is way slower and doesnt support incremental backups.

@akurzawa I think the stats looks fine to be honest. Maybe there is something with the TCP offloading but I have no experience from that.
Please share if you manage to find anything that speeds up the performance.

@kevdog said in HTTP to HTTPS redirection:

@nikade Hey thanks for for the link. I ended up just using a LetsEncrypt cert rather than self signed. I think had to add a DNS host override on my router to associate the Local LAN address of the xo server with the domain name of the server contained in the certificate -- Like 10.0.1.50 ---> xo.example.com. Thanks for pointing me in the right direction on this one.

Yeah that is a good solution as well, I hope this helps others in the future who wants to secure their XO with https

To be honest there is a lot of "magic" stuff happening during the handling of disks. I've noticed as well there there might be activity on the sources-server before the destination-server even reacts.

I guess you'd have to read the code to be able to give a certain answer to that question - All tho I wouldnt suprise me if there is some snapshotting and stuff going on on the source-server.

Are you using 7200k disks by any chance? The increase going to 10k or even 15k sas-drives is incredible and I know that XS/XCP is really picky when it comes to I/O.
Even our machines at work that has PERC H710p controllers with 1Gb cache and SSD-drives seems slow at times....

Oh I see, sorry for reading a bit too fast @akurzawa
I dont use the Disaster-recovery myself, we use Continous-replication because we want it to be incremental.

Is tapdisk reaching 100% on both hosts or only on the sender?

I have been running Windows 10 and Windows Server 2016 now for a long time on xcp-ng.
last weekend I upgraded one of my test-servers to xcp-ng 8.0 and Windows Server 2016 still runs fine.
This is the output of systeminfo

C:\Windows\system32>systeminfo

Host Name:                 HEL-DC1
OS Name:                   Microsoft Windows Server 2016 Standard
OS Version:                10.0.14393 N/A Build 14393
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Primary Domain Controller
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                00377-60000-00000-AA934
Original Install Date:     2018-10-20, 17:26:33
System Boot Time:          2019-10-01, 23:13:37
System Manufacturer:       Xen
System Model:              HVM domU
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 158 Stepping 9 GenuineIntel ~3600 Mhz
BIOS Version:              Xen 4.7.5-5.4.1.xcp, 2018-08-02

I am using these tools:

@kevdog said in HTTP to HTTPS redirection:

I'm just curious on using this --- do you have a SSL cert with the server name and are you accessing XO through an address like: https://xoserver.example.com? I'm just curious since my XO server is located at 10.0.1.11 and Chrome states cert is invalid -- since I believe SSL needs to resolve to hostnames and not IP addresses.

I have generated a self-signed certificate which I am using to reach my XOA. You can do this very easily by using the "openssl"-tool.
https://www.linux.com/tutorials/creating-self-signed-ssl-certificates-apache-linux/ explains how to create both they cert and key-files that you need.

It would be interesting if you could try the DR-copy to another remote-machine just to see if tapdisk is as busy.
I noted that it is having a rather high % in the busy-column, around 90%, which would indicate that there's a bottleneck somewhere.

Did you try any other backup-type to a remote nfs for example?