Posts made by nikade

Did anyone try the performance of VM export/import yet?
I am really excited to see the increase since this was one of the improvements mentioned by Citrix.

Did anyone try VM export/import to see how much the performance has been improved?
More interesting would be if someone tried it with 10G or making backups with XOA.

@akurzawa said in Extrlemy slow backup speec = about few MB/s:

@nikade said in Extrlemy slow backup speec = about few MB/s:

Maybe you saw the same thing I did - I also had major issues with backups. Then in the summer of 2019 I upgraded our XOA (Yes, we have the paid version) and everything was going 3-4x times faster. Must have been some major improvement in the XOA code.

So maybe some official info that if you have XO from source Your backup speeds will be as low as 3-4 MB/s? Or what?

Sorry if I was unclear - XOA is built from the same sources so there is no difference in that aspect.
All tho, there was a big performance issue regarding backups which was resolved.

I am using XO from sources in my "private" infrastructure and I can assure you there are no "limits" on purpose.

@synologic said in Extrlemy slow backup speec = about few MB/s:

Ok, upgrading to the latest version worked. Thank you.

I see some improvements in the backup speed like:

• from xcp 8.1 - no compression - about 33 MiB/s
• from xcp 8.1 - compression (zstd) about 15 MiB/s

both are up from about 4 MiB/s

However, i am also seeing vast improvements when backing up from XCP 8.0.0 with transfer speeds (as reported by iptraf-ng) going up to 700 Mbps, which was not happening before.

Viorel

Maybe you saw the same thing I did - I also had major issues with backups. Then in the summer of 2019 I upgraded our XOA (Yes, we have the paid version) and everything was going 3-4x times faster. Must have been some major improvement in the XOA code.

Hi,

I've been struggling with this for many years and I havent found any solution yet.
Our setup is not exactly like yours, but it is simular.

We have 10st XenServer's with 2x10Gb each in 2 different pools, the SR's are located on a NFS share over 4x10Gb and benchmarks from within a VM gives 450-600MB/s in both read and write so I know that is not the limiting factor.
XOA runs in Pool A, so the physical host that it runs on has 2x10Gb NIC.

The backup NFS are 2 different FreeNAS machines with 2x1Gbit nic in each, running a VM from there and running a disk-benchmark gives me around 100-110MB/s per NFS share, in total around ~200MB/s which is fine.

When we backup we backup about 100 VM's to the 2 NFS shares, Pool A backups to NFS Backup A and Pool B backups to NFS Backup B and no matter what we do the network seems to be limited to 800Mbit/s.
If we backup a single VM it goes up to 800Mbit/s, if we backup 4 at the same time it is still limited to 800Mbit/s, since all traffic passes through XOA I am under the understanding that this must be a limitation in XOA or the XAPI-mechanism used by XOA to export the VM's.

Are you using XO from sources?
If yes, this XO will run inside a regular VM and therefor I'd say yes, you can put this script into crontab.

If you are using the XOA appliance I wouldnt recommend it just due to the fact that they could blame it incase anything goes wrong

We have been using CR at work for over a year now and it is working great. There is no point of using the DR-feature since it is way slower and doesnt support incremental backups.

@akurzawa I think the stats looks fine to be honest. Maybe there is something with the TCP offloading but I have no experience from that.
Please share if you manage to find anything that speeds up the performance.

@kevdog said in HTTP to HTTPS redirection:

@nikade Hey thanks for for the link. I ended up just using a LetsEncrypt cert rather than self signed. I think had to add a DNS host override on my router to associate the Local LAN address of the xo server with the domain name of the server contained in the certificate -- Like 10.0.1.50 ---> xo.example.com. Thanks for pointing me in the right direction on this one.

Yeah that is a good solution as well, I hope this helps others in the future who wants to secure their XO with https

To be honest there is a lot of "magic" stuff happening during the handling of disks. I've noticed as well there there might be activity on the sources-server before the destination-server even reacts.

I guess you'd have to read the code to be able to give a certain answer to that question - All tho I wouldnt suprise me if there is some snapshotting and stuff going on on the source-server.

Are you using 7200k disks by any chance? The increase going to 10k or even 15k sas-drives is incredible and I know that XS/XCP is really picky when it comes to I/O.
Even our machines at work that has PERC H710p controllers with 1Gb cache and SSD-drives seems slow at times....

Oh I see, sorry for reading a bit too fast @akurzawa
I dont use the Disaster-recovery myself, we use Continous-replication because we want it to be incremental.

Is tapdisk reaching 100% on both hosts or only on the sender?

I have been running Windows 10 and Windows Server 2016 now for a long time on xcp-ng.
last weekend I upgraded one of my test-servers to xcp-ng 8.0 and Windows Server 2016 still runs fine.
This is the output of systeminfo

C:\Windows\system32>systeminfo

Host Name:                 HEL-DC1
OS Name:                   Microsoft Windows Server 2016 Standard
OS Version:                10.0.14393 N/A Build 14393
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Primary Domain Controller
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                00377-60000-00000-AA934
Original Install Date:     2018-10-20, 17:26:33
System Boot Time:          2019-10-01, 23:13:37
System Manufacturer:       Xen
System Model:              HVM domU
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 158 Stepping 9 GenuineIntel ~3600 Mhz
BIOS Version:              Xen 4.7.5-5.4.1.xcp, 2018-08-02

I am using these tools:

@kevdog said in HTTP to HTTPS redirection:

I'm just curious on using this --- do you have a SSL cert with the server name and are you accessing XO through an address like: https://xoserver.example.com? I'm just curious since my XO server is located at 10.0.1.11 and Chrome states cert is invalid -- since I believe SSL needs to resolve to hostnames and not IP addresses.

I have generated a self-signed certificate which I am using to reach my XOA. You can do this very easily by using the "openssl"-tool.
https://www.linux.com/tutorials/creating-self-signed-ssl-certificates-apache-linux/ explains how to create both they cert and key-files that you need.

It would be interesting if you could try the DR-copy to another remote-machine just to see if tapdisk is as busy.
I noted that it is having a rather high % in the busy-column, around 90%, which would indicate that there's a bottleneck somewhere.

Did you try any other backup-type to a remote nfs for example?

Thanks for the output. Is this "top" running while you are doing a DR? I dont see any processes related to copying vdi's or even sparse_dd.

1. How many VM's do you have running on this host?
2. What does "top" show on the two hosts when a backup-job is running?
3. XO is very much depending on the storage you are backing up to, what kind of remote are you using and are you able to benchmark it with something else than xcp-ng?

Hi @declan-marks,

I've been in the same situation as you and I will share my configuration which is working:

root@xoa:~# cat /opt/xen-orchestra/packages/xo-server/.xo-server.yaml
# BE *VERY* CAREFUL WHEN EDITING!
# YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
# visit http://www.yamllint.com/ to validate this file as needed

#=====================================================================

# Example XO-Server configuration.
#
# This file is automatically looking for at the following places:
# - `$HOME/.config/xo-server/config.yaml`
# - `/etc/xo-server/config.yaml`
#
# The first entries have priority.
#
# Note: paths are relative to the configuration file.

#=====================================================================

# It may be necessary to run XO-Server as a privileged user (e.g.
# `root`) for instance to allow the HTTP server to listen on a
# [privileged ports](http://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html).
#
# To avoid security issues, XO-Server can drop its privileges by
# changing the user and the group is running with.
#
# Note: XO-Server will change them just after reading the
# configuration.

# User to run XO-Server as.
#
# Note: The user can be specified using either its name or its numeric
# identifier.
#
# Default: undefined
#user: 'nobody'

# Group to run XO-Server as.
#
# Note: The group can be specified using either its name or its
# numeric identifier.
#
# Default: undefined
#group: 'nogroup'

#=====================================================================

# Configuration of the embedded HTTP server.
http:

  # Hosts & ports on which to listen.
  #
  # By default, the server listens on [::]:80.
  listen:
    # Basic HTTP.
    -
      # Address on which the server is listening on.
      #
      # Sets it to 'localhost' for IP to listen only on the local host.
      #
      # Default: all IPv6 addresses if available, otherwise all IPv4
      # addresses.
      #hostname: 'localhost'

      # Port on which the server is listening on.
      #
      # Default: undefined
      port: 80

      # Instead of `host` and `port` a path to a UNIX socket may be
      # specified (overrides `host` and `port`).
      #
      # Default: undefined
      #socket: './http.sock'

    # Basic HTTPS.
    #
    # You can find the list of possible options there https://nodejs.org/docs/latest/api/tls.html#tls.createServer
    -
    #   # The only difference is the presence of the certificate and the
    #   # key.
    #   #
    #   #hostname: '127.0.0.1'
        port: 443

    #   # File containing the certificate (PEM format).
    #
    #   # If a chain of certificates authorities is needed, you may bundle
    #   # them directly in the certificate.
    #   #
    #   # Note: the order of certificates does matter, your certificate
    #   # should come first followed by the certificate of the above
    #   # certificate authority up to the root.
    #   #
    #   # Default: undefined
        cert: '/etc/ssl/private/xoa.cert'

    #   # File containing the private key (PEM format).
    #   #
    #   # If the key is encrypted, the passphrase will be asked at
    #   # server startup.
    #   #
    #   # Default: undefined
        key: '/etc/ssl/private/xoa.key'

  # If set to true, all HTTP traffic will be redirected to the first
  # HTTPs configuration.
  redirectToHttps: true

  # List of files/directories which will be served.
  mounts:
    '/': '/opt/xen-orchestra/packages/xo-web/dist'

  # List of proxied URLs (HTTP & WebSockets).
  proxies:
    # '/any/url': 'http://localhost:54722'

# HTTP proxy configuration used by xo-server to fetch resources on the
# Internet.
#
# See: https://github.com/TooTallNate/node-proxy-agent#maps-proxy-protocols-to-httpagent-implementations
#httpProxy: 'http://jsmith:qwerty@proxy.lan:3128'

#=====================================================================

# Connection to the Redis server.
redis:
    # Unix sockets can be used
    #
    # Default: undefined
    #socket: /var/run/redis/redis.sock

    # Syntax: redis://[db[:password]@]hostname[:port][/db-number]
    #
    # Default: redis://localhost:6379/0
    #uri: redis://redis.company.lan/42

    # List of aliased commands.
    #
    # See http://redis.io/topics/security#disabling-of-specific-commands
    #renameCommands:
    #  del: '3dda29ad-3015-44f9-b13b-fa570de92489'
    #  srem: '3fd758c9-5610-4e9d-a058-dbf4cb6d8bf0'


# Directory containing the database of XO.
# Currently used for logs.
#
# Default: '/var/lib/xo-server/data'
#datadir: '/var/lib/xo-server/data'

Try creating a directory within that share as root and then chown -R nobody:nogroup folder/
After doing so you should be able to mount /path/to/your/share/folder

@blafuente said in XenApp or similar on XCP-ng?:

@nikade Great! Hope it meets your expectations and needs. If you need any help we'll be more than happy to help you out

I havent gotten my free serial yet so I havent been able to test it out, but it looks pretty neat in the videos.

@blafuente said in XenApp or similar on XCP-ng?:

Hi,

Yes, you can use UDS Enterprise for Windows (with RDS) and Linux (with X2Go) virtual applications on XCP-ng. There's a Free version for 10 users lifetime. You can request it here: https://www.udsenterprise.com/en/downloads/

I just submitted for a free version to check this out, looks promising on the webpage.