Backup and the replication - Functioning/Scale

olivierlambert

Adding @florent and @bastien-nollet in the loop for reading those ideas on better reports and proxy automation on assignation.

florent

@fcgo

compression:
on incremental : only if you use the mode "block" on the remote setting, it is done by xoa ( or proxy ) using brotli with setting BROTLI_MIN_QUALITY . The goal is to compress the empty parts of the data blocks exported as zeroes
on full: done by the host, with gzip or zstd
generally the limiting factor is the individual export speed of the xapi, you scale by increasing concurrency . increasing Dom0 resource is good here
latency from host to backup runner is killing performance : proxies are great to have them as near as possible as your source hosts
as Pilow said: proxies are dumbed down XO ( same backup code ) , but lack the scheduling and configuration part. XO is the one launching the
Each backup job use its own process ( so cpu and memory) on XO and proxies, so it's another way to scale

The biggest backup job I saw in the wild ( during a support ) was a few hundreds VMs. Maybe there are bigger one that are working without issue.

fcgo

@florent @pilow
Thank you all, it is clearer.

Can backup copies between sites flow through one proxy directly to another,proxy or does it need to flow from one proxy to XOA to another proxy ?
In this case, could the XOA become a bottleneck ?

Thank you

Pilow

@fcgo when adding an XOPROXY to a job, it flows from XOPROXY to the remote
I think XOA is not involved, checked the network bandwidth, xoa was sleeping

XOPROXY read/writes from source remote to destination remote

fcgo

@Pilow I was asking in the case where the sites are only allowed to communicate via HTTPS, meaning NFS remote at site 1 is accessible through proxy 1, and NFS remote at site 2 is accessible through proxy 2 (XOA being on site 1).

Pilow

@fcgo this is where you reach a limit...
in an XOA backup copy job configuration, you select the proxy, and it drives source AND destination

you cannot have a proxy for SOURCE and a proxy for DESTINATION

Pilow

but each remote (SOURCE and DESTINATION) can be attached to the same proxy... so, the proxy read/writes

and you have to manage your network accordingly so that this proxy can reach each remote (here goes the static routing or vpns...)

fcgo

@florent can you confirm for the replication job ?

Thank you

florent

@fcgo it goes from xapi (= pool/hosts) to xapi , and xapi to remote as of now, you can't chain proxy .

xapi call uses https, that means that replication accross sites uses https . If you do backup, and your backup is in the same site as your NFS, then the xapi<-> proxy path is in https, and the proxy <-> NFS will be local

that is one of the strength ( and complexity ) of xcp-ng infra : everything go through an API ( the xapi ) nobody access host/pool data directly from the outside

fcgo

@florent Hi,

Thank you.

And during backup which host is working hard during the vdi export to XOA (or to the proxy) : is it the host which contains the backed up VM, or is the master also doing some work (other than control channel calls) ? Through which host the network flow is going through ?

florent

@fcgo if the storage is shared : the export is done by one of the host of the pool
If the storage is not the export is done by the host with the storage . Same for the host receiving the data

The command channel, as you said, is always the master to the xoa (and eventually xo-proxy)

So for a replication :

[source SR] => source host =https export call=> xoa / xo-proxy =https import call=> target host => [target SR]

if the xoa is running on the host doing an export, it does not use the physical network
the network use between the host and SR is dependent on the storage used