Backup and Backup copy with XO Proxies. Bug or by design ?

[Pilow]

Hi,
I didn't find anyone with same problem as I encounter on the forums. So perhaps is it a new functionnality ?

Here's the deal. I have two pools : DC1 and DC2. Same XOA that manages the two pools.
Each pool host a minio VM to emulate S3 as a remote. So I have DC1-REMOTE dans DC2-REMOTE.
Each pool host an XOproxy, DC1-proxy and DC2-proxy

Backup of VMs works flawlessly without the proxies involved.
From DC1 i can backup to DC1-Remote, even to DC2-Remote. And vice versa.
Backup copy (VM MIRROR BACKUP) is also okay, I can choose different target as SOURCE and DESTINATION remote.

BUT ... XOA has all the burden of the jobs, and transfers.

WITH the proxies involved, i can also backup flawlessly in either way between the remotes of each pools. This time, the proxies are involved in network transfer, XOA is sleeping. good, working as intended.

is this a new thing ? When you attach a REMOTE to a PROXY, this REMOTE becomes unaivailable in the target of the backup job without proxy use. The target remotes are filtered BY the selected proxy.
For a normal backup job, this is not a problem.

But for a backup COPY (ie. VM MIRROR BACKUP) this becomes a problem as SOURCE remote and TARGET remotes are filtered to the SAME remote ?! defeating the purpose of copying backups to another remote...

1- I want to be able to uses proxies in VM MIRROR backup jobs
My "backup copy" job will have different retention in term of numbers of backup points (on remote S3 i want 14 points, and only 7 on local S3)

2- I could add a second target on the main job, but this target will have same retention policy, so it's a nope.
(and even if second target is okay without proxies, with xoproxy involved i can only put ONE target, filtered by the proxy... u see what I mean ?)

3- I dont want XOA involved in backup network transfers, so need proxies (XOA is in another third pool, on another host)

Ideal world : attach proxy to remotes is OK, but list OTHER possible remotes as target too.
Or a way to define SOURCE proxy and DESTINATION proxy in VM MIRROR backups ?

Hope this is clear, and get your attention, thank you !

Halp me :')

4- attach multiple proxies to one remote ? this could solve my issue !

[jshiells]

@Pilow thats what we had todo..

we had to allow XOA AND ALL proxies access to ALL backup remotes.

[Pilow]

@jshiells but how do you do that ?

[jshiells]

@Pilow not easily. as an Telco/ISP we run massive multiple 40 to 100gb/s links across the country so data transfers from site to site is not an issue allowing us to have priv routed IP networks spanning anywhere. Much harder todo for SMB deployments.
may also involve adding more vnics and IP's to XOA and XOA-PROXY

SiteA:
XOA
proxy-A
Storage-A-SR
Backup-A-SR

priv network routing in both directions to:

SiteB:
proxy-B
Storage-B-SR
Backup-B-SR

so XOA, Proxy-A, Proxy-B ALL have access to:
Storage-A-SR
Backup-A-SR
Storage-B-SR
Backup-B-SR

sorry.. this is probably not helpfull for most people

[florent]

@Pilow a Backup job cannot mix proxy and non proxy remote
The goal of the proxy is to keep the data flow out of XO, mixing both will contradict this goal

since you seem to want to access a remote by the proxy and the xoa, you can attach the remote twice ( one to the proxy, the other to xoa) and chose it it's better to have one read locally and write remotely or the opposite

The only things that should be noted when using multiple xo/proxy to a remote is that you may have "lock already held" error if both are trying to read and write the same VM (but for mirror job, at worse, it means that the transfer will be done on next run

@jshiells you had the right idea (and you have one nice network)

[Pilow]

@florent I get your point.

I could duplicate the remotes, one with proxy, one without proxy

buuuuuuuuut... when i select a proxy, i only see ONE remote attached to it.
So it defeats the possibility to use proxies in a VM MIRROR job.

And my XOA will have to do the transfers, on remote without proxies

Screenshot WITH proxy : i'm forced on ONE remote

Screenshot WITHOUT proxy : I can see all my remotes (but XOA is the only involved)

I just want to do a VM MIRROR backup job, by proxies.
Can't do that because I am FORCED to have the same remote in SOURCE and TARGET... Mirroring to the same remote makes no sense.

[florent]

@Pilow can you show us you remote settings ?

let's say you have remoteproxy and remotexoa and you want to mirror the backup from remoteproxy to remotexoa

You create a second remote on the proxy, called remotexoafromproxy that point exactly to the same folder as remotexoa, but is set on the proxy

then you should be able to do a mirror from remoteproxy to remotexoafromproxy

You can do the same from proxy1 to proxy2 or xoa to proxy.

A backup job can only run on one VM , be it a proxy or a xoa. So this VM must have access to all the relevant remote

[Pilow]

@florent okaaaay, I was taking it the wrong way.

I managed to have one proxy to see two remotes as intended, thanks to your advice.
Need to do some routing (DC1 and DC2 are on two distant datacenters, separate subnets) to make each proxy see the good route to each distant remote.

Thank you !

[florent]

@Pilow nice, inform us if everything is ok

[Pilow]

@florent everything is okay now.
It's tedious to have some duplicate remotes (to be accessible from different XOproxies) but it is working.

side question !
the FOLDER param of a S3 Remote is defined on the BR itself...
Could you think of a way of putting this parameter also/inplaceof in the backup job definition ?

Say I have an S3 server, and want to differenciate the backup folders of different clients in an MSP solution based.
Today I have to create a BR by client for each job with subsequent folder on the SAME remote.

Would be easy to monitor globally the occupied space by client in different main client folders, than parsing the UUID of Jobs/VMs in a big BR full of all client VMs.

any idea if this could be implemented ? would be a great addon of functionnality for MSP/CSP partners using S3 BR like us

[florent]

@Pilow what would need to change a lot of things. We are starting a project to rewrite the ACL, and I think it's a better way to handle it. But it will take time before releasing it.

[Pilow]

okay we will dev the matching of vm/jobs UUIDs in the meantime to aggregate "backup size by clients"

or look after the api calls you do in XOA where you have the total backup size (by VM).

we lack the TAGS here by the way
we can tag VMs, but can't tag backups... neither filter VM backups by VM tag

[florent]

@Pilow This one should be doable, I am adding it to our backlog