@florent rest and xo-cli
tunnel ID sent by message

@florent okay disabled.
We have quite an extensive use of API calls too, all over the place
and websockets (to get VM console in in-house web app)

I don't know if this could have an impact

@florent not really, it is activated and I tried to fiddle a bit with it, why ?

@florent said in backup mail report says INTERRUPTED but it's not ?:

is the memory stable on the proxy ?

can't really say, as i reboot them everyday with my XOA

but when I do not reboot them, they dont seem to be so impacted as I need to reboot them.
I will detach them from the reboot job and will report RAM consumption on proxies

@florent do you have (heavy?) use of backup jobs on your XOA ?
we all noticed this because of correlation between backup jobs & memory growth

I do have all my backup jobs through XO PROXIES though, and XOA still explodes in RAM Usage...

@acebmxer @ronivay stumbled upon this too
https://github.com/ronivay/XenOrchestraInstallerUpdater#backup-proxy

Backup proxy
Proxy installation method is experimental, use at your own risk. Proxy installation from sources is not documented by Xen Orchestra team. Method used here is the outcome of trial and error.

Proxy source code will be edited slightly to disable license check which only works with official XOA and there is no documented or working procedure to bypass it properly (there used to be but not anymore)

@acebmxer Great job (I guess, will need to give it a try)
looked the script and seems pretty complete.

Do you plan to add XoProxy install/update too ?

@acebmxer as per your screenshots, we can guess you have daily backup jobs operated by the XOA
and the memory graph is significant of a reguler OOM kill (out of memory)

there is some kind of memory leak happening and the devs are investigating it in another thread in here

upgrading to 8GB or 16GB your XOA memory will take it more days before OOM happens, but it will happen

check this thread https://xcp-ng.org/forum/topic/11721/backup-mail-report-says-interrupted-but-it-s-not/48

lol your previous post answered my questions. other-config.

@DustyArmstrong I have hard times to understand how the VM is aware of the XOA it is/was attached to...

a VM resides on a host
a Host in in a pool
a Pool is attached to a (or many) XO/XOA

how the hell your VM is aware of its previous XO ?

@DustyArmstrong said in Detached VM Snapshots after Warm Migration:

I mainly just want to know the best method to wipe XO and start over so it can rebuild the database.

ha, to wipe XO , check the troubleshooting section of documentation here :
https://docs.xen-orchestra.com/troubleshooting#reset-configuration

it is a destructive command for your XO database ! 

@DustyArmstrong mmmm what do you mean by pool master and pool slave ?

You have one pool with 2 hosts ? one master and one slave ?
could you screenshot the HOME/Hosts page ?
and the SETTINGS/SERVERS page ?

@DustyArmstrong ho okay, so old XO is downed.

You could have 2 XO connected to the same pools, as long as they do not have same IP address.

beware, I do not tell you to do that, it is not best practice at all if you ever are in this situation, you must understand what could happen (detached snaps, licence problems, ...)

but I thought this was the case

in your case, did you do snaps/backups BEFORE reverting the XO config of old XO to new XO ?
as far as I understand, this could have get you detached snapshots (snapshots being on VMs but not initiated by the restored version of your current online XO)

or do I overthink this too much ?!

keep us informed if you manage to clear the situation, curious about it

@DustyArmstrong I have a corner case where I see these detached snapshots, related to backup.

I have a (remote) pool, the master is attached by a XOproxy on main XOA.
All snapshots done by remote XOA are seen as "detached" snapshots on the main XOA where the pool is a distant one.

Main XOA do not own backup jobs, so it is not aware of the snapshots, so mark them as detached.

I think this is your current situation, you have 2 XO servers, one doing snapshots, the other seeing them as detached.

@Danp ha thanks for the correction, I was so certain to have seen it, that I didn't check on my master

@DustyArmstrong on your slave host, do a

# cat /etc/xensource/pool.conf
slave:xxx.xxx.xxx.xxx

you should see IP address of the master. If not, correct it.
the master must be pingable and accessible from management of the slaves in order for the slaves to have correct network propagation.

you can try the command on MASTER host, you should see

master

if you corrected the file on slave host, reboot it, it should come back normally

@blueh2o said in Health check scripts. Where is the example?:

xen-orchestra/backups/docs/healthcheck example/wait30seconds.sh

that would be here https://github.com/vatesfr/xen-orchestra/tree/master/%40xen-orchestra/backups/docs

@DustinB if he has CR, he could stop it, Failover, patch primary host, and eventually failback or reverse CR job

should you not enter the ESX ip and not the vCenter IP to migrate ?

give it a try

@DustinB I'd like to step in, from past experience

• avoid Broadcom nics
• my preference goes ton INTEL cpus, saw some threads on poor network perf with AMD (but I could be wrong on this one)
• avoid adaptec HBAs

Storage wise XO STOR would need 3 hosts to be compliant, so you should follow advice of shared storage like a NAS or iSCSI SAN.

Plan accordingly if you need multipathing to shared storage (no NFS) or if you need thin provisionning (no iSCSI - lvmoiscsi is thick)