Detached VM Snapshots after Warm Migration

DustyArmstrong

@Pilow Yes one pool, 2 servers (a master and a secondary/slave).

I think I've realised per my last post update, when I warm migrated I didn't select "delete source VM", which has probably broken something since I retired the old hosts afterwards.

I mainly just want to know the best method to wipe XO and start over so it can rebuild the database.

Pilow

@DustyArmstrong said in Detached VM Snapshots after Warm Migration:

I mainly just want to know the best method to wipe XO and start over so it can rebuild the database.

ha, to wipe XO , check the troubleshooting section of documentation here :
https://docs.xen-orchestra.com/troubleshooting#reset-configuration

it is a destructive command for your XO database !

DustyArmstrong

@Pilow Yeah I don't particularly want to, but in the absence of any alternatives! I'll give it a day in case anyone responds, otherwise I'll just wipe it - assuming it leaves the hosts untouched. I just need to do whatever needs to be done to re-associate the VM UUIDs and would hope a total rebuild of XO would do that.

acebmxer

@DustyArmstrong

Try to deploy XO from sources and connect host to that and see if problem persist.

I use this script for install - https://github.com/ronivay/XenOrchestraInstallerUpdater

Just dont do what i did and connect SR to seperate pools

DustyArmstrong

@acebmxer My container is built from the sources, only fairly recently. I may try rebuilding it with the latest commit and deploy completely new, if that would work. I'm just trying to avoid a situation where I end up with even more confusion in the database.

In the end, my ultimate goal is just to re-associate the VMs with XO and the wider XAPI database. Whatever method will achieve that should work here as I'm starting from scratch anyway - for the most part. Whatever way achieves that cleanly really, I'm just a bit unsure how best to do it as moving XO clearly caused me issues, need to understand how to rebuild the VM associations.

DustyArmstrong

Looks like I unfortunately blew my chance to do this cleanly.

Decided to plug in my old hosts again, just to see if XO felt the VM - despite being on a new host - was still associated with or on the old host. Turns out that was the right assumption, as when I snapshotted a problem VM, it did not have the health warning. Took that to mean if I removed the old VMs, the references would go with it, but alas, I should've snapshotted all of them before doing that.

Back to square 1, but now the one VM I did snapshot no longer has the health warning. Completely wiped XO, still getting the same problem. Kind of at a loss now.

I would still really appreciate any assistance.

DustyArmstrong

Well, scratch the help, I have fixed it.

Bad VM:

xe vm-param-list uuid={uuid} | grep "other-config"

other-config (MRW): xo:backup:schedule: one-time; xo:backup:vm: {uuid}; xo:backup:datetime: 20260214T18:13:03Z; xo:backup:job: 97665b6d-1aff-43ba-8afb-11c7455c16ff; xo:backup:sr: {uuid}; auto_poweron: true; base_template_name: Debian Buster 10;

Clean VM:

other-config (MRW): auto_poweron: true; base_template_name: Debian Buster 10; [...]

Stale references to my old backup job causing the VM to disassociate with the VDI chain. Removing the extra config resolved it. Simple when you know where to look.

xe vm-param-remove uuid={uuid} param-name=other-config param-key=xo:backup:job
xe vm-param-remove uuid={uuid}  param-name=other-config param-key=xo:backup:sr
xe vm-param-remove uuid={uuid}  param-name=other-config param-key=xo:backup:vm
xe vm-param-remove uuid={uuid}  param-name=other-config param-key=xo:backup:schedule
xe vm-param-remove uuid={uuid}  param-name=other-config param-key=xo:backup:datetime

VMs now snapshot clean.

Pilow

@DustyArmstrong I have hard times to understand how the VM is aware of the XOA it is/was attached to...

a VM resides on a host
a Host in in a pool
a Pool is attached to a (or many) XO/XOA

how the hell your VM is aware of its previous XO ?

Pilow

lol your previous post answered my questions. other-config.

DustyArmstrong

@Pilow Time for a drink, I think.

florent

nice catch
you now know the internal magic of the xo backup .

florent

@Pilow said in Detached VM Snapshots after Warm Migration:

@DustyArmstrong I have hard times to understand how the VM is aware of the XOA it is/was attached to...

a VM resides on a host
a Host in in a pool
a Pool is attached to a (or many) XO/XOA

how the hell your VM is aware of its previous XO ?

a running vm has a property resident_on exposed by xapi
the queries from pools are handled separatly, thus we can decorate all the object of a xapi with a $pool property
this is the server collection that track which server data we know, and which one are connected or not

Mots of the data lives in the xapi, that means that multiple XO can handle multiple pools. The exception are the SDN Controller ( best handled by only one XO) and the backups of the same VM.

DustyArmstrong

@florent Thanks, had to put the DFIR hat on.

May as well ask as I thought about a PR for this - would it be feasible/practical/desirable to allow this to be done from XO's UI? I don't know how much of an edge case this was for me, but being able to remove "other-config" data following a migration (e.g. you do what I did and want the VMs to start over independently on a new host) might be beneficial to others.

Obviously it would be quite destructive I imagine, if used inappropriately. Even just reporting those ghostly associations would be nice - again not sure of your overall design ethos so there may be good reasons why it's not a solid idea.

florent

I think at least showing them could be a good idea. this is planned ( but low priority)

DustyArmstrong

@florent I'd be interested in giving it a shot if you accept PRs, but if you already have it planned and would rather do it yourselves at a later date (and it's too low a priority to review a PR anyway) then that's fair enough. I'll be content enough knowing it's a thing to be aware of under the hood.

florent

@DustyArmstrong I think we won't do too much on XO5 for now, and for xo6 we are trying to do everything with a UX designer to ensure coherence, at least until we have a full working app

DustyArmstrong

@florent No problem, just thought it would be fun.

Thanks for your work anyway!