@acebmxer It's not pretty, but its failsafe. The proceedure looked like this in our case:

1. Disable HA in the "old pool"
2. Put a host in the "old pool" into maintenance mode
3. Reinstall that host and connect it to XOA and then patch it
4. Create a "new pool" from that host
5. Create a new LUN or NFS share in the SAN for "new pool" and attach it to "new pool"
6. Live migrate VM's over from "old pool" to "new pool"
7. Once you've freed up another host you repeate step 2 and 3 and then join that host to "new pool". It is important that you patch it before joining it to the pool, that is done by going to Settings -> Servers in XOA and connect to it manually.

And then just continue untill you're done. Live migration is pretty reliable now days, so this works pretty good and since we had 10G network its not taking as long as it used to do with 1G network.
We did this after a major incident on our primary production site where 2 out of 4 hosts in a pool "suddenly" lost their NIC's after updating them. Since then we never updated the pools again. Standalone hosts are fine tho, they never did this.

Luckily we had 2 other pools where we could migrate the VM's to, but we couldn't realy trust the updating after that.

@acebmxer did it ever finish? Did it work or was the vm shutdown'd?

This is a classic issue with XAPI, once you have hosts in a pool and the slave cannot reach the master it will go crazy. Never seen this issue with standalone hosts tho.

We usually had this issue when upgrading xenserver, so we simply stopped doing that and then never had any issues. We went to "new" versions by simply standing up a new pool and migrate all the vm's over to it

I have no clue, I usually back it up with XO, its the one called "XO config & pool metadata backup" and only had to restore it once.

@vague said in Bitten by XSA-474:

starting up database engine D:2a59ed50ccb5 failed with exception Xmlm.Error(2:317874, "malformed character stream")

My guess is that your xapi database has been corrupted, if you dont have a backup this will be a rough one.

To add some content I'll summarize what you wrote on IRC:

1. You got an error, it was not resolved.
2. You then decided to start upgrading your xcp installation
3. Now the database is corrupt, not sure it it was already or if it happend in step 2.

Cool, thnx for the info!

@marcoi said in How to Setup IPMI in XO:

curious how does this work, dont you need to put in a username and password/IP address for ipmi details. I havent looked at docs or tried this out, just wondering. I think on ESXI i had to setup the ipmi with such details before i could issue power on etc.

I am also curious, since I asume authentication would be required before you're allowed to read or write data from iDRAC, we have a read-only "statsuser" to poll from iDRAC with Observium.

But maybe there is some kind of "special" integrated pass-through authentication done since the ipmitool is being executed from the same host.

@florent no im using EXT local SR. I tried downloading a VMDK and it worked fine, so not really sure whats going on.

I also tried from an esxi 8 host within the same network, same thing, I could also download a vmdk from it.

I just migrated from 2x vmware vsan hosts and 1x vmware standalone to xcp-ng in my lab and every migration resulted in a stop of the source vm. It's not a big deal for me in our lab, but I had major issues with some VM's never "finished" the migration.

Xen Orchestra, commit b89c2
Master, commit b89c2

I noticed that after the task has timed out, the VM that was migrated is often able to start, even tho it was not "finished". Is this expected?

I just learned something new, thats awesome

@bvitnik Thanks for explaining, I was always under the impression that dom0 was the "host" so to say.

@bvitnik said in Unable to MIgrate VDI when host is low on free memory:

@nikade This is not related to Dom0 RAM. It's related to the host RAM.

Could you please explain the difference between the term dom0 and host?
I havent been using xcp-ng professionaly for a while, but back when we were using it dom0 actually represented the "host" and it was also running all the management tools and services, hence why it some times needed a bit more ram.

Not really sure if there's some kind of caching involved when it is handling the VHD, but we went from a 50/50 chance of succeeding with a vdi migration to something near 90% after giving dom0 16Gb ram. The only exception was bigger VDI's, those kept on failing.

@bvitnik said in Unable to MIgrate VDI when host is low on free memory:

@hitechhillbilly I'd say this is normal if you are low on RAM and you are doing a live VDI migration. XCP-ng requires some amount of free RAM on the host to be able to live migrate the VDI. The larger the VDI, the more RAM is needed but exact sizing is unknown to me. I've encountered this error numerous times so I consider it common.

The way around this is to shutdown the VM and then migrate the VDI. RAM requirements in that case are much much lower.

Yea, I suggested giving dom0 16Gb. That was our standard sizing back in the days and it worked pretty well.

@hitechhillbilly said in Unable to MIgrate VDI when host is low on free memory:

@nikade I am trying to move the VDI from one storage SR to another on the same host. This is not in a cluster. This is a standalone host.

Alright, check the /var/log/SMlog as well as /var/log/xensource to see whats going on, you'll find an error there.

You say you're trying to migrate a vdi, where are you trying to migrate it?
To a SR attached to your current host or to another host/pool?

If you're trying to migrate it to another host/pool this is something that i've seen as well when dom0 is not assigned enough RAM, we usually gave them 16Gg which seemed to resolved these kind of issues back when we were running 8.2.0.

I'm seeing something similar, not sure if its the same issue, but mine stops at 95% and just hangs there:

[12:18 sto-xcp1 ~]# xe task-list
uuid ( RO) : c1056d36-b195-056a-4121-e82d7fc851fb
name-label ( RO): [XO] Importing content into VDI [ESXI]DEBIAN 12 fiona.iextreme.org-flat.vmdk on SR Local storage
name-description ( RO):
status ( RO): pending
progress ( RO): 0.950

The nbdkit debug-log can be found here:
https://mirror2.iextreme.org/temp/stderr

Edit: some additional info

Xen Orchestra, commit 1640a
Master, commit 1640a

@Pilow alright, cool! that sounds pretty much like a standard solution, too much options or details usually just confuses the end users

Looks very promising!

Just a question, what does the customer care about what pool, host or SR the VM is deployed to? I mean that's normally nothing you get to choose at the other cloud providers i've tried.

Or is that something only available to "resellers" who might have to balance their customers?

without this annoying bug we had on XOA (@ or / character needs you to refresh console page otherwise you can't type anymore in the console)

Haha, yeah I always run in to that when trying to login in the console at some random machine with a generated password...

Looks good!
I can't overlook the fact that it is in french, I guess "others" will have english in their UI?