Topics created by stevewest15

@stevewest15 Thx for SMlog. I don't see anything suspicious 😞

BR & SR are vastly different, and not mounted by the same machine (XO vs XCP-ng). Happy to see it works now.

Thank you all! I was on an older release of XOCE. I haven't wanted to update as the last few times I did, it broke the nightly backups. So stuck with an older version of XOCE that seems to have been stable in our environment.

I updated XOCE to latest build:
Xen Orchestra, commit 32228
Master, commit 32228

I was able to manually update the second host (by running yum update). Then restarted the pool master and now I'm trying to vacate VMs from second host back to master so I can reboot the second host.

However the live migration seems to be taking much longer from the second host back to master. All VMs use shared storage connected via NFS to this pool. I was thinking it should just take a few minutes to do a live migrate but so far it's been 12+ minutes and it's only at 12%:

dca691d2-5be0-41f5-8669-92cfcf5175b6-image.png

The network I selected for the live migration is a 20Gbe (bond of two 10 Gbe).

UPDATE: The live migration progress sped up and it was completed in about 16 minutes instead of the 1hr it was estimating.

Indeed, XenCenter isn't a supported client for XCP-ng. To delete those, you need to use xe CLI. I don't remember exactly which part is used by XenCenter, might worth some community feedback or taking a look at the XenCenter code.

@Danp Deleting and then re-creating doesnt always work if there's another partition (swap) after.
I usually do this:

fdisk -l /dev/xvda (note the end blocks of the last partition under the End column)
fdisk /dev/xvda
n
accept the defaults except starting blocks (use next free block as starting block)
w
vgextend almalinux /dev/xvda3 (if thats the number you saw when creating the new partition)
lvresize /dev/template-vg/almalinux-root -l +100%FREE -r

@jimmymiller Thank you! I destroyed the old VM and created a new one and then installed the guest tools using your instructions. XO detects the guest tools as version 8.2.x:

83a24b19-060b-4e2d-8c90-675f3ffc0200-image.png

Thank you again!

SW

@julien-f Thank you! That appears to fix the error. Greatly appreciate your help and @olivierlambert.

First of all, sorry for hijacking this old topic but as it was so well documented, it made more sense to expand upon it then to try and recreate everything that was already here.

Second, I generally don't use XCP-ng Center. However, once coming across this post I did open it to verify that what I saw in this post was the same as what my XCP-ng Center was showing.

Third, I am on one of the latest community additions but I have seen this occur a couple of time on previous versions as well.

Okay with that out of the way, I now believe that I know what is causing this to happen...

When you have a VM with multiple VDI's in it and one of those VDI's is both large and fairly busy, if the following exists then XO will not show one or more of the smaller drives despite the hosts still functioning normally while working on that VM:

The virtual machine has multiple snapshots of all of the VDI's in it. (IE like after several backups) For some reason a bunch of the snapshots are consolidated at one. All the snapshots that were consolidated are removed from the VM and the snapshots of the VDI's have coalesced but the system is still working on coalescing the large VDI. If a backup process tries to work on this VM, it will create a snapshot that XO shows to stay connected to the smaller drives even though the backup is skipped because a new snapshot of the large VDI can't be created due to the older snapshots not being coalesced yet.

Once the large VDI has properly coalesced all it's snapshots, the hosts will continue to treat it as any other VM but that VDI never does show up in XO again.

The only solution that I have found to fix it is to turn off the VM and then create a copy of missing VDI or VDI's and use those copies to replace the missing VDI's.

I know that I am just a member of the community but I hope that this helps everyone to understand what is going when this happens to them. 😉

It happened to me a couple of times before I was able to peace together the whole puzzle.

+1 to @Andrew setup

@stevewest15 I would also increase your dom0 memory, 16 GB is already pretty much all being used and for migrating VMs, you need memory. Some of the resources are used for dealing with storage, whoch requires such resources. If the migration is slow, run iostat and see what resources are being used; there may be a storage bottleneck, as well.

Very likely. With an UX designer, we'll be sure to avoid those problems 😄

Thank you greatly appreciate your assistance as always!

@olivierlambert Thank you, I didn't know this. Just reading the docs about it.

Dunno it's @florent you should ask 😉

@stevewest15 I don't have any advice of the fiability of b2
Their design require us to make a minor modification to our upload service .

During a full xva backup to a S3 like service, each part uploaded is uploaded along its hash and that is used to control the integrity

Thank you! So I must be doing something wrong as I'm getting the following error when I add in the old NFS mount which has the backups:

dc65c765-1542-4c58-8082-fbb54e33f3c9-image.png

The /mnt/Tank/VMs mount contains the previous XO backups as well as some active VM storage:
680455d2-9559-4fe9-a8a2-127762d5b61b-image.png

FreeNAS has two 10Gbe nics which are isolated based on the type of traffic:

10.10.1.5 is for VM traffic

10.10.2.3 is for Backups/Migration traffic

I can confirm it's now working. Thank You! Health page loads fine w/ the latest commit.

@olivierlambert Thanks, I tried running fstrim but it returned "the discard operation is not supported" on this NFS SR mount. I also tried running on SSD mounts which do support TRIM and they all returned the same message about "discard...not supported."

The VM OS (CentOS 7.8) does support TRIM in kernel so not sure if xcp-ng or xenserver is the reason why the VM can't issue fstrim on the NFS SR mount or the SSD SR mounts.

85b94dd0-b129-45ec-9ee2-bc4b7dd92b5e-image.png

If migration failed, there's a reason. It's usually stable.

Obviously, because your hosts weren't in the pool, a different CPU model can be the problem (they are automatically adapted if they are in the same pool).