@dthenot Thanks a lot for the information. I did some more testing on my end and I've now noticed differences in handling VDI format across different SR types (local LVM and EXT). Should I expect even more differences across remote SR types like LVMoHBA or NFS or are these differences more like block based vs file system based SRs?

Any way, it looks like I have to consult the following keys:

• image-format
• vdi_type
• type

In my tests, local EXT SRs tend to have only one of them, while local LVM SRs tend to have first two or all three, depends which one was used when creating the VDI.

@dthenot Aaaaah, so it even depends on the SR type. That also explains the difference I'm seeing in my lab with 8.1 vs 8.3. The first one uses LVM backed SR and the other one uses file system (ext) backed SR.

I'll have to investigate this a little bit more.

Was image-format key added by you (XCP-ng team) and is specific to XCP-ng or was it used at any point in XenServer also?

Hi,

It has come to my attention that in recent times there was some change in how VDI format is tracked in XAPI database, especially with arrival of QCOW2 support, first in XenServer (GFS2 SRs only) and then in XCP-ng 8.3 (other SR types).

Currently, I have access to XCP-ng 8.1 and 8.3 (with latest updates) and I'm seeing these differences:

XCP-ng 8.1:

$ xe vdi-param-list uuid=SOMEUUID
...
               sm-config (MRO): vdi_type: vhd
...

XCP-ng 8.3:

$ xe vdi-param-list uuid=SOMEUUID
...
               sm-config (MRO): image-format: vhd
...

So it seems what was earlier vdi_type is now image-format key in the sm-config parameter of a VDI.

On the other hand, some people are reporting having both keys present in sm-config, e.g.:

...
               sm-config (MRO): image-format: vhd; vdi_type: vhd
...

Neither are well documented and I can't find any official info on what keys in sm-config are supported and what is their purpose. Maybe devs can shed some light on this?

To add to the confusion, official XenServer documentation mentiones a key called just type when creating raw VDIs:

xe vdi-create sr-uuid=sr-uuid type=user virtual-size=virtual-size \
        name-label=VDI name sm-config:type=raw

In my testing this transparently gets renamed to image-format in the latest version of XCP-ng.

The reason this bothers me is that I was asked to implement reporting of VDI format in my Ansible modules for XCP-ng but now I'm not sure how to do it across all of the supported versions of XenServer and XCP-ng. Currently it looks like I should try to get vdi_type and fallback to image-format if the first one is not found but I'm not certain if these keys are interchangable and have the same purpose or they just happen to sometimes have the same value. Also, ISO image VDIs seems to be missing any format indication.

Any info on this subject is well appreciated. Thanks.

@Tristis-Oris maybe a few comparative screenshots would help

@benapetr just a small notice, at least in 0.0.5 version. When adding an NFS ISO library SR, there is no choice for NFS version (v3 and v4) like in XCP-ng Center. The XAPI then defaults to v3 which did not work in my environment.

@hawkpro I believe it will be worse with continuous replication because your replica will be in a shut down state. When you decide to start it, you will have to shut down the original VM and start the replica. You will have a downtime during shut down and start up sequence.

Downtime during VM migration is a necessity so there is nothing unexpected there. All types of migrations require a VM to be suspended for some time (usually seconds) during the switchover from one host or SR to the other host or SR.

If you have extended downtimes of your VM migrations, then something is not quite right with your setup.

@hawkpro If you have multiple SRs, move the VM to some other SR.

@delacosta456 This is a limitation of Debian's "/etc/network/interfaces" network configuration mechanism. It quite old and static - in other words it just applies network configuration on manual networking service restart. There is no hot plug handling.

For something more advanced, you have to use netplan (like Ubuntu) and/or NetworkManager (some other distros)... or possibly systemd-networkd.

@nikade You can even see that Dom0 VM has 16 vCPUs and host has 40 CPUs, in my case anyway. So Dom0 sees just some chunk of host resources.

@nikade said in Unable to MIgrate VDI when host is low on free memory:

Could you please explain the difference between the term dom0 and host?

This is what I mean:

Dom0 is not "the host". It contains a management layer for the host but in reality it is just another VM, highly specialized one but stil just a VM. It does not see host's RAM as it's own.

@nikade This is not related to Dom0 RAM. It's related to the host RAM.

@hitechhillbilly I'd say this is normal if you are low on RAM and you are doing a live VDI migration. XCP-ng requires some amount of free RAM on the host to be able to live migrate the VDI. The larger the VDI, the more RAM is needed but exact sizing is unknown to me. I've encountered this error numerous times so I consider it common.

The way around this is to shutdown the VM and then migrate the VDI. RAM requirements in that case are much much lower.

@benapetr This looks phenomenal. Will have to give it a try.

This is a nice coincidence. Just the other day one of my younger colleagues boasted about making XCP-ng Center work under Wine... everything just works. It would save a number of folks having to have a Windows VM on stand by just for the XCP-ng Center. Oh boy, how he was disappointed when I showed him everything that does not work ... but with this, there is still hope.

@denis.grilli I understand... but my experience is that even with the default scanning interval the logs become the problem when you get in the range of tens of SRs, thousands of disks. MajorP93's infra is quite small so I believe there is something additional that is spamming the logs... or there is some additional trigger for SR scan.

Update: maybe the default value changed in recent versions?

@Pilow agreed. This shouldn't be the norm. auto-scan-interval=120 is not going to be good for everyone. The majority of people probably don't have any problem with the default value, even in larger deployments.

On the other hand, the real cause of the issue is still elusive.

@MajorP93 Amount of logging is directly proportional to the number of hosts, VMs, SRs and clients (Xen Orchestra, XCP-ng Center...). If you have a lot of those, it's rather normal to have huge logs.

Now, 5 hosts and 2 SRs does not seem to be much so I wouldn't expect you to have problems with huge logs. There could be something going on there. Try restarting your hosts to clear any stuck processes and internal tasks that could potentially spam the logs.

We started having problems with /var/log size when we got in a range of 15+ hosts, 10+ SRs and 1000+ VMs per pool. Unfortunately, log partition cannot be expanded as it is at the end of the disk, followed only by the swap. The workaround we did is to patch the installer to create a large 8GB log partition instead of standard 4GB. Of course, we had to reinstall all of our hosts.

@acebmxer rc in first column means "residual configuration". This means that the package is removed but there are some leftover configuration files so that, for example, when you reinstall the package at later time, the package will use preserved configuration. To remove residual configuration and package completely, use:

sudo apt purge xen-guest-agent

@acebmxer Great. These are some YAML basics. You should read more about it . Following AI instructions without understanding is not going to take you far.

@acebmxer said in Custom config / cloud-init:

...

network:
  version: 2
  ethernets:
    enX0:      # or whatever your interface name is
      dhcp4: false
      addresses: - 10.100.10.206/24
      gateway4: 10.100.10.254
      nameservers:
          addresses:
               - 10.100.10.254
               - 1.1.1.1

Address should be on the next line:

      addresses:
      - 10.100.10.206/24

Regarding 50-cloud-init.yaml, AI is lying .

@nuentes metadata in this context is just XAPI database. In other words, it only contains information about your VMs, SRs, networks, pools etc. It does not contain anything system level. It is not a backup of the host system.

As far as I know, but someone from Vates can confirm, metadata backup functionality in XO is based on XAPI pool-dump-database command:

xe pool-dump-database file-name=dump.xml

There is some info about it here:

https://docs.xenserver.com/en-us/xenserver/8/dr/backup.html

P.S. I guess metadata backup is also XML just like XAPI state file (database). I don't know why JSON came to my mind regarding metadata backup.