I have found a work round for this, another problem i found is the host 2&3 server have wrong credentials setup so can't access them locally, so a rebuild on them will be starting shortly.

Thanks @olivierlambert i looked into the networking a bit more, I was able to get successful pings between the servers and more looking into the networking on these 2 servers I had an IP address but was showing disconnected, so i started to look at forcefully removing the unresponsive hosts and found the following article.

https://support.citrix.com/article/CTX126382/how-to-remove-a-xenserver-slave-when-it-no-longer-exists-in-the-pool

1. I ran this from the master (still working) to find the list of hosts.
   xe host-list

2. Get a list of all the VMs that are resident on the unresponsive host(s) to get their uuid's.
   xe vm-list resident-on=aaaaaaaaa-bbbb-cccc-ddddddddddd

3.Force powerstate reset for each of the VMs on the host(s).
xe vm-reset-powerstate uuid=eeeeeee-dddd-cccc-bbbb-aaaaaaaaaaaa --force

4. Force the removing of the host(s).
   xe host-forget uuid=aaaaaaaaa-bbbb-cccc-ddddddddddd

This has now allowed me to spin up the VMs on the working host while I get the other rebuilt.

Thanks you @olivierlambert and @Gheppy for advice and pointers for going some solution for this.

Hi,

I have suddenly come across a issue that I am guessing has appeared with a recent change in Ubuntu that is now preventing growpart in the cloud-init to resize the disk when spun up from template.

I do recall a post on here that i have saved within my notes as a reminder that something will go wrong at some point. They has provided detailed instructions to get cloud-init working with Ubuntu and finished the post with the following:

"This post will no longer lead to a working cloud-init deployment within 3-6 months based on my previous experience"
"You have to examine the default file you get line by line and work towards the values above until the template works properly."
(@FOHDEESHA, Vates - Pro Support Team)

I completed the following steps before converting to temaplate:

Install packages: cloud-init cloud-utils cloud-initramfs-growroot
Remove all or comment out from /etc/resolv.conf
Remove all but localhost from /etc/hosts

Remove any datasource_list or datasource blocks, and replace them with these values in /etc/cloud/cloud.cfg

#XOA Cloud-init Data Source
datasource_list: [ NoCloud, ConfigDrive ]
datasource:
    ConfigDrive:
        dsmode: local
    NoCloud:
        fs_label: cidata

Add network handling in /etc/cloud/cloud.cfg

#XOA Network Handling
manage_resolv_conf: true
manage_etc_hosts: true
preserve_hostname: false

Remove any of the old configs left by the Ubuntu install.

sudo rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg /etc/cloud/cloud.cfg.d/90_dpkg.cfg /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg

Clean up logs, history and remove any ssh keys.

sudo rm -rf /etc/ssh/ssh_host_*
sudo cloud-init clean --logs
sudo cat /dev/null > ~/.bash_history && history -c

Convert to template.

Cloud Config file is very basic, to try reduce the amount that could be going wrong to locate the isssue.

#cloud-config
hostname: {name}
fqdn: {name}.example.com 

growpart:
  mode: auto
  devices: ['/']
  ignore_growroot_disabled: false

packages:
  - nano
  - wget
  - unzip
  - tmux
  - cron

package_update: true
package_upgrade: true

final_message: "Setup complete in $UPTIME seconds, please allow two minuets for DNS records to update!"

My template is build on a 16GB disk, as the point of VM creation I set the disk to 128GB.
running lsblk shows the 128GB is there.

NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sr0                        11:0    1 1024M  0 rom
xvda                      202:0    0  128G  0 disk
├─xvda1                   202:1    0    1M  0 part
├─xvda2                   202:2    0  1.8G  0 part /boot
└─xvda3                   202:3    0 14.2G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0 14.2G  0 lvm  /

as you can see my / mount is still at the 14GB. Running vgdisplay and df -h both do not show the 128GB disk and has also preventing me from resizing.

 --- Volume group ---
  VG Name               ubuntu-vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <14.25 GiB
  PE Size               4.00 MiB
  Total PE              3647
  Alloc PE / Size       3647 / <14.25 GiB
  Free  PE / Size       0 / 0
  VG UUID               NZTIlD-5YHf-eUf1-xCzL-RMX7-BI72-HG200D

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              389M  744K  388M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   14G  4.5G  8.9G  34% /
tmpfs                              1.9G     0  1.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              1.9G     0  1.9G   0% /run/qemu
/dev/xvda2                         1.7G  252M  1.4G  16% /boot
tmpfs                              389M  4.0K  389M   1% /run/user/1000

Has anyone come across this issue, i feel this is something new that has caused this with a update in Ubuntu as this has been working.

@olivierlambert Yer, i've just been looking at clock speeds, and they are more powerful but a big jump in price, but something for me to look into, thank you for you help!

@olivierlambert Hi Thanks for coming back to me with that, I've not had much dealing with AMD within the server world so Intel is possible what I'd like to stick with in that case, storage will be SAS apposed to any networking solution for DB performance.

Are there any clear difference for between the 2 CPUs?

Hi,

I'm in the process of getting specs for some new servers and i've been quoted for the following:

2x Dell R760 - Intel Xenon-Platinum 8468v 48C/96T
1x Dell ME5024 Storage Array - SAS

Checking on the XenServer HCL i can't see the server listed.
https://hcl.xenserver.com/servers/?serversupport__version=21&vendor=3&form_factor=2

I'd like to think i'm 99% sure xcp-ng would be fine and i'm guessing it's just not been though a round of testing or am i best of dropping to a R750 which would reduce me down to a max of 40C/80T CPU?

Thanks in advanced.

I have found a work round for this, another problem i found is the host 2&3 server have wrong credentials setup so can't access them locally, so a rebuild on them will be starting shortly.

Thanks @olivierlambert i looked into the networking a bit more, I was able to get successful pings between the servers and more looking into the networking on these 2 servers I had an IP address but was showing disconnected, so i started to look at forcefully removing the unresponsive hosts and found the following article.

https://support.citrix.com/article/CTX126382/how-to-remove-a-xenserver-slave-when-it-no-longer-exists-in-the-pool

1. I ran this from the master (still working) to find the list of hosts.
   xe host-list

2. Get a list of all the VMs that are resident on the unresponsive host(s) to get their uuid's.
   xe vm-list resident-on=aaaaaaaaa-bbbb-cccc-ddddddddddd

3.Force powerstate reset for each of the VMs on the host(s).
xe vm-reset-powerstate uuid=eeeeeee-dddd-cccc-bbbb-aaaaaaaaaaaa --force

4. Force the removing of the host(s).
   xe host-forget uuid=aaaaaaaaa-bbbb-cccc-ddddddddddd

This has now allowed me to spin up the VMs on the working host while I get the other rebuilt.

Thanks you @olivierlambert and @Gheppy for advice and pointers for going some solution for this.

@olivierlambert I now have all 3 hosts back up:

• Host 1 Started VM it already had running on there
• Host 2 Shows enabled but no storage, shared or local
• Host 3 Shows enabled but no storage, shared or local

In the VM list i still get 6 VMs on host 2 in Running / Paused state and host 3 shows 3 VMs running.

Is there a way to clear out the VM states?

@olivierlambert I've just restarted all hosts to bring them back up one by one, I still had this issue with all the host powered up and showing as available. I'll let you know where i'm at once i've got them booted back up.

@olivierlambert there looks to be a disconnect from what XOA is showing and the real world, currntly i have VMs showing as running on Host 2 & 3 and showing the hosts are online but they are physically powered down.

Couple of the errors i've come across when trying to power off a VM that paused or showing as running.

host.restart
{
  "id": "4f969212-d381-40e5-b19a-862395faa1e3",
  "force": false
}
{
  "code": "VM_BAD_POWER_STATE",
  "params": [
    "OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9",
    "running",
    "paused"
  ],
  "task": {
    "uuid": "d1db6e59-b65e-8daa-5a5f-2e73c8802ab1",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20221010T22:56:05Z",
    "finished": "20221010T22:56:05Z",
    "status": "failure",
    "resident_on": "OpaqueRef:53a20064-bf98-467f-af13-0491a672b3a0",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_BAD_POWER_STATE",
      "OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9",
      "running",
      "paused"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 560))((process xapi)(filename hashtbl.ml)(line 266))((process xapi)(filename hashtbl.ml)(line 272))((process xapi)(filename hashtbl.ml)(line 277))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 556))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "VM_BAD_POWER_STATE(OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9, running, paused)",
  "name": "XapiError",
  "stack": "XapiError: VM_BAD_POWER_STATE(OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9, running, paused)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:954:24)
    at forEach (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:988:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:978:12)
    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:1144:14)"
}

vm.start
{
  "id": "db3f5735-7f21-63c2-2c6b-5c60db8cd9df",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "53e2f76a-4200-8ed1-64f3-282af7b44351",
    "name_label": "Async.VM.unpause",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20221010T22:02:49Z",
    "finished": "20221010T22:02:50Z",
    "status": "failure",
    "resident_on": "OpaqueRef:53a20064-bf98-467f-af13-0491a672b3a0",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 6065))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:954:24)
    at forEach (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:988:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:978:12)
    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:1144:14)"
}

@olivierlambert here is the error that i manged to get from connecting one of the storage repos:

pbd.connect
{
  "id": "948638ac-2edd-5493-4354-84adb0345891"
}
{
  "code": "CANNOT_CONTACT_HOST",
  "params": [
    "OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a"
  ],
  "task": {
    "uuid": "f2100dad-3c79-3d03-89d8-647c6e0e549f",
    "name_label": "Async.PBD.plug",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20221010T22:42:46Z",
    "finished": "20221010T22:43:24Z",
    "status": "failure",
    "resident_on": "OpaqueRef:53a20064-bf98-467f-af13-0491a672b3a0",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "CANNOT_CONTACT_HOST",
      "OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 138))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "CANNOT_CONTACT_HOST(OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_CONTACT_HOST(OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:954:24)
    at forEach (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:988:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:978:12)
    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:1144:14)"
}

Hi @olivierlambert
It doesn't give a error just hangs.