XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. jubin3
    3. Posts
    J
    Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 4
    • Posts 12
    • Groups 0

    Posts

    Recent Best Controversial
    • Ubuntu 22.04 Cloud-init disk size issue

      Hi,

      I have suddenly come across a issue that I am guessing has appeared with a recent change in Ubuntu that is now preventing growpart in the cloud-init to resize the disk when spun up from template.

      I do recall a post on here that i have saved within my notes as a reminder that something will go wrong at some point. They has provided detailed instructions to get cloud-init working with Ubuntu and finished the post with the following:

      "This post will no longer lead to a working cloud-init deployment within 3-6 months based on my previous experience"
      "You have to examine the default file you get line by line and work towards the values above until the template works properly."
      (@FOHDEESHA, Vates - Pro Support Team)

      I completed the following steps before converting to temaplate:

      Install packages: cloud-init cloud-utils cloud-initramfs-growroot
      Remove all or comment out from /etc/resolv.conf
      Remove all but localhost from /etc/hosts

      Remove any datasource_list or datasource blocks, and replace them with these values in /etc/cloud/cloud.cfg

      #XOA Cloud-init Data Source
      datasource_list: [ NoCloud, ConfigDrive ]
      datasource:
          ConfigDrive:
              dsmode: local
          NoCloud:
              fs_label: cidata
      

      Add network handling in /etc/cloud/cloud.cfg

      #XOA Network Handling
      manage_resolv_conf: true
      manage_etc_hosts: true
      preserve_hostname: false
      

      Remove any of the old configs left by the Ubuntu install.

      sudo rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg /etc/cloud/cloud.cfg.d/90_dpkg.cfg /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
      

      Clean up logs, history and remove any ssh keys.

      sudo rm -rf /etc/ssh/ssh_host_*
      sudo cloud-init clean --logs
      sudo cat /dev/null > ~/.bash_history && history -c
      

      Convert to template.

      Cloud Config file is very basic, to try reduce the amount that could be going wrong to locate the isssue.

      #cloud-config
      hostname: {name}
      fqdn: {name}.example.com 
      
      growpart:
        mode: auto
        devices: ['/']
        ignore_growroot_disabled: false
      
      packages:
        - nano
        - wget
        - unzip
        - tmux
        - cron
      
      package_update: true
      package_upgrade: true
      
      final_message: "Setup complete in $UPTIME seconds, please allow two minuets for DNS records to update!"
      

      My template is build on a 16GB disk, as the point of VM creation I set the disk to 128GB.
      running lsblk shows the 128GB is there.

      NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
      sr0                        11:0    1 1024M  0 rom
      xvda                      202:0    0  128G  0 disk
      ├─xvda1                   202:1    0    1M  0 part
      ├─xvda2                   202:2    0  1.8G  0 part /boot
      └─xvda3                   202:3    0 14.2G  0 part
        └─ubuntu--vg-ubuntu--lv 253:0    0 14.2G  0 lvm  /
      

      as you can see my / mount is still at the 14GB. Running vgdisplay and df -h both do not show the 128GB disk and has also preventing me from resizing.

       --- Volume group ---
        VG Name               ubuntu-vg
        System ID
        Format                lvm2
        Metadata Areas        1
        Metadata Sequence No  4
        VG Access             read/write
        VG Status             resizable
        MAX LV                0
        Cur LV                1
        Open LV               1
        Max PV                0
        Cur PV                1
        Act PV                1
        VG Size               <14.25 GiB
        PE Size               4.00 MiB
        Total PE              3647
        Alloc PE / Size       3647 / <14.25 GiB
        Free  PE / Size       0 / 0
        VG UUID               NZTIlD-5YHf-eUf1-xCzL-RMX7-BI72-HG200D
      
      Filesystem                         Size  Used Avail Use% Mounted on
      tmpfs                              389M  744K  388M   1% /run
      /dev/mapper/ubuntu--vg-ubuntu--lv   14G  4.5G  8.9G  34% /
      tmpfs                              1.9G     0  1.9G   0% /dev/shm
      tmpfs                              5.0M     0  5.0M   0% /run/lock
      tmpfs                              1.9G     0  1.9G   0% /run/qemu
      /dev/xvda2                         1.7G  252M  1.4G  16% /boot
      tmpfs                              389M  4.0K  389M   1% /run/user/1000
      

      Has anyone come across this issue, i feel this is something new that has caused this with a update in Ubuntu as this has been working.

      posted in Advanced features
      J
      jubin3
    • RE: Dell R760 Servers Compatable?

      @olivierlambert Yer, i've just been looking at clock speeds, and they are more powerful but a big jump in price, but something for me to look into, thank you for you help!

      posted in Hardware
      J
      jubin3
    • RE: Dell R760 Servers Compatable?

      @olivierlambert Hi Thanks for coming back to me with that, I've not had much dealing with AMD within the server world so Intel is possible what I'd like to stick with in that case, storage will be SAS apposed to any networking solution for DB performance.

      Are there any clear difference for between the 2 CPUs?

      posted in Hardware
      J
      jubin3
    • Dell R760 Servers Compatable?

      Hi,

      I'm in the process of getting specs for some new servers and i've been quoted for the following:

      2x Dell R760 - Intel Xenon-Platinum 8468v 48C/96T
      1x Dell ME5024 Storage Array - SAS

      Checking on the XenServer HCL i can't see the server listed.
      https://hcl.xenserver.com/servers/?serversupport__version=21&vendor=3&form_factor=2

      I'd like to think i'm 99% sure xcp-ng would be fine and i'm guessing it's just not been though a round of testing or am i best of dropping to a R750 which would reduce me down to a max of 40C/80T CPU?

      Thanks in advanced.

      posted in Hardware
      J
      jubin3
    • RE: Storage not connecting, VMs stuck in state and unable to get back.

      I have found a work round for this, another problem i found is the host 2&3 server have wrong credentials setup so can't access them locally, so a rebuild on them will be starting shortly.

      Thanks @olivierlambert i looked into the networking a bit more, I was able to get successful pings between the servers and more looking into the networking on these 2 servers I had an IP address but was showing disconnected, so i started to look at forcefully removing the unresponsive hosts and found the following article.

      https://support.citrix.com/article/CTX126382/how-to-remove-a-xenserver-slave-when-it-no-longer-exists-in-the-pool

      1. I ran this from the master (still working) to find the list of hosts.
        xe host-list

      2. Get a list of all the VMs that are resident on the unresponsive host(s) to get their uuid's.
        xe vm-list resident-on=aaaaaaaaa-bbbb-cccc-ddddddddddd

      3.Force powerstate reset for each of the VMs on the host(s).
      xe vm-reset-powerstate uuid=eeeeeee-dddd-cccc-bbbb-aaaaaaaaaaaa --force

      1. Force the removing of the host(s).
        xe host-forget uuid=aaaaaaaaa-bbbb-cccc-ddddddddddd

      This has now allowed me to spin up the VMs on the working host while I get the other rebuilt.

      Thanks you @olivierlambert and @Gheppy for advice and pointers for going some solution for this.

      posted in Xen Orchestra
      J
      jubin3
    • RE: Storage not connecting, VMs stuck in state and unable to get back.

      @olivierlambert I now have all 3 hosts back up:

      • Host 1 Started VM it already had running on there
      • Host 2 Shows enabled but no storage, shared or local
      • Host 3 Shows enabled but no storage, shared or local

      In the VM list i still get 6 VMs on host 2 in Running / Paused state and host 3 shows 3 VMs running.

      Is there a way to clear out the VM states?

      posted in Xen Orchestra
      J
      jubin3
    • RE: Storage not connecting, VMs stuck in state and unable to get back.

      @olivierlambert I've just restarted all hosts to bring them back up one by one, I still had this issue with all the host powered up and showing as available. I'll let you know where i'm at once i've got them booted back up.

      posted in Xen Orchestra
      J
      jubin3
    • RE: Storage not connecting, VMs stuck in state and unable to get back.

      @olivierlambert there looks to be a disconnect from what XOA is showing and the real world, currntly i have VMs showing as running on Host 2 & 3 and showing the hosts are online but they are physically powered down.

      posted in Xen Orchestra
      J
      jubin3
    • RE: Storage not connecting, VMs stuck in state and unable to get back.

      Couple of the errors i've come across when trying to power off a VM that paused or showing as running.

      host.restart
      {
        "id": "4f969212-d381-40e5-b19a-862395faa1e3",
        "force": false
      }
      {
        "code": "VM_BAD_POWER_STATE",
        "params": [
          "OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9",
          "running",
          "paused"
        ],
        "task": {
          "uuid": "d1db6e59-b65e-8daa-5a5f-2e73c8802ab1",
          "name_label": "Async.host.evacuate",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20221010T22:56:05Z",
          "finished": "20221010T22:56:05Z",
          "status": "failure",
          "resident_on": "OpaqueRef:53a20064-bf98-467f-af13-0491a672b3a0",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_BAD_POWER_STATE",
            "OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9",
            "running",
            "paused"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 560))((process xapi)(filename hashtbl.ml)(line 266))((process xapi)(filename hashtbl.ml)(line 272))((process xapi)(filename hashtbl.ml)(line 277))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 556))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
        },
        "message": "VM_BAD_POWER_STATE(OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9, running, paused)",
        "name": "XapiError",
        "stack": "XapiError: VM_BAD_POWER_STATE(OpaqueRef:2c1a2394-0413-4fae-bedb-30c93c0e5cc9, running, paused)
          at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_XapiError.js:16:12)
          at _default (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_getTaskResult.js:11:29)
          at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:954:24)
          at forEach (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:988:14)
          at Array.forEach (<anonymous>)
          at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:978:12)
          at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:1144:14)"
      }
      
      vm.start
      {
        "id": "db3f5735-7f21-63c2-2c6b-5c60db8cd9df",
        "bypassMacAddressesCheck": false,
        "force": false
      }
      {
        "code": "INTERNAL_ERROR",
        "params": [
          "Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd"
        ],
        "task": {
          "uuid": "53e2f76a-4200-8ed1-64f3-282af7b44351",
          "name_label": "Async.VM.unpause",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20221010T22:02:49Z",
          "finished": "20221010T22:02:50Z",
          "status": "failure",
          "resident_on": "OpaqueRef:53a20064-bf98-467f-af13-0491a672b3a0",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "INTERNAL_ERROR",
            "Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 6065))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
        },
        "message": "INTERNAL_ERROR(Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd)",
        "name": "XapiError",
        "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id db3f5735-7f21-63c2-2c6b-5c60db8cd9df/config does not exist in xenopsd)
          at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_XapiError.js:16:12)
          at _default (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_getTaskResult.js:11:29)
          at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:954:24)
          at forEach (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:988:14)
          at Array.forEach (<anonymous>)
          at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:978:12)
          at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:1144:14)"
      }
      

      @olivierlambert here is the error that i manged to get from connecting one of the storage repos:

      pbd.connect
      {
        "id": "948638ac-2edd-5493-4354-84adb0345891"
      }
      {
        "code": "CANNOT_CONTACT_HOST",
        "params": [
          "OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a"
        ],
        "task": {
          "uuid": "f2100dad-3c79-3d03-89d8-647c6e0e549f",
          "name_label": "Async.PBD.plug",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20221010T22:42:46Z",
          "finished": "20221010T22:43:24Z",
          "status": "failure",
          "resident_on": "OpaqueRef:53a20064-bf98-467f-af13-0491a672b3a0",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "CANNOT_CONTACT_HOST",
            "OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 138))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
        },
        "message": "CANNOT_CONTACT_HOST(OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a)",
        "name": "XapiError",
        "stack": "XapiError: CANNOT_CONTACT_HOST(OpaqueRef:5c091131-ad50-4f5d-b1f2-2d6e4f1a776a)
          at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_XapiError.js:16:12)
          at _default (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/_getTaskResult.js:11:29)
          at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:954:24)
          at forEach (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:988:14)
          at Array.forEach (<anonymous>)
          at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:978:12)
          at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202209081641/packages/xen-api/src/index.js:1144:14)"
      }
      
      posted in Xen Orchestra
      J
      jubin3
    • RE: Storage not connecting, VMs stuck in state and unable to get back.

      Hi @olivierlambert
      It doesn't give a error just hangs.

      c58a8f01-59bb-4e46-8ed7-6d0605ad9c4d-image.png

      posted in Xen Orchestra
      J
      jubin3
    • Storage not connecting, VMs stuck in state and unable to get back.

      Hi all,

      Quick server setup:
      3x Dell r650
      1x Dell ME5 (iscsi)
      Running thought a pair of FS Switches on 10GB and 25GB

      Had a network drop and lost connection to the storage unit across all 3 servers, after powering all 3 down and bringing them back up 1 at a time, i'm now stuck here 2 host wont connect to any storage (not even it's local) and VMs showing as running on a host which has no storage.
      36914af1-914f-4aa6-bff8-6438cd6937b0-image.png

      b2a00c12-e7da-4b8c-af1b-bc287fd433d1-image.png

      Things I've tried:

      • Force shutdown
      • Migrate (Host 1 - Master is connected to storage)
      • Restart Toolstack on all hosts
      • Reconnect storage to the hosts.
      • Running xe vm-reset-powerstate uuid=uuid force=true
        (get This operation cannot be completed because the server is still live.)
        xe vm-reset-powerstate uuid=uuid force=true
      • Running xe vm-shutdown vm="Servername" --force
        (hangs untill i restart toolstack)

      I have tried restarting the 2 problematic servers and in xoa i have tasks sat there for "server_init" again until i restart the tool stack.

      Any help on this will be massively appreciated, i've always been able to wriggle my way though most of the issues i've come across just not this one.

      Thank you in advanced!

      posted in Xen Orchestra
      J
      jubin3
    • web-hooks Keeps giving errors

      Hi,

      I'm trying to get web-hooks plugin working with Microsoft Teams and keep getting a "Code: -32000
      Message: unknown error from the peer" error message

      The URL i am using does work as it goes into the same channel as my two Grafana servers for alerts using the "Incoming Webhook" connection in Teams.

      Any pointers would be greatly appreciated.

      posted in Xen Orchestra
      J
      jubin3