Did my production this morning... Not a smooth upgrade this time.

The master took so long that the process timed out, I can't tell too much more because I was doing other work and was doing this remotely.

I handled the next host manually, and that took so much time I gathered my hearing protection and went over to the rack. It was stuck in a reboot phase but hadn't shut down for about 20 minutes. Held my finger on the power button. booted back up with a couple reboots in the process and it finally was ready.

Moved the VMs off the third host, manually triggered the update, and sat and watched looking at the VGA output to see what was happening. The reboot phase saw the XCP-ng animation progress all the way to an empty bar, but sat there another 5-10 minutes until I again held my finger on the power button to shut it off. Power up and after a bit of time it was ready again. All in all, updating three hosts took me around 2 hours today.

Here's the one condition that I think could contribute to this issue:

I have two NAS normally connected, an ISO and NFS connection on each. One of the servers is powered down for construction, but I did not disconnect it from the hosts. Could this severed connection be the reason why my updates took so long, something around not being able to purge or drain the state before the reboot?

I've disconnected those SR from the hosts, and I'll probably do a rolling pool reboot later today or next week and see if things go better.

And all that said, my XO sources is not happy with this update. XO6 isn't grabbing the data so dashboards are blank or take a long time with spinning wheels gathering the data. XO5 is immediate. One of my XO sources updated this morning, the other was yesterday so they should be pretty close to current.

@Rod-G

These have 9.4.2-xxxx and I need to go through and update everything now that I see how far behind I am.

the second VM completed fine after its reboot, something was just stuck in a dirty state and got in the way of the import or health check.

I didn't wait long enough for XO to catch up, the manual backup I did after rebooting the VM worked, I'll hit the other VM before I start some other work and see what I see. If I don't respond back toady, then it was a "dirty" VM because of some odd reboot condition. I've just never seen that before and odd that it happened 8 days after installing MS updates.

Both VMs had a reboot condition that they needed, but even after reboot I have a fail.

I watched this one, snapshot and transfer all went fine, during the import stage it only got to around 30% and then failed out and all the IO dropped off.

These are the only two VMs that I can't backup with compression due to the large amount of empty space on the disk. It's an old problem and I'm not sure it has been fixed. I may try setting zstd after lunch and trying again. If that doesn't work, it's a tomorrow problem.

@Danp

Yes, I'm doing a health check on each backup job. I updated XO sources but it looks like the manual backup I started is going to fail, it is almost at 30 minutes for a 22 minute process. I'll reboot the VMs next and see if there is just something dirty in the state that needs to be cleaned up.

I have 2 VMs that are now failing and have been for the last two days, the three day old worked. Just sorting through it now.

Both VMs are "bigger" than the rest, both have a disk that are 500gb and more than a lot of empty space in them. Compression is off, no updates have been performed to the XCP hosts for a while, and MS updates were applied 8 days ago. Running Xenserver 9.4.2 agent and drivers, same as the VMs that are still working. There are no snapshots for any of the VMs.

XO sources with Xen Orchestra, commit c5b7e and Master, commit e3a58 currently 31 commits behind so I guess I'll be updating it to see if something got changed.

The only error listed is a timeout error. Going to reboot and try manually running them again to see what happens. Including the the error report

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1779369775885",
  "jobId": "05f70792-9624-4150-9597-5917c045bea6",
  "jobName": "DC2-backup",
  "message": "backup",
  "scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9",
  "start": 1779369775885,
  "status": "failure",
  "tasks": [
    {
      "id": "0mpfitnx5-n8d213qn7md",
      "start": 1779369778265,
      "status": "failure",
      "tasks": [
        {
          "id": "0mpfitogj-nllocq70f6q",
          "start": 1779369778963,
          "status": "success",
          "end": 1779369780410,
          "result": "4353adcc-9de6-1ff1-8172-f600c7e0cb4c",
          "message": "snapshot"
        },
        {
          "id": "0mpfitpl7-j4loeu6sjhj",
          "start": 1779369780427,
          "status": "success",
          "tasks": [
            {
              "id": "0mpfitplr-condtv8ouhw",
              "start": 1779369780447,
              "status": "success",
              "end": 1779370618949,
              "result": {
                "size": 151409389568
              },
              "message": "transfer"
            }
          ],
          "end": 1779370618976,
          "message": "export",
          "data": {
            "id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52",
            "type": "remote",
            "isFull": true
          }
        },
        {
          "id": "0mpfjbp3d-wpp55k5n2b",
          "start": 1779370619593,
          "status": "failure",
          "tasks": [
            {
              "id": "0mpfjbp3i-c4eg55kbkjn",
              "start": 1779370619598,
              "status": "success",
              "end": 1779371188852,
              "result": {
                "size": 151409389568,
                "id": "e6c3c4c5-f272-a475-8f7a-d9715c257557"
              },
              "message": "transfer"
            },
            {
              "id": "0mpfjnwc5-sqxh8e4sfsr",
              "start": 1779371188853,
              "status": "failure",
              "end": 1779371789013,
              "result": {
                "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
                "name": "Error",
                "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
              },
              "message": "vmstart"
            }
          ],
          "end": 1779371792327,
          "result": {
            "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
            "name": "Error",
            "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
          },
          "message": "health check"
        },
        {
          "id": "0mpfk0tzk-vg8hq80to3g",
          "start": 1779371792336,
          "status": "success",
          "end": 1779371792896,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        }
      ],
      "end": 1779371792897,
      "result": {
        "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
        "name": "Error",
        "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
      },
      "message": "backup VM",
      "data": {
        "id": "1c163183-d863-f8cc-159a-5c5281aafee3",
        "type": "VM",
        "name_label": "BMC-DC2"
      }
    }
  ],
  "end": 1779371792898,
  "infos": [
    {
      "data": {
        "vms": [
          "1c163183-d863-f8cc-159a-5c5281aafee3"
        ]
      },
      "message": "vms"
    }
  ]
}

The above suggests the drivers are a problem, Xenserver has 9.6.0 available which requires removing current, then install new. I'll have to plan that upgrade to make sure the AD VMs stay functional, probably a job for tomorrow.

@Andrew

Thanks, my updates and my RPU are normally at different times. I also don't backup multiple times per day like I probably should.

@Andrew

Why did you need to disable backups before applying this?

@MajorP93

Before the LTS release UEFI worked extremely well, something kind of went sideways when it hit release. I only have a single UEFI VM, it was created under version 8.2 and does not give me trouble.

I guess I need to get my lab back up, had a UPS fail and shut everything off a few days ago. Then I need to test UEFI VMs and see what I can see. But back when it went LTS I reinstalled from scratch and tried to do everything through XO-lite to get the first VM running and found that Debian 13 or Windows Server 2022 would not boot if I created them with UEFI. Since then I haven't tested this again.

I'm also tempted to just move my whole lab to version 9, waiting for the next ISO to land and I may do this. I can run real testing workloads on Harvester for the time being and maybe play around migrating back and forth between both systems.

@escape222

Maybe try again as a BIOS boot VM, but the times for BIOS boot are nearing an end, so many things have climbed on UEFI only these days that a problem like this is going to be hard to ignore.

@escape222

I assume you made it UEFI? I had the same problem with XO-lite about a year ago. If I created it as bios boot it worked as expected.

@Bytevenidos

That's an interesting way of doing this, I'll have to remember it. I think the Fuel Ignition tool has a vhd output, it also produces both ignition and combustion parts for you. Worked pretty good on one test machine, but I need to go deeper into some of the other choices you can type in.

This is mostly a thought exercise for discussion, and maybe you've already thought about all this and made choices based on performance or other metrics.

Since version 9 is being designed, and it's being designed around Alma 10 as a base, would it make sense to revisit the HCI options?

Everyone seems to scream for Rook/Ceph. I've never used it so not sure.

Some people want Kubernetes or some container system installed.

What about setting up Kubernetes (k3s or rke2) and using it to provide Longhorn as storage?

Do I know what this would take to get going? No, not yet. I've been working with Harvester which uses Longhorn V1 or experimentally Longhorn v2. I'm also finding out that you really need to have Kubernetes running and have Rancher running to manage more aspects of Harvester without going to yaml or Helm scripts. So I'm also diving down the must learn a bit of Kubernetes so I can get Rancher so I can use Harvester the way other platforms work. It's a rabbit hole.

Downsides of Longhorn or maybe just downsides of Harvester, can not use NFS "out of the box". You need to install the CSI driver for NFS, and then it is only for backups, you can also install S3 and do the same. Longhorn is also fairly slow, even going across a 25gbps connection to an nvme. The local performance of this nvme has been tested to 3GBps with ESXi, so not sure why I can't get faster, but not the point of this discussion.

Just though I would bring up another choice that I've never seen discussed, and throw it to the winds for consideration going forward. V9 is so young that major changes could be accomplished if they made functional sense. I know you spent years working with Linbit to get what we have now, and I've not tried this either so hard to say anything good or bad about it.

Replying to an old thread because this one popped up in a search.

There is an open source and locally running ignition/combustion generator that I've used on bare metal. To use it with XCP-ng, we would need the ability to have a second "cd rom" drive during the install phase, one for the OS installer, one for the combustion/ignition ISO generated from the tool.

https://opensuse.github.io/fuel-ignition/

They say it only runs locally, no info is send out to the web. I didn't monitor my connection on the one time I used it, and probably won't the next time (coming soon to a bare metal cluster in my lab). For the record, I'm using the LEAP version of Micro for my tasks.

@AlexanderK

The ronivay script requires you to select an option (#2 to update).

I look at things this way, it's good to have more people working on scripts like this.

@acebmxer

I haven't tried this yet, but liking the menu you just showed!

How many of these are critical? I haven't even had time to apply the last round of patches to either lab or production.

In theory Harvester supports nested virtualization, but I haven't tried it yet. Might be worth setting up a single host to try it. I think Proxmox also supports it, again might be worth a single host if that would handle the load.

@ph7

Do you have compression turned on for the backups?

I have a couple vms that I don't compress because of the old error, I didn't update yet or Red enable compression to see what that status is.

@robertblissitt

Normal for the pool master to take a bit longer but never seen 15 minutes before.