@acebmxer
I did mine tonight over a remote connection, only took about 20 minutes total for all three hosts. If it's taking much longer then there might be something to look at.

I also used the slow externally connected XO sources box, which is pretty common for me when doing updates, that way the XO can't "get lost" as it migrates around the system. The external XO only has a gigabit connection, an old HP T630 with either 4gb or maybe 8gb of ram. That device has been in use since my first lab over 5 years ago, and still seems valuable.

@acebmxer

I will recommend a local monitor you can attach during RPU to make sure you can see what's happening. You can also go into the IPMI console and watch.

I'm old school, I still have a KVM switch on my stuff and used it during a recent move from temporary building back to permanent home. XCP2 didn't start up as expected and I had some fooling around to do. The KVM is often quicker than the IPMI option for me.

Probably going to do this update in the afternoon when things slow down a little, didn't realize we had more high priority bugs to fix. Damn AI! Have a firewall update I need to get done, and need to check Windows for the Paint update (yes paint has an exploit).

Ok, so what's the scoop here? Safe to roll this out or wait a few days? I just saw the notice in my XO that there are updates and I don't want to wreck my system right now after fixing a few other "issues".

[edit] I see this is just the certificate update so pushing it out and listening to servers reboot.

OK, marking this solved... It was the phantom reboots causing issues because the tools were not fully installed, only thing I can't understand is why only 2 out of 3 Windows servers would quit. Any way, got the Xen Tools updated and a couple reboots later, everything is working again.

Also I can now report that a bug that was fixed maybe a year ago is confirmed to work in my system. The bug was backing up a Windows VM that had a lot of empty space in the drive (over 250GB free) and turning on compression. It would time out counting zeros because it thought the process wasn't doing work. Both of these VMs are now compressing with zstd, one of them is substantially smaller now, the other only shrunk about 20GB.

Because of the way Windows uses the drive, these are all full backups. The differential never seemed to be small because Windows does a lot of changes on the drive (there is a post about this somewhere in here from Olivier).

The UEFI and possibly SecureBoot requirements on a lot of stuff and going to turn a lot of old systems into garbage.

Also going to be an issue when more current stuff gets upgrade from 8.3 in BIOS mode to requiring UEFI for version 9. That's the boat I'm in for production.

Alma and openSuse may be a few of the last holdouts allowing v2 and BIOS boot.

Pretty sure I figured this out. I went to update the Xenserver tools, and the updater said the installer was busy. That meant the machine was in a "dirty condition" and couldn't work right. The dirty condition was the newest Xenserver tools needing a couple more reboots to get fully insatalled.

testing now and will have to wait on the DC until the network use get lighter this afternoon.

Running the second VM right now, but pretty sure it will now pass the health check. I'll update after I check the DC in a few hours.

@DustinB Everything I care about gets backed up and health checked each night, some things really should be several times a day, but 1 day is tollerable.

I'll have to get things updated after I deal with some php2shell stuff, that will need to come first. And now node.js in my XO machines.

I'm guessing this might be having updates applied, and now I need new Xen Tools. I have a feeling tomorrow will be filled with updating the tools and checking things because I bet all my backups fail tonight. And I bet all fail after they try to start.

@DustinB

Ok, generally I'm doing backups after everyone has gone home for the night, so changes to the DB should be minimal.

I'll have to look into this more deeply and maybe use the Server Backup tool to schedule both DC going forward. Used to use that when bare metal, and had to restore a couple times too.

@DustinB Hmmm... They have always completed in the past, but I'll admit that I haven't tried to restore one and see if it really works. too busy to perform the test the last couple years.

but now I know the time out is after the machine boots, and I've seen the console come up and wait longer than 4 minutes, so probably it is OK if it could connect to the other DC and compare the database.

@DustinB

Probably too long ago, mentioned it on my follow up post while you were posting.

Replying with an update. As usual, a manually trigger backup completes. 12 minutes to send to the remote, 10 minutes to bring it back. But with the domain controller, it look like I may need to update the management agent, it's taking a long time to get that running.

When a DC comes up, it reaches out to the network to check with it's peer and sync data while it was gone. During the health check the network is not there so it takes 5+ minutes to fully boot. it's now sitting waiting. took about 10 minutes total for the MA to report and then shut the health check down. Not sure if there is a better way to do this for a domain controller or not, mostly just a data point for other people that think their health check may be stuck.

Time to go fix the other VM that failed last night.

[edit] it said it failed again, though I watched the console for about 4 minutes and clearly it booted. I'll have to update the MA and drivers and see what happens. But it looks like 10 minutes waiting for the MA is too long now.

I'm getting an increasing number of random backup failures lately, and almost always at the same step. They almost always fail at the health check while waiting for the VM to start. The storage is NFS on a server that has a 10g connection, same as the production storage so it should be fine, not detecting issues with the storage used for backup so not sure what's going on. Here is the short log of one of these, storage reports no errors on the Truenas server.

XCP-NG and XO (sources) are fairly up to date, I think XO was updated about a week ago and XCP is current patch level. I had a different backup job that was routinely failing, but I think it was happening when a scub in the storage was happening. Changed the time and it was OK after that.

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1792559700001",
  "jobId": "05f70792-9624-4150-9597-5917c045bea6",
  "jobName": "DC2-backup",
  "message": "backup",
  "scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9",
  "start": 1792559700001,
  "status": "failure",
  "tasks": [
    {
      "id": "0mvhnriw7-cddb6rumww7",
      "start": 1792559702887,
      "status": "failure",
      "tasks": [
        {
          "id": "0mvhnrjkn-ibpty24qqpg",
          "start": 1792559703767,
          "status": "success",
          "end": 1792559705242,
          "result": "05a07f3b-2c62-13ef-3405-17166901cefb",
          "message": "snapshot"
        },
        {
          "id": "0mvhnrkq7-v610f7ynxw",
          "start": 1792559705263,
          "status": "success",
          "tasks": [
            {
              "id": "0mvhnrkrm-un28zyxt91k",
              "start": 1792559705314,
              "status": "success",
              "end": 1792560387346,
              "result": {
                "size": 153946460672
              },
              "message": "transfer",
              "data": {
                "progress": 0
              }
            }
          ],
          "end": 1792560387560,
          "message": "export",
          "data": {
            "id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52",
            "type": "remote",
            "isFull": true
          }
        },
        {
          "id": "0mvho67u3-ty880j4lxuk",
          "start": 1792560388395,
          "status": "failure",
          "tasks": [
            {
              "id": "0mvho67w9-okbayxed8k",
              "start": 1792560388473,
              "status": "success",
              "end": 1792560962736,
              "result": {
                "size": 153946460672,
                "id": "4ac820a4-133b-e314-a6fd-d1817051ccad",
                "liveMounts": []
              },
              "message": "transfer",
              "data": {
                "progress": 100
              }
            },
            {
              "id": "0mvhoij02-se35olnw9kf",
              "start": 1792560962738,
              "status": "failure",
              "end": 1792561562889,
              "result": {
                "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.",
                "name": "Error",
                "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)"
              },
              "message": "vmstart"
            }
          ],
          "end": 1792561565797,
          "result": {
            "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.",
            "name": "Error",
            "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)"
          },
          "message": "health check",
          "data": {
            "progress": 0
          }
        },
        {
          "id": "0mvhovgc2-ih7l8hmrak",
          "start": 1792561565810,
          "status": "success",
          "end": 1792561567438,
          "result": {
            "removedFiles": [],
            "changedFiles": [],
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        }
      ],
      "end": 1792561567439,
      "result": {
        "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.",
        "name": "Error",
        "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)"
      },
      "message": "backup VM",
      "data": {
        "id": "1c163183-d863-f8cc-159a-5c5281aafee3",
        "type": "VM",
        "name_label": "BMC-DC2",
        "progress": 0
      }
    }
  ],
  "end": 1792561567595,
  "infos": [
    {
      "data": {
        "vms": [
          "1c163183-d863-f8cc-159a-5c5281aafee3"
        ]
      },
      "message": "vms"
    }
  ]
}

@coolsport00

I would delete the snapshots just to make sure they are gone.

@john.c

Updated both of mine yesterday and no issues, Debian 12 for both. One of them is an ancient HP T630, the other a vm as BIOS boot. Here might be the wrinkle, I haven't applied the latest XCP-ng updates, haven't had the time when things are clear enough to handle it.

@poddingue I haven't had time to get back and check this, my lab hasn't even had power on it for about 4 months.

@dinhngtu

Let me ask a question... Since the ability to set IP has been given in xe commands, will we be getting this ability through XO in the future? That would be really nice and would also help prevent shifting IP address when the driver updates.

Also what about NTP timing, if it is joined to a windows domain, will it still work using the domain controller as the time reference? I assume yes, but worth asking.

I normally have my hosts sync to GPS ntp server, the AD guests also sync to same ntp server, and all clients and member servers sync to AD (physical or virtual).

@poddingue

My three hosts are all EDT, drift on mine is pretty small because I have a local NTP server with GNSS.

So it could be a timezone issue and might be worth looking into.

I still haven't set any of that stuff up, still just local accounts. Never enough time!

@poddingue

I need to update a few things and then I plan on trying to see if this was fixed for me. I think I only had 2 Windows machines that were a problem, and only on machines that had a larger disk and were basically empty. The counting zeros seemed to lead to a time out. Just haven't had the time to look at much of anything lately.