jr-m4

@Octopuss

Are you running a pool with only 1 host inside? If so, then I believe Rolling Pool Update (RPU) isn't available.

The point of it is to have a automated way to let XO migrate VMs in the pool to the other hosts. And when the intended host (master first) is empty, it will place it in maintenance mode, and afterwards reboot it. This will then re-iterate until all hosts in the pool have been updated.

More can be found here:
https://docs.xcp-ng.org/management/updates/#from-xen-orchestra

3 pools updated. RPU went fine, where one is HA. A couple of test-backups went fine as well.

When you're in the Metadata section of the restore The button for Refresh backup list disapears.

I'm kind of expecting this to not be worked on. Due to it being so small, and in XO5. But for the interest of reporting things. I wanted to share anyways.

VS

Updated both test-pools: old-xcp-dev & new-xcp-dev

Backups seem to be working fine. But I'm noticing some warnings now (See below). However, these can very well be the result of clumsy and crude troubleshooting with fixing the previous test-release symptoms.

Dashboard/Health doesn't report any issues. And I'm not seing issues with the VMs in their running state.
Backups are seemingly clearing HealthChecks as well.

I will update this post in a living matter. So to not spam the forum-thread with many small posts.

Symptom 1:

Update 1:
So the only way I could "solve" the issue, was to delete all of the backups for the VM in question. Afterwards it started behaving as expected. I haven't found anything else problematic so far.

Now trying to see if pool works with generic operations.

Trying to do a RPR on new-xcp-dev. And I'm met with

Ofcourse there are no backups being run. They are even disabled right now. And the only active task happening is "Garbage collection" on the main SR.

pool.rollingReboot
{
  "pool": "1f16dad2---------------6c8b8b7",
  "bypassBackupCheck": false,
  "shutdownPinnedVms": false
}
{
  "code": "CANNOT_EVACUATE_HOST",
  "params": [
    "VM_LACKS_FEATURE,OpaqueRef:aa5210c4-a------------1b80-9bca5d0d0f8d|OPERATION_BLOCKED,OpaqueRef:c7a08---------------a0--------------9306,VM migration is blocked during backup"
  ],
  "call": {
    "duration": 23,
    "method": "host.assert_can_evacuate",
    "params": [
      "* session id *",
      "OpaqueRef:0145019c-2a32-13bb-5951-3bca9e59e63a"
    ]
  },
  "message": "CANNOT_EVACUATE_HOST(VM_LACKS_FEATURE,OpaqueRef:aa52-----------------------9bca5d0d0f8d|OPERATION_BLOCKED,OpaqueRef:c7a0------------------------306,VM migration is blocked during backup)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_EVACUATE_HOST(VM_LACKS_FEATURE,OpaqueRef:aa521-------------------8d|OPERATION_BLOCKED,OpaqueRef:c7a08c--------------------------9306,VM migration is blocked during backup)
    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at processImmediate (node:internal/timers:541:9)"
}

Update:
I believe this might be related due to a VDI being orphaned, and trying to delete it results in VDI_NOT_MANAGED-error. Trying a new rescan of disks on the SR, and hopefully garbage collection will finish until Monday. Otherwise I will somehow try to forcefully destroy this VDI.

Update 2:
Managed to do a RPR now.
Seems that the flag "Prevent Migration" was applied to one of the VMs. My guess is that this is applied when backups are taken. And since the backups broke with the updates. The flag never got cleared.
I did a quick visual confirmation to see that my guess of this behaviour seems correct. And it indeed sets the "Prevent migration"-flag whilst the backup runs.

Running backups on my pools to see if functionality is working as intended again.

Update 3:
Maybe related, maybe not. But my backup-tests are piling up with lots of VBD.unplug_force, that have gone on for <=30 minutes. And this is on both pools. And it seems very much so that the backup-jobs doesn't complete properly. This could also be entirely XO-related, since there has been several commits towards backups as well.

Update 4:
Updated XO from source to the latest commit d2ffe, and things have not improved.
Pretty much, many actions with VDIs are problematic. I'll happily provide logs. But will only do so if I get somewhere to upload them to.
I'll be taking the weekend now. So won't be able to respond before monday,

@dthenot said:

@flakpyro Hello,

It would be preferred to use another method.
You can use the resetvdis.py script in the /opt/xensource/sm/ to reset values instead,
using vdi-forget is unadvised in this case, it can cause issues by losing information that are stored in XAPI about VDIs.

/opt/xensource/sm/resetvdis.py single <VDI UUID>

I was just about to look at maybe doing what @flakpyro suggested. More likely I would've just restored from backup. As this is dev-instances, and non-important.

But your resetvdis.pysuggestion solved it entirely! Thanks for the quick tip!

@Emmanuel-V

Rebooting hosts and/or VMs doesn't solve the problems.
As @andrew said. This one got ugly. But this is also why we test stuff

Update:
Doing a "Rescan all disks" on the SR (lvmohba), seems to help out a little bit. But not everything.
Some VMs were able to be started again. But others are still showing problems and erroring out

"The VDI is not available [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:7---------2c1b', 'activating']]",

@MrQ said:

@jr-m4 thank you so much for testing this updated driver.

Just to make sure, I understand now that you confirm the driver is working as expected and the issues you had were not related to it? Or are there any problems still with this new driver?

Many thanks again for spending the time to test and report.

Hi

Yes. I believe I was chasing a red herring of my own doing. I have not found anything actually problematic with this driver at all.

jr-m4 said:

Updated two dev-pools successfully!

new-xcp-dev: 2x Dell Inc. PowerEdge R650xs
old-xcp-dev: 3x Dell Inc. PowerEdge R350 (HA)

I even mistankenly did a wrong updating order on old-xcp-dev. And neglected to disable HA before updating, leading to two hosts being updated and promoting the next master. And by chance the last master was the non-updated one. Here I disabled HA, but couldn't promote another master (not unexpected)... Post update, and everything seems to be good anyways. And HA re-enabled successfully

Note:
No further testing other than the hosts booting and showing good life signs have been done, yet.

I'm now also seeing issues with backups being stuck for multiple hours.
Update: Correction. I'm now having issues with almost everything on both pools.

• migrations
• snapshots
• changing status e.g. rebooting or hard shut down of VMs

Updated two dev-pools successfully!

new-xcp-dev: 2x Dell Inc. PowerEdge R650xs
old-xcp-dev: 3x Dell Inc. PowerEdge R350 (HA)

I even mistankenly did a wrong updating order on old-xcp-dev. And neglected to disable HA before updating, leading to two hosts being updated and promoting the next master. And by chance the last master was the non-updated one. Here I disabled HA, but couldn't promote another master (not unexpected)... Post update, and everything seems to be good anyways. And HA re-enabled successfully

Note:
No further testing other than the hosts booting and showing good life signs have been done, yet.