"NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update

archw

Yesterday I upgraded to the most recent version of XCP-NG 8.3 via yum update.

Last night's backup is full of this error using the following:
Xen Orchestra, commit 8448b
Xen Orchestra, commit 1de60

I looked at the logs and they pointed to a snapshot error.

For the heck of it , I tried running a snapshot on any of the VM's. Each one fails with this error:

vm.snapshot
{
  "id": "0b65cf83-8309-3dcc-fa94-a511c77c96bd"
}
{
  "code": "NOT_SUPPORTED_DURING_UPGRADE",
  "params": [],
  "task": {
    "uuid": "71b630bb-d299-bab2-c2e9-065cd4bfef5c",
    "name_label": "Async.VM.snapshot",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20251219T14:33:39Z",
    "finished": "20251219T14:33:39Z",
    "status": "failure",
    "resident_on": "OpaqueRef:3ea53cca-30d6-7c19-3f77-3f7eb26c6675",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "NOT_SUPPORTED_DURING_UPGRADE"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 755))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename fun.ml)(line 33))((process xapi)(filename fun.ml)(line 38))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1788))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1740))((process xapi)(filename ocaml/xapi/rbac.ml)(line 229))((process xapi)(filename ocaml/xapi/rbac.ml)(line 239))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "NOT_SUPPORTED_DURING_UPGRADE()",
  "name": "XapiError",
  "stack": "XapiError: NOT_SUPPORTED_DURING_UPGRADE()
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202512190914/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202512190914/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202512190914/packages/xen-api/index.mjs:1078:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202512190914/packages/xen-api/index.mjs:1112:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202512190914/packages/xen-api/index.mjs:1102:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202512190914/packages/xen-api/index.mjs:1275:14)"

I've tried restarting the toolstack on the pool master as well as the hosts.

Any ideas?

Danp

Were the hosts rebooting after the patches were applied?

archw

@Danp
Nope!

MajorP93

@archw I'd advise to read documentation at the part related to rebooting after package upgrades: https://docs.xcp-ng.org/management/updates/#-when-to-reboot

In this case "xen" updates were included in December package updates which results in one of the criteria for having to reboot being met.

Personally I reboot every time package updates were installed. You stay on the safe side by doing so.

archw

@MajorP93
I always reboot at the conclusion of the update and did as well this time.

Danp

@archw I'm confused by your recent responses. I asked --

Were the hosts rebooting after the patches were applied?

to which you replied --

Nope!

but then you stated

I always reboot at the conclusion of the update and did as well this time.

Which is correct?

MajorP93

@archw Mhh, Danp asked if you rebooted your hosts after applying the patches and you said nope.

archw

@MajorP93
My bad....because of the word "rebooting" I took the statement "Were the hosts rebooting after the patches were applied?' to mean 'Were the hosts rebooting while the patches were being applied?'

To recap:
I did reboot after the patches had been applied.

Sorry !!

MajorP93

@archw Ohh I get it now! The "rebooting" instead of "rebooted" can be understood as "did applying the patches cause the systems to reboot" as in system crash or similar.
Gotcha!
Understood what you meant now.

//EDIT: anyways, back to topic. In case the systems were already rebooted after applying the updates I currently do not have an idea what might cause this...

Danp

s/rebooting/rebooted

Apologies for the miswording on my end.

archw

@Danp
WDE

john.c

@Danp said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

s/rebooting/rebooted

Apologies for the miswording on my end.

Please be careful of tenses (past tense etc), as it will change the meaning of the question and thus the answer. As rebooted can be understood as it has already completed rebooting the host, following the update reboot request.

Danp

@john.c said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

Please be careful of tenses (past tense etc),

Yeah... I was trying to be helpful, but I was clearly in too much of a hurry.

Mea culpa

john.c

@Danp said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

@john.c said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

Please be careful of tenses (past tense etc),

Yeah... I was trying to be helpful, but I was clearly in too much of a hurry.

Mea culpa

It’s alright great that you’re willing to help even during your own time. I posted what I did in order to help you get better with the English language.

Next time you think you’re in too much of a hurry say to yourself or in your mind, this old saying “Haste makes waste”.

archw

Mystery solved:
There are numerous hosts under one master in the pool. I ran the yum update command on the master first, then rebooted, I then ran yum update on the rest and rebooted all but one. I could not reboot the last one until this morning (Saturday).

I rebooted the last one a few minutes ago and all is well.

Because I think its interesting, I'd love someone smart to explain the process of why other hosts could not do a snapshot because a different host in the pool had not been rebooted.

olivierlambert

Because doing an update without rebooting doesn't reload the updated main programs, like XAPI. A host in only updated after a full reboot.

magicker

@olivierlambert said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

Because doing an update without rebooting doesn't reload the updated main programs, like XAPI. A host in only updated after a full reboot.

Reply

Hi there
Is it just me or is this a chicken and egg situation.

you upgrade the master... how the pool is in NOT_SUPPORTED_DURING_UPGRADE() stage. You cant move vms off the master so all you can do is close down vms.. reboot.. pray

then move the a non master.. you cant move the vms off here either NOT_SUPPORTED_DURING_UPGRADE(). So you have do the same..

needless to say I hit issues on each reboot which caused 30- 60 min delays in getting vms back up and running.

can you Warm migrate or is this dead also (to scared to test)

MajorP93

@magicker said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

@olivierlambert said in "NOT_SUPPORTED_DURING_UPGRADE()" error after yesterday's update:

Because doing an update without rebooting doesn't reload the updated main programs, like XAPI. A host in only updated after a full reboot.

Reply

Hi there
Is it just me or is this a chicken and egg situation.

you upgrade the master... how the pool is in NOT_SUPPORTED_DURING_UPGRADE() stage. You cant move vms off the master so all you can do is close down vms.. reboot.. pray

then move the a non master.. you cant move the vms off here either NOT_SUPPORTED_DURING_UPGRADE(). So you have do the same..

needless to say I hit issues on each reboot which caused 30- 60 min delays in getting vms back up and running.

can you Warm migrate or is this dead also (to scared to test)

For me this workflow worked every time there were upgrades available:

-disable HA on pool level
-disable load balancer plugin
-upgrade master
-upgrade all other nodes
-restart toolstack on master
-restart toolstack on all other nodes
-live migrate all VMs running on master to other node(s)
-reboot master
-reboot next node (live migrate all VMs running on that particular node away before doing so)
-repeat until all nodes have been rebooted (one node at a time)
-re-enable HA on pool level
-re-enable load balancer plugin

Never had any issues with that. No downtime for none of the VMs.

olivierlambert

Exactly and that's what does the Rolling Pool Update feature