XCP-ng 8.3 updates announcements and testing

Andrew

@lmolina Updates ran fine, but I'm having CR backup problems too... Mostly ran fine, but I now have 3 stuck with VBD.unplug_force. I can't get them to clear. I'll have to try harder before I revert updates.

flakpyro

@rzr I was worried about this because of XAPI. Luckily i have a host thats not currently running any VMs so i tried it there first hoping it was a small enough change that i could get away with it. Good to know i could have just disabled NBD for now though!

acebmxer

@rzr said:

@acebmxer said:
(...)
yum history undo <ID> --enablerepo=xcp-ng-testing
(...)

Your're lucky ! this is a risky operation we don't recommend to downgrade because some packages like xapi are not handling this properly, prefer other workarounds if there are any (like disabling nbd until @storage team suggest a better option)

Let me thank you again @flakpyro for this regression report it is very valuable to us,

I didn't run the commands I stated. As I was asking first.. I did the command @flakpryo

Of course this is was before i saw your post. As this is the home lab i am fine with if it blows up. but for others this is good to know.

rzr

We have just released security updates for ca-certificates.

Full details are available on the blog:
https://xcp-ng.org/blog/2026/10/22/october-2026-security-updates-3-for-xcp-ng-8-3-lts/

Greg_E

Ok, so what's the scoop here? Safe to roll this out or wait a few days? I just saw the notice in my XO that there are updates and I don't want to wreck my system right now after fixing a few other "issues".

[edit] I see this is just the certificate update so pushing it out and listening to servers reboot.

acebmxer

@rzr

I reinstalled the updates, and confirm backups work with NBD disablled in backup job. - I missed the part about NBD in the notes..

However the task stay at 0% while i can see traffic on my trunas..

jr-m4

jr-m4 said:

Updated two dev-pools successfully!

new-xcp-dev: 2x Dell Inc. PowerEdge R650xs
old-xcp-dev: 3x Dell Inc. PowerEdge R350 (HA)

I even mistankenly did a wrong updating order on old-xcp-dev. And neglected to disable HA before updating, leading to two hosts being updated and promoting the next master. And by chance the last master was the non-updated one. Here I disabled HA, but couldn't promote another master (not unexpected)... Post update, and everything seems to be good anyways. And HA re-enabled successfully

Note:
No further testing other than the hosts booting and showing good life signs have been done, yet.

I'm now also seeing issues with backups being stuck for multiple hours.
Update: Correction. I'm now having issues with almost everything on both pools.

migrations
snapshots
changing status e.g. rebooting or hard shut down of VMs

Andrew

@lmolina @olivierlambert Sorry, but this update got ugly fast. CR backups got stuck. Migrations got stuck. VMs would not start any more. After reverting all the patches and rebooting all hosts I had several VMs that would not start again (including primary XO) because the VDIs stuck marked as activating. It took some time to fix them but at least there was no data loss. Some backups lost their snapshots and had to revert to full backups again.

Emmanuel V

Some message formats changed between the two releases, so VMs started before the update (with an 11.3 tapdisk and tapback) can’t understand some of the changed commands. Rebooting XCP-ng or doing an RPU should work. Since not every command is affected, restarting tapback is usually enough: systemctl restart tapback.service
We're going to work to avoid that.

jr-m4

@Emmanuel-V

Rebooting hosts and/or VMs doesn't solve the problems.
As @andrew said. This one got ugly. But this is also why we test stuff

Update:
Doing a "Rescan all disks" on the SR (lvmohba), seems to help out a little bit. But not everything.
Some VMs were able to be started again. But others are still showing problems and erroring out

"The VDI is not available [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:7---------2c1b', 'activating']]",

flakpyro

@jr-m4 I had this issue on one of the pools i tested on as well and was able to fix the issue by following the fix here:
https://xcp-ng.org/forum/topic/10603/sr_backend_failure_46-the-vdi-is-not-available-not-detached-cleanly/3

dthenot

@flakpyro Hello,

It would be preferred to use another method.
You can use the resetvdis.py script in the /opt/xensource/sm/ to reset values instead,
using vdi-forget is unadvised in this case, it can cause issues by losing information that are stored in XAPI about VDIs.

/opt/xensource/sm/resetvdis.py single <VDI UUID>

jr-m4

@dthenot said:

@flakpyro Hello,

It would be preferred to use another method.
You can use the resetvdis.py script in the /opt/xensource/sm/ to reset values instead,
using vdi-forget is unadvised in this case, it can cause issues by losing information that are stored in XAPI about VDIs.
/opt/xensource/sm/resetvdis.py single <VDI UUID>

I was just about to look at maybe doing what @flakpyro suggested. More likely I would've just restored from backup. As this is dev-instances, and non-important.

But your resetvdis.pysuggestion solved it entirely! Thanks for the quick tip!

jr-m4

Now trying to see if pool works with generic operations.

Trying to do a RPR on new-xcp-dev. And I'm met with

Ofcourse there are no backups being run. They are even disabled right now. And the only active task happening is "Garbage collection" on the main SR.

pool.rollingReboot
{
  "pool": "1f16dad2---------------6c8b8b7",
  "bypassBackupCheck": false,
  "shutdownPinnedVms": false
}
{
  "code": "CANNOT_EVACUATE_HOST",
  "params": [
    "VM_LACKS_FEATURE,OpaqueRef:aa5210c4-a------------1b80-9bca5d0d0f8d|OPERATION_BLOCKED,OpaqueRef:c7a08---------------a0--------------9306,VM migration is blocked during backup"
  ],
  "call": {
    "duration": 23,
    "method": "host.assert_can_evacuate",
    "params": [
      "* session id *",
      "OpaqueRef:0145019c-2a32-13bb-5951-3bca9e59e63a"
    ]
  },
  "message": "CANNOT_EVACUATE_HOST(VM_LACKS_FEATURE,OpaqueRef:aa52-----------------------9bca5d0d0f8d|OPERATION_BLOCKED,OpaqueRef:c7a0------------------------306,VM migration is blocked during backup)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_EVACUATE_HOST(VM_LACKS_FEATURE,OpaqueRef:aa521-------------------8d|OPERATION_BLOCKED,OpaqueRef:c7a08c--------------------------9306,VM migration is blocked during backup)
    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at processImmediate (node:internal/timers:541:9)"
}

Update:
I believe this might be related due to a VDI being orphaned, and trying to delete it results in VDI_NOT_MANAGED-error. Trying a new rescan of disks on the SR, and hopefully garbage collection will finish until Monday. Otherwise I will somehow try to forcefully destroy this VDI.

Update 2:
Managed to do a RPR now.
Seems that the flag "Prevent Migration" was applied to one of the VMs. My guess is that this is applied when backups are taken. And since the backups broke with the updates. The flag never got cleared.
I did a quick visual confirmation to see that my guess of this behaviour seems correct. And it indeed sets the "Prevent migration"-flag whilst the backup runs.

Running backups on my pools to see if functionality is working as intended again.

Update 3:
Maybe related, maybe not. But my backup-tests are piling up with lots of VBD.unplug_force, that have gone on for <=30 minutes. And this is on both pools. And it seems very much so that the backup-jobs doesn't complete properly. This could also be entirely XO-related, since there has been several commits towards backups as well.

Update 4:
Updated XO from source to the latest commit d2ffe, and things have not improved.
Pretty much, many actions with VDIs are problematic. I'll happily provide logs. But will only do so if I get somewhere to upload them to.
I'll be taking the weekend now. So won't be able to respond before monday,

JeffBerntsen

Updated and running on my test pool and twinstor test pair. I don't use XO for backups so I'm not seeing the possible issues associated with that but all else seems to be running just fine.

gduperrey

The storage and XAPI teams are currently investigating the issues that have been reported.

There is no final solution yet, but we are getting closer. Unfortunately, there is no ETA at this time.

Thank you very much for all your testing and feedback!

olivierlambert

Thanks everyone for testing! As you can all see here, it’s been extremely useful

acebmxer

This post is deleted!

lmolina

New packages in `xcp-ng-testing` for the next XCP-ng 8.3 LTS maintenance update

This is a follow-up of the the previous post and addresses the issue reported by our testers.

First of all, a huge thanks to all testers who allowed us to catch this backup issue early:
@acebmxer, @jr-m4, @bufanda, @flakpyro, @Andrew.

We decided to revert the change that improves the performance of xapi-nbd since it caused the backup issue. This change will be reworked and refined, then presented to you again in a future update, without this issue. While investigating, we also found a bug in blktap: in rare cases, the process handling a VM disk (tapdisk) could crash during a backup and leave the disk impossible to detach. The new blktap version fixes it.

Versions

blktap: 3.55.5-12.2.xcpng8.3 -> blktap-3.55.5-12.3.xcpng8.3
xapi: 26.1.21-1.1.xcpng8.3 -> 26.1.21-1.2.xcpng8.3

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

If you reverted the previous candidates with yum history undo, update normally with the commands above. Note that downgrading is not recommended. If you disabled NBD in your backup jobs as a workaround, please enable it again after the update.

The usual update rules apply: pool coordinator first, etc.

What to test

If you have already carried out the initial updates and performed the tests mentioned in our previous post, please confirm that the XO backups are now back to a working state.

If you did not run the initial update and have not yet performed any tests, please be sure to review the "what to test" section of the the previous post.

Test window before official release of the updates

~7 days

acebmxer

@lmolina

of course i just downgraded again prior to this release... Sorry for the post. This was assisted with AI. I was also trying to do this while at work . I tried to verify this was not caused be my actions or configuration. If this is the results of upgrading and downgrading then everyelse use this a learning point.

[13:10 xcp-ng-disjqdnc ~]# yum history
Loaded plugins: fastestmirror
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    16 | update --enablerepo=xcp- | 2026-10-27 13:04 | Update         |   35 EE
    15 | history undo 14          | 2026-10-27 10:37 | Downgrade      |   35 EE
    14 | update --enablerepo=xcp- | 2026-10-22 19:00 | Update         |   35 EE
    13 | update --disablerepo=* - | 2026-10-22 15:40 | Update         |    1   
    12 | history undo 11          | 2026-10-22 13:37 | Downgrade      |   35 EE
    11 | update --enablerepo=xcp- | 2026-10-22 05:58 | Update         |   35 EE
    10 | update --disablerepo=* - | 2026-10-13 13:13 | Update         |    6   
     9 | update --disablerepo=* - | 2026-10-05 15:04 | Update         |   20   
     8 | update --enablerepo=xcp- | 2026-10-01 09:12 | Update         |   45 EE
     7 | update --disablerepo=* - | 2026-09-08 16:06 | Update         |    6   
     6 | update --enablerepo=xcp- | 2026-08-26 08:39 | Update         |   23 EE
     5 | update --enablerepo=xcp- | 2026-08-14 13:23 | I, U           |   53 EE
     4 | update                   | 2026-07-28 10:46 | Update         |    5   
     3 | update --enablerepo=xcp- | 2026-07-10 11:27 | Update         |   23 EE
     2 | install -y /root/twinsto | 2026-07-05 20:16 | Install        |   12   
     1 | update --disablerepo=* - | 2026-07-05 01:56 | I, O, U        |  138 EE

history list

updates installed no issues. I did a full backup job prior to new updates. after updates I did a delta backup. (vms are in VHD format)

migrated VMS back to QCOW2 and ran another backup. Because of migration this was a full backup..

@stormi @lmolina

Ran into a massive storage deadlock testing this batch of candidates on my NFS setups with Mellanox ConnectX links.

Here is the exact timeline and raw outputs of what went down:

Clean baseline: Rolled back previous stuff via yum history undo so the host was at a completely clean slate first.
Pre-test: Ran a full backup on standard VHD layout on the stable build. Zero issues, finished perfectly fine.
The update: Upgraded to the testing candidate repo (ID 16 in my history output above).
The trigger: Cleared a delta backup right after, then kicked off a full backup job. No other out-of-band tasks or live migrations were happening.

The Breakdown:
Mid-way through that full VHD backup, the entire storage layer completely locked up.

Toolstack went blind: xe task-list was completely blank. The host totally lost track of the job it was supposed to be running.
VMs deadlocked: Every user VM instantly frozen into a hard Blocked/Halted (B H) state. Trying a low-level xl destroy just turned them into permanent un-killable zombie domains (D P H).

[13:33 xcp-ng-rngyaiub ~]# xl destroy 1
[13:33 xcp-ng-rngyaiub ~]# list_domains
id |                                 uuid |  state
 0 | 2083b5bd-7878-4598-960f-bb425814d338 |     R 
 1 | 49f51614-c105-4fea-3103-566613ec79f2 | D P  H
 2 | dcbfaff1-d782-b42b-6eed-3d6b6d1ca032 | D P  H
 3 | 8cc17a78-ef65-1c06-4ac1-8e56b609d714 | D P  H
 4 | 2f724b50-5b16-9975-839b-fd85dd8ed065 | D P  H

Hardware / Config Diagnostics:
Before pointing fingers at the candidate code, I checked if my Mellanox links or the network switch choked under the backup stream load. Ran ethtool immediately following recovery and the physical interfaces are pristine:

[14:16 xcp-ng-disjqdnc ~]# ethtool -S eth1 | grep -E "discard|drop|filter|error"
     rx_xdp_drop: 0
     tx_queue_dropped: 0
     rx_oversize_pkts_sw_drop: 0
     rx_pet_hdr_lookup_drop: 0
     rx_pet_mdata_lookup_drop: 0
     rx_xsk_xdp_drop: 0
     rx_xsk_oversize_pkts_sw_drop: 0
     rx_crc_errors_phy: 0
     rx_in_range_len_errors_phy: 0
     rx_discards_phy: 0
     tx_discards_phy: 0
     tx_errors_phy: 0
     rx_prio0_discards: 0
     rx_prio1_discards: 0
     rx_prio2_discards: 0
     rx_prio3_discards: 0
     rx_prio4_discards: 0
     rx_prio5_discards: 0
     rx_prio6_discards: 0
     rx_prio7_discards: 0
     tx_pause_storm_error_events: 0

[14:18 xcp-ng-disjqdnc ~]# ethtool -k eth1 | grep -i vlan
rx-vlan-offload: on
tx-vlan-offload: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]

Conclusion:
Network layout config problems, buffer overflows, or silent packet drops are 100% ruled out here (all hardware discards are flat zero).

This wasn't a user mistake or an accidental configuration mismatch. The new sm or blktap components hit a deep kernel lockup handling the multi-stream VHD storage exports. The file locks deadlocked the kernel handles so badly that the hosts ignored standard and low-level CLI commands entirely, forcing a hard out-of-band physical power cycle of the XCP-ng hosts and the TrueNAS storage server just to clear the uninterruptible I/O sleep queue.

I haven't touched a thing since getting the hardware back up and verifying what choked. The system is sitting exactly as it is right now. Let me know what specific debugging paths you want me to look at or what logs you need from that window to trace this down.