XCP-ng 8.3 updates announcements and testing

gduperrey

Several things in your message raise questions for me.

First, this SR:

1x Linstor vSAN iSCSI configured as QCOW2

What is this storage and how is it configured?
Currently, we do not support QCOW2 on Linstor/Xostor. So, is it simply an iSCSI device to which you have applied the QCOW2 format, or is it a Xostor on the XCP-ng pool?

Regarding jumbo frames, we do not use them internally and are not currently testing them. It is often recommended not to use them for the management interface, as indicated in the XCP-ng documentation: https://docs.xcp-ng.org/networking/

Non-standard MTUs (such as jumbo frames) are not supported on management interfaces. Using them can lead to serious issues, including failed pool member joins or unexpected network outages.

Is that the case here?

bleader

@majorp93 no, these updates should not change anything on the MTU behavior. From your screenshots and ip a output, I assume eth4 is the only link used, than your vlan1060 network is indeed your management network. I guess with some confidence that:

eth4 is your actual NIC
xenbr4 is the pool-wide network without vlan using eth4
xapi1 is the bridge for vlan1060
If that's right indeed on your host side, the MTUs are properly set.

As Gaël said, jumbo on management network is not officially supported, because we always end up in situation similar to yours, where something stops working for "some" reason

But as you said, when everything is setup properly, it does work.

I have test hosts at home up to date but no similar setup to yours for SR. I tried with a pool-wide and a pool-wide + vlan as management with 9000 and the ping -M do -s 8972 does work fine in both cases, so nothing I can see here.

When it fails, what do you see? Message too long? mtu=1500 ? something else?

Do you have multiple hosts in that pool? Can you try that between the hosts as well and not toward the storage systems? I would also check on ovs with ovs-vsctl list interface that each of eth4, xennbr4 and xapi1 report 9000 mtu.

I know you said it was properly setup everywhere, but I would still be tempted to think there is a setup issue somewhere.

MajorP93

@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

marcoi

for the latest updates. the test pool didnt move the vms when i went to reboot. so i moved them manually. rebooted master, move vms back patched and reboot 2nd test server in the pool.
Skipped trying rolling reboot in prod and followed the same process. both pools went well without other issues.

gleh

New maintenance update candidates for XCP-ng 8.3 LTS

This batch of updates focuses on storage bug fixes and performance improvements.

What changed

Storage

blktap:
When CBT is enabled, an issue occurred where tapdisk opened cbtlog instead of the primary QCOW2 image. This previously caused commit operations (ENOTSUPP) and leaf-coalesce routines to fail, resulting in excessive storage consumption on the SR. Tapdisk now correctly bypasses the cbtlog image to allow proper coalescing.
sm: Fix a bug where the coalesce on a secondary host of a QCOW2 VDI on LVMSR could corrupt LVM metadata and would need manual intervention to restore them.
xapi:
- Migration speeds are now improved for sparse QCOW2-backed VDIs, as empty sectors are no longer transferred.
- Preserve VDI tags on migration. This is a foundational fix for kubernetes integration.

Versions

blktap: 3.55.5-9.3.xcpng8.3 -> 3.55.5-9.4.xcpng8.3
sm: 3.2.12-23.4.xcpng8.3 -> 3.2.12-23.5.xcpng8.3
xapi: 26.1.16-1.1.xcpng8.3 -> 26.1.16-1.2.xcpng8.3

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

As usual, normal use and anything else you want to test.

Test window before official release of the updates

~24 hrs

We would like to thank users who shared feedback since our last call for testing: @Andrew, @JeffBerntsen, @MajorP93, @XCP-ng-JustGreat, @acebmxer, @bufanda, @flakpyro

bufanda

@gleh installed on my lab pool. Haven’t seen any issues so far.

acebmxer

@gleh

Installed...

Updated:
  blktap.x86_64 0:3.55.5-9.4.xcpng8.3                forkexecd.x86_64 0:26.1.16-1.2.xcpng8.3                
  message-switch.x86_64 0:26.1.16-1.2.xcpng8.3       qcow-stream-tool.x86_64 0:26.1.16-1.2.xcpng8.3         
  rrdd-plugins.x86_64 0:26.1.16-1.2.xcpng8.3         sm.x86_64 0:3.2.12-23.5.xcpng8.3                       
  sm-cli.x86_64 0:26.1.16-1.2.xcpng8.3               sm-fairlock.x86_64 0:3.2.12-23.5.xcpng8.3              
  squeezed.x86_64 0:26.1.16-1.2.xcpng8.3             varstored-guard.x86_64 0:26.1.16-1.2.xcpng8.3          
  vhd-tool.x86_64 0:26.1.16-1.2.xcpng8.3             wsproxy.x86_64 0:26.1.16-1.2.xcpng8.3                  
  xapi-core.x86_64 0:26.1.16-1.2.xcpng8.3            xapi-nbd.x86_64 0:26.1.16-1.2.xcpng8.3                 
  xapi-rrd2csv.x86_64 0:26.1.16-1.2.xcpng8.3         xapi-storage-script.x86_64 0:26.1.16-1.2.xcpng8.3      
  xapi-tests.x86_64 0:26.1.16-1.2.xcpng8.3           xapi-xe.x86_64 0:26.1.16-1.2.xcpng8.3                  
  xcp-networkd.x86_64 0:26.1.16-1.2.xcpng8.3         xcp-rrdd.x86_64 0:26.1.16-1.2.xcpng8.3                 
  xenopsd.x86_64 0:26.1.16-1.2.xcpng8.3              xenopsd-cli.x86_64 0:26.1.16-1.2.xcpng8.3              
  xenopsd-xc.x86_64 0:26.1.16-1.2.xcpng8.3

MajorP93

@gleh Installed those RC packages in my test environment.
Tested basic operation. Looks good so far.
(Booted, all 3 SR connected, VMs able to start, both VHD and QCOW2)
Best regards

flakpyro

Installed on my usual hosts without issues. I am only making limited use of QCOW2 however.

gduperrey

Thank you everyone for your tests and your feedback!

The updates are live now: https://xcp-ng.org/blog/2026/08/27/august-2026-updates-2-for-xcp-ng-8-3-lts/

Andrew

@gduperrey Installed and running. Rolling pool update worked.

mthird

First time rolling update worked for my pool. Woot!

mthird

Spoke to soon. While the updates succeeded, one of the nodes is rebooting every few minutes due to an HA self-fence.

manilx

This is a new one:

Updated all our 3 pools:
ScreenShot 2026-08-28 at 07.20.40.png
Backup via CLI, runs XOA

Then DR.
ScreenShot 2026-08-28 at 07.20.28.png
Tasks showed RPU on pool PRD instead of DR. RPU finished just fine but task stayed.

Then RPU on PRD. Task did not change. Finished also fine. task stayed.

gduperrey

@manilx If the pools are up to date, I would try restarting the XO VM to see if the task persists.

If it does persist, it might be best to open a separate topic for the XO team.

manilx

@gduperrey Restarted and the task persisted. Deleted it from the list.

acebmxer

1 out of 3 pools at work failed rolling pool update.

When I try to put host 2 into maintence mode to finish updates i get this error...

Support Ticket - Ticket#7763405

host.setMaintenanceMode
{
  "id": "60701efd-089c-4822-97c0-1a1057f3f9aa",
  "maintenance": true
}
{
  "code": "VM_REQUIRES_SR",
  "params": [
    "OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8",
    "OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2"
  ],
  "task": {
    "uuid": "a24f5565-b2cb-f392-8095-1beceb7a4a17",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260828T19:57:29Z",
    "finished": "20260828T19:57:29Z",
    "status": "failure",
    "resident_on": "OpaqueRef:43c121e7-4ba4-193b-1e41-08c0f7e15690",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_REQUIRES_SR",
      "OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8",
      "OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 629))((process xapi)(filename hashtbl.ml)(line 159))((process xapi)(filename hashtbl.ml)(line 165))((process xapi)(filename hashtbl.ml)(line 170))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 625))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "VM_REQUIRES_SR(OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8, OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2)",
  "name": "XapiError",
  "stack": "XapiError: VM_REQUIRES_SR(OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8, OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1229:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1263:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1253:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1460:14)"
}

SR are not connected to master host after reboot from updates. When try to connect them i get error...

Screenshot 2026-08-28 160401.png

pbd.connect
{
  "id": "e96ec5d8-c70f-c877-4263-f2cf763e7048"
}
{
  "code": "SR_BACKEND_FAILURE_108",
  "params": [
    "",
    "Unable to detect an NFS service on this target.",
    ""
  ],
  "task": {
    "uuid": "5574bfe5-1874-35f8-f53b-c9fc1abaebb5",
    "name_label": "Async.PBD.plug",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260828T20:03:19Z",
    "finished": "20260828T20:03:22Z",
    "status": "failure",
    "resident_on": "OpaqueRef:43c121e7-4ba4-193b-1e41-08c0f7e15690",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "SR_BACKEND_FAILURE_108",
      "",
      "Unable to detect an NFS service on this target.",
      ""
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-idl/storage/storage_interface.ml)(line 455))((process xapi)(filename src/lib/idl.ml)(line 558))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 148))((process xapi)(filename lib/backtrace.ml)(line 251))((process xapi)(filename ocaml/xapi/xapi_pbd.ml)(line 196))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 6061))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "SR_BACKEND_FAILURE_108(, Unable to detect an NFS service on this target., )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_108(, Unable to detect an NFS service on this target., )
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1229:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1263:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1253:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1460:14)"
}

Update - bad port on switch.......

Update 2 - After updating synology to latest version i am no longer able to mount nfs 4.1 shares. I can not create one using 4.1. I will let me create with 4.0 not 4.1 How to remount NFS that were mounted as 4.1 now as 4.0?

ALl VMS down at this location currently.

Update 3 - Resolved... Synology update broke NFS 4.1. I had to ssh into synology and edit /etc/nfs/syno_nfs_conf and add nfs_minor_ver_enable=1 Now SR's mounted.

Update 4 - Found Patch notes about NFS version change. -
Version: 7.0.1-42218
NFSv4.1 and related advanced functions (multipathing) are no longer supported on Synology NAS models with the following package architectures: Alpine, Alpine4k, armada38x, Avoton, Braswell, Bromolow, Cedarview, Grantley, Monaco. If you had previously enabled NFSv4 and NFSv4.1, the maximum NFS protocol is set to NFSv4 by default after the update. For more information about the package architecture of your model, please refer to this article.

Our unit - RS1219+ Intel Atom C2538 Avoton

XCP-ng-JustGreat

I deferred applying the preview patches this time in order to try my luck again with RPU. Unfortunately, it did not work for me again as it has not in the past. My results are similar to others here: primary host patch application went fine including the reboot. However, once it began doing a secondary host, it started throwing errors e.g. CANNOT_EVACUATE_HOST and VM_REQUIRES_SR etc. However, manually putting the host in maintenance mode from the GUI evacuated each host just fine and I was able to apply the patches and reboot each subsequent host from the XO GUI. Also, as with others here, the RPU task hung in the task list and even a reboot of the XO VM would not clear it forcing me to delete the task using xo-cli e.g. xo-cli rest del tasks.
ENVIRONMENT: Home lab consisting of 4 x Dell OptiPlex 7040 i7-6700 SFF hosts, 48GB RAM each, 10 Gbps storage connections to a TrueNAS home-built NAS via NFS and XO from source (XOS) using @ronivay build script on AlmaLinux 10.2 minimal install VM with XO commit 6a441 compiled on 2026-08-28 from master branch. FWIW, RPU functionality remains unavailable to me, though obviously, this is not a showstopper for a 4 x host home lab pool. If there is anything I can do to help resolve this, please let me know as I remain an enthusiastic proponent of the Vates virtualization stack.

gduperrey

Hello @acebmxer,

I can see from the support ticket and the summary that the issue appears to be resolved and was linked to the NFS update on the Synology. It’s great that you were able to find the solution.

gduperrey

Hello @XCP-ng-JustGreat,

I cannot comment on RPU issues via XO. I would recommend creating a new topic to report the problem directly to the XO team so they can analyze it.

As you can see, we recommend validating updates via the command line directly on XCP-ng before releasing them. That is a different scenario.

However, if several of you are experiencing RPU issues with XO, a dedicated topic will allow that team to investigate the situation and consult other teams if necessary. This would ensure the issue is better addressed and analyzed