@pkgw I tested it with a cluster size of 2 megabytes. I got similar results to those with the default size.

@olivierlambert said:

Might be interesting to test a different cluster size and see the impact

I tested it with a cluster size of 2 megabytes, and nothing changed

@stormi XCP-ng QCOW2 vs. VHD Performance Feedback on NVMe
First of all, I would like to thank the team for all the hard work in bringing QCOW2 support to a production-ready state. It is a very welcome feature.

I have performed some quick I/O benchmarks comparing the new QCOW2 format against the traditional VHD. In my tests, QCOW2 appears significantly slower than VHD on my hardware.

Test Environment
Hypervisor: Dell PowerEdge R420

CPU: Intel Xeon E5-2470 v2

Storage: Intel SSDPELKX010T8 NVMe

VM OS: Debian 13

VM Specs: 2 vCPUs, 1GB RAM

Setup: One 10GB VHD and one 10GB QCOW2 disk, both pre-filled from /dev/random.

Methodology
I used a custom test suite available here: https://vm01.unsoft.hu/~ventura/fio/fio_test_20250408.tar.gz

I also ran a simplefio loop with the following results:
VHD:

root@Debian-13-CloudInit-20250810:/mnt/vhd# for mode in read write; do for jobs in 1 16; do for bs in 4 64; do for t in "" rand; do printf "%2i qd %2ik % 4s " $jobs $bs $t; fio --name=random-write --rw=$t$mode --bs=${bs}k --numjobs=1 --size=1g --iodepth=$jobs --runtime=10 --time_based --direct=1 --ioengine=libaio|grep -e BW -e runt ; done; done; done; done
 1 qd  4k        read: IOPS=9625, BW=37.6MiB/s (39.4MB/s)(376MiB/10001msec)
 1 qd  4k rand   read: IOPS=5414, BW=21.2MiB/s (22.2MB/s)(212MiB/10001msec)
 1 qd 64k        read: IOPS=2657, BW=166MiB/s (174MB/s)(1661MiB/10001msec)
 1 qd 64k rand   read: IOPS=2575, BW=161MiB/s (169MB/s)(1610MiB/10001msec)
16 qd  4k        read: IOPS=45.7k, BW=178MiB/s (187MB/s)(1785MiB/10001msec)
16 qd  4k rand   read: IOPS=45.9k, BW=179MiB/s (188MB/s)(1794MiB/10001msec)
16 qd 64k        read: IOPS=16.7k, BW=1041MiB/s (1092MB/s)(10.2GiB/10001msec)
16 qd 64k rand   read: IOPS=16.7k, BW=1042MiB/s (1093MB/s)(10.2GiB/10001msec)
 1 qd  4k        write: IOPS=8842, BW=34.5MiB/s (36.2MB/s)(345MiB/10001msec); 0 zone resets
 1 qd  4k rand   write: IOPS=8880, BW=34.7MiB/s (36.4MB/s)(347MiB/10001msec); 0 zone resets
 1 qd 64k        write: IOPS=6095, BW=381MiB/s (399MB/s)(3810MiB/10001msec); 0 zone resets
 1 qd 64k rand   write: IOPS=6006, BW=375MiB/s (394MB/s)(3755MiB/10001msec); 0 zone resets
16 qd  4k        write: IOPS=49.3k, BW=193MiB/s (202MB/s)(1928MiB/10001msec); 0 zone resets
16 qd  4k rand   write: IOPS=47.3k, BW=185MiB/s (194MB/s)(1848MiB/10001msec); 0 zone resets
16 qd 64k        write: IOPS=14.3k, BW=891MiB/s (934MB/s)(8910MiB/10001msec); 0 zone resets
16 qd 64k rand   write: IOPS=15.5k, BW=966MiB/s (1013MB/s)(9663MiB/10001msec); 0 zone resets

QCOW2

root@Debian-13-CloudInit-20250810:/mnt/qcow2# for mode in read write; do for jobs in 1 16; do for bs in 4 64; do for t in "" rand; do printf "%2i qd %2ik % 4s " $jobs $bs $t; fio --name=random-write --rw=$t$mode --bs=${bs}k --numjobs=1 --size=1g --iodepth=$jobs --runtime=10 --time_based --direct=1 --ioengine=libaio|grep -e BW -e runt ; done; done; done; done
 1 qd  4k        read: IOPS=5866, BW=22.9MiB/s (24.0MB/s)(229MiB/10001msec)
 1 qd  4k rand   read: IOPS=4000, BW=15.6MiB/s (16.4MB/s)(156MiB/10001msec)
 1 qd 64k        read: IOPS=2229, BW=139MiB/s (146MB/s)(1394MiB/10001msec)
 1 qd 64k rand   read: IOPS=2161, BW=135MiB/s (142MB/s)(1351MiB/10001msec)
16 qd  4k        read: IOPS=16.9k, BW=66.2MiB/s (69.4MB/s)(662MiB/10001msec)
16 qd  4k rand   read: IOPS=17.6k, BW=68.8MiB/s (72.1MB/s)(688MiB/10001msec)
16 qd 64k        read: IOPS=7244, BW=453MiB/s (475MB/s)(4529MiB/10002msec)
16 qd 64k rand   read: IOPS=6994, BW=437MiB/s (458MB/s)(4372MiB/10002msec)
 1 qd  4k        write: IOPS=5551, BW=21.7MiB/s (22.7MB/s)(217MiB/10001msec); 0 zone resets
 1 qd  4k rand   write: IOPS=5159, BW=20.2MiB/s (21.1MB/s)(202MiB/10001msec); 0 zone resets
 1 qd 64k        write: IOPS=4024, BW=252MiB/s (264MB/s)(2515MiB/10001msec); 0 zone resets
 1 qd 64k rand   write: IOPS=4027, BW=252MiB/s (264MB/s)(2517MiB/10001msec); 0 zone resets
16 qd  4k        write: IOPS=14.5k, BW=56.8MiB/s (59.6MB/s)(568MiB/10002msec); 0 zone resets
16 qd  4k rand   write: IOPS=14.0k, BW=54.7MiB/s (57.4MB/s)(547MiB/10001msec); 0 zone resets
16 qd 64k        write: IOPS=6360, BW=398MiB/s (417MB/s)(3976MiB/10002msec); 0 zone resets
16 qd 64k rand   write: IOPS=6090, BW=381MiB/s (399MB/s)(3807MiB/10002msec); 0 zone resets

I would be interested to know if I'm overlooking something, or if the qcow2 format simply provides lower performance compared to VHD for the time being?

@anthoinebThe problem still persists; when can we expect it to be resolved?

@florent The following error occurred:

Mar 16 18:38:10 XOA xo-server[4001]: 2026-03-16T17:38:10. 246Z xo:rest-api:error-handler INFO [GET] /vms/0691be81-7ce9-7dba-9387-5620f8e0c52f/vdis (404)

XO version: Master, commit 15917
xcp-ng version: 8.3 with the latest updates.

What’s interesting is that there are two xcp-ng servers, and the problem only occurs on one of them.

@FritzGerald workaround

yum update --exclude=net-snmp*

@kagbasi-ngc

RHEL 10.x only supports CPUs from amd64_v3 onwards, while yours is amd64_v2. Rocky / Almalinux builds for even older CPUs.

https://xcp-ng.org/forum/topic/11447/almalinux-10-dvd-won-t-boot-in-uefi-mode/2

@ashinobi Known issue ...
https://xcp-ng.org/forum/topic/11538/failed-to-migrate-vdi-two-local-sr/5?_=1763186433370

Which ISO are you booting?
The default Alamlinux 10 only supports CPUs from amd64 v3 upwards. You need to download the amd64 v2 installer separately.

@kiu What CPU does xcp-ng run on?

@hoh
How can I boot it, because so far I have not been able to. I tried PV-bootloader=pygrub but it fails.
xl dmesg output

(XEN) [939408.303345] d47v0 Triple fault - invoking HVM shutdown action 3
(XEN) [939408.303347] *** Dumping Dom47 vcpu#0 state: ***
(XEN) [939408.303350] ----[ Xen-4.17.5-13  x86_64  debug=n  Not tainted ]----
(XEN) [939408.303351] CPU:    0
(XEN) [939408.303352] RIP:    0008:[<000000000367a562>]
(XEN) [939408.303354] RFLAGS: 0000000000010046   CONTEXT: hvm guest (d47v0)
(XEN) [939408.303356] rax: 0000000000000000   rbx: 00000000566e6558   rcx: 0000000000000000
(XEN) [939408.303357] rdx: 00000000000000e9   rsi: ffffffff849600a6   rdi: 0000000000000004
(XEN) [939408.303359] rbp: 0000000040000000   rsp: 000000000373cf70   r8:  65584d4d566e6558
(XEN) [939408.303360] r9:  0000000000000026   r10: 6920485650206e65   r11: 7a696c616974696e
(XEN) [939408.303361] r12: 0000000000000005   r13: 0000000000000000   r14: 0000000000000000
(XEN) [939408.303362] r15: 0000000000000000   cr0: 0000000080000011   cr4: 0000000000000020
(XEN) [939408.303363] cr3: 0000000002c10000   cr2: 0000000000000000
(XEN) [939408.303364] fsb: 0000000000000000   gsb: 0000000003738f80   gss: 0000000000000000
(XEN) [939408.303366] ds: 0010   es: 0010   fs: 0000   gs: 0000   ss: 0010   cs: 0008

Since the xcp-ng 8.3 update, there is again no console for PVH VMs, neither XO nor XO Lite.

@florent said in XO VM Backup Started and never completed:

this look like the most recent backup job did not use CBT , but the previous was using it
is it intendend ?

Válasz

This backup has been running like this for a long time, until yesterday's XO update. It has not been configured since then.

Same issue, here is xo-server logs:

Jul 01 04:00:07 xoa xo-server[15851]: }
Jul 01 04:00:07 xoa xo-server[15851]:   }
Jul 01 04:00:07 xoa xo-server[15851]:     code: 'UND_ERR_ABORTED'
Jul 01 04:00:07 xoa xo-server[15851]:       at async Promise.all (index 1) {
Jul 01 04:00:07 xoa xo-server[15851]:       at async file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/backups/_incrementalVm.mjs:67:5
Jul 01 04:00:07 xoa xo-server[15851]:       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jul 01 04:00:07 xoa xo-server[15851]:       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/Xapi.mjs:194:18)
Jul 01 04:00:07 xoa xo-server[15851]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Jul 01 04:00:07 xoa xo-server[15851]:       at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
Jul 01 04:00:07 xoa xo-server[15851]:       at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
Jul 01 04:00:07 xoa xo-server[15851]:       at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
Jul 01 04:00:07 xoa xo-server[15851]:       at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
Jul 01 04:00:07 xoa xo-server[15851]:       at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301858/node_modules/undici/lib/api/readable.js:51:13)
Jul 01 04:00:07 xoa xo-server[15851]:   error: RequestAbortedError [AbortError]: Request aborted
Jul 01 04:00:07 xoa xo-server[15851]: 2025-07-01T02:00:07.508Z xo:backups:worker WARN possibly unhandled rejection {
Jul 01 04:00:07 xoa xo-server[15851]: 2025-07-01T02:00:07.484Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jul 01 04:00:06 xoa xo-server[15851]: }
Jul 01 04:00:06 xoa xo-server[15851]:   }
Jul 01 04:00:06 xoa xo-server[15851]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 141))((process xapi)(f>
Jul 01 04:00:06 xoa xo-server[15851]:     subtasks: [],
Jul 01 04:00:06 xoa xo-server[15851]:     subtask_of: 'OpaqueRef:NULL',
Jul 01 04:00:06 xoa xo-server[15851]:     other_config: {},
Jul 01 04:00:06 xoa xo-server[15851]:     ],
Jul 01 04:00:06 xoa xo-server[15851]:       ''
Jul 01 04:00:06 xoa xo-server[15851]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Jul 01 04:00:06 xoa xo-server[15851]:       '',
Jul 01 04:00:06 xoa xo-server[15851]:       'SR_BACKEND_FAILURE_460',
Jul 01 04:00:06 xoa xo-server[15851]:     error_info: [
Jul 01 04:00:06 xoa xo-server[15851]:     result: '',
Jul 01 04:00:06 xoa xo-server[15851]:     type: '<none/>',
Jul 01 04:00:06 xoa xo-server[15851]:     progress: 1,
Jul 01 04:00:06 xoa xo-server[15851]:     resident_on: 'OpaqueRef:a08170fe-0f8d-4a9d-b890-b65bfa0f5aeb',
Jul 01 04:00:06 xoa xo-server[15851]:     status: 'failure',
Jul 01 04:00:06 xoa xo-server[15851]:     finished: '20250701T02:00:06Z',
Jul 01 04:00:06 xoa xo-server[15851]:     created: '20250701T02:00:05Z',
Jul 01 04:00:06 xoa xo-server[15851]:     current_operations: {},
Jul 01 04:00:06 xoa xo-server[15851]:     allowed_operations: [],
Jul 01 04:00:06 xoa xo-server[15851]:     name_description: '',
Jul 01 04:00:06 xoa xo-server[15851]:     name_label: 'Async.VDI.list_changed_blocks',
Jul 01 04:00:06 xoa xo-server[15851]:     uuid: 'f6981d73-138e-3fac-c944-1218e9f54934',
Jul 01 04:00:06 xoa xo-server[15851]:   task: task {
Jul 01 04:00:06 xoa xo-server[15851]:   url: undefined,
Jul 01 04:00:06 xoa xo-server[15851]:   call: undefined,
Jul 01 04:00:06 xoa xo-server[15851]:   ],
Jul 01 04:00:06 xoa xo-server[15851]:     ''
Jul 01 04:00:06 xoa xo-server[15851]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Jul 01 04:00:06 xoa xo-server[15851]:     '',
Jul 01 04:00:06 xoa xo-server[15851]:   params: [
Jul 01 04:00:06 xoa xo-server[15851]:   code: 'SR_BACKEND_FAILURE_460',
Jul 01 04:00:06 xoa xo-server[15851]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
Jul 01 04:00:06 xoa xo-server[15851]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1270:14)
Jul 01 04:00:06 xoa xo-server[15851]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1097:12)
Jul 01 04:00:06 xoa xo-server[15851]:     at Array.forEach (<anonymous>)
Jul 01 04:00:06 xoa xo-server[15851]:     at file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1107:14
Jul 01 04:00:06 xoa xo-server[15851]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1073:24)
Jul 01 04:00:06 xoa xo-server[15851]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/_getTaskResult.mjs:13:29)
Jul 01 04:00:06 xoa xo-server[15851]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/_XapiError.mjs:16:12)
Jul 01 04:00:06 xoa xo-server[15851]: 2025-07-01T02:00:06.151Z @xen-orchestra/xapi/disks/Xapi INFO openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for give>

@olivierlambert I updated the XO from source, it looks good for now.

@olivierlambert

downgraded to version 20.19.2 . Unfortunately no change.

@olivierlambert

root@xoa:~# node -v
v22.16.0

@olivierlambert
I will check with XOA. For now, it seems to me to be a local problem, because I use quite a few XOs and I haven't had this problem yet. If I find something I will report it.

@olivierlambert

None. The XCP-ng host also runs XO. Same VLAN, same IP range (XCP-ng 172.31.30.3, XO: 172.31.30.4). So the packet doesn't even need to go out to a switch.

The XO has an old Xenserver 7.0 (I'm migrating from it), it runs stable, its IP is 172.31.30.5.

Hello !

When using a freshly installed xcp-ng 8.3 and source compiled XO, the connection between XO and XCP-ng is always broken.

I have attached the log:

un 03 12:45:24 xoa xo-server[572]: 2025-06-03T10:45:24.140Z xo:main INFO + Console proxy (bogi.kornel - ::ffff:172.31.60.45)
Jun 03 12:46:00 xoa xo-server[572]: _watchEvents SocketError: other side closed
Jun 03 12:46:00 xoa xo-server[572]:     at TLSSocket.<anonymous> (/opt/xo5/xo-builds/xen-orchestra-202506021322/node_modules/undici/lib/dispatcher/client-h1.js:701:24)
Jun 03 12:46:00 xoa xo-server[572]:     at TLSSocket.emit (node:events:530:35)
Jun 03 12:46:00 xoa xo-server[572]:     at TLSSocket.patchedEmit [as emit] (/opt/xo5/xo-builds/xen-orchestra-202506021322/@xen-orchestra/log/configure.js:52:17)
Jun 03 12:46:00 xoa xo-server[572]:     at endReadableNT (node:internal/streams/readable:1698:12)
Jun 03 12:46:00 xoa xo-server[572]:     at processTicksAndRejections (node:internal/process/task_queues:90:21) {
Jun 03 12:46:00 xoa xo-server[572]:   code: 'UND_ERR_SOCKET',
Jun 03 12:46:00 xoa xo-server[572]:   socket: {
Jun 03 12:46:00 xoa xo-server[572]:     localAddress: '172.31.30.4',
Jun 03 12:46:00 xoa xo-server[572]:     localPort: 58826,
Jun 03 12:46:00 xoa xo-server[572]:     remoteAddress: '172.31.30.3',
Jun 03 12:46:00 xoa xo-server[572]:     remotePort: 443,
Jun 03 12:46:00 xoa xo-server[572]:     remoteFamily: 'IPv4',
Jun 03 12:46:00 xoa xo-server[572]:     timeout: undefined,
Jun 03 12:46:00 xoa xo-server[572]:     bytesWritten: 49957,
Jun 03 12:46:00 xoa xo-server[572]:     bytesRead: 1673042
Jun 03 12:46:00 xoa xo-server[572]:   },
Jun 03 12:46:00 xoa xo-server[572]:   call: {
Jun 03 12:46:00 xoa xo-server[572]:     duration: 13,
Jun 03 12:46:00 xoa xo-server[572]:     method: 'event.from',
Jun 03 12:46:00 xoa xo-server[572]:     params: [
Jun 03 12:46:00 xoa xo-server[572]:       '* session id *',
Jun 03 12:46:00 xoa xo-server[572]:       [Array],
Jun 03 12:46:00 xoa xo-server[572]:       '00000000000001936776,00000000000001936233',
Jun 03 12:46:00 xoa xo-server[572]:       60.1
Jun 03 12:46:00 xoa xo-server[572]:     ]
Jun 03 12:46:00 xoa xo-server[572]:   }
Jun 03 12:46:00 xoa xo-server[572]: }
Jun 03 12:46:00 xoa xo-server[572]: 2025-06-03T10:46:00.438Z xo:main INFO - Console proxy (bogi.kornel - ::ffff:172.31.60.45)