bogikornel

bogikornel

Hello !
Not enough CPU and not enough RAM, because the RAM is 32G (dynamic max) and the CPU is 16 (8 socket 2 core). The host has max 8 CPU and 27G ram.

bogikornel

Unfortunately, I also get "body timeout error" errors during backups. I run several xen-orchestra servers, but it only occurs where the version is up to date. It also occurs with random VMs, and even with metadata backups. I've attached this morning's log. [xo-server.log](Invalid MIME type)

bogikornel

Which ISO are you booting?
The default Alamlinux 10 only supports CPUs from amd64 v3 upwards. You need to download the amd64 v2 installer separately.

bogikornel

@anthoinebThe problem still persists; when can we expect it to be resolved?

bogikornel

Hello !
I'm a senior linux/unix administrator/system engineer and I'm also working with networks (L2/L3).
I first started using virtualization in 2006 on a Debian/xen basis, then Xenserver 5.5 -> 6.0.2 -> 6.2 -> 7.2 -> xcp-ng ...
I've also been using xen-orchestra since the first public version.

bogikornel

@andSmv thanks for the reply
The server is a home test, power consumption is more important than stability, so I'll try to recompile xen.I guess it will be enough to recompile only version 4.13.4 ?

PS.: I just looked at the source code and I don't see that there is a switch to turn it off. I will modify mwait-idle.c to see if it works

PS2: Done :

[09:50 home-xcp ~]# xenpm get-cpuidle-states 0
Max possible C-state: C9

cpu id               : 0
total C-states       : 5
idle time(ms)        : 134037
C0                   : transition [                5351]
                       residency  [                6507 ms]
C1                   : transition [                  13]
                       residency  [                  15 ms]
C2                   : transition [                  71]
                       residency  [                 120 ms]
C3                   : transition [                 701]
                       residency  [                6956 ms]
C4                   : transition [                4566]
                       residency  [              120437 ms]
pc3                  : [               13371 ms]
pc6                  : [               23594 ms]
cc3                  : [               14446 ms]
cc6                  : [               63719 ms]

The intel.c had to be modified to be correct. Let's see how stable it will be.

bogikornel

@olivierlambert I updated the XO from source, it looks good for now.

bogikornel

@jsox79
Initrd does not include the xen drivers. You need to regenerate and you will be fine.
boot with sysrescue CD, and in a chroot olt environment you can create a new initrd with the xen drivers.

bogikornel

@john-c

I installed from source
Xen Orchestra, commit 85696
8.2.1
I looked at the log, I didn't try anything.

full log:

márc 18 14:07:08 xoa xo-server[22559]: 2024-03-18T13:07:08.398Z xo:xapi WARN importVdiContent:  {
márc 18 14:07:08 xoa xo-server[22559]:   error: Error: self-signed certificate
márc 18 14:07:08 xoa xo-server[22559]:       at TLSSocket.onConnectSecure (node:_tls_wrap:1550:34)
márc 18 14:07:08 xoa xo-server[22559]:       at TLSSocket.emit (node:events:514:28)
márc 18 14:07:08 xoa xo-server[22559]:       at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202403181341/@xen-orchestra/log/configure.js:52:17)
márc 18 14:07:08 xoa xo-server[22559]:       at TLSSocket._finishInit (node:_tls_wrap:967:8)
márc 18 14:07:08 xoa xo-server[22559]:       at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:743:12)
márc 18 14:07:08 xoa xo-server[22559]:       at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
márc 18 14:07:08 xoa xo-server[22559]:     code: 'DEPTH_ZERO_SELF_SIGNED_CERT',
márc 18 14:07:08 xoa xo-server[22559]:     originalUrl: 'https://192.168.200.42/import_raw_vdi/?format=raw&vdi=OpaqueRef%3A5dfc22a9-6b39-43f1-9ff5-ec5380deb77f&session_id=OpaqueRef%3Aa71a87ce-8161-47d9-a732-09ac68767b77&task_id=OpaqueRef%3Aaf48a4d5-f585-4081-a4ce-dc8b6d76847c',
márc 18 14:07:08 xoa xo-server[22559]:     url: 'https://192.168.200.42/import_raw_vdi/?format=raw&vdi=OpaqueRef%3A5dfc22a9-6b39-43f1-9ff5-ec5380deb77f&session_id=OpaqueRef%3Aa71a87ce-8161-47d9-a732-09ac68767b77&task_id=OpaqueRef%3Aaf48a4d5-f585-4081-a4ce-dc8b6d76847c',
márc 18 14:07:08 xoa xo-server[22559]:     pool_master: host {
márc 18 14:07:08 xoa xo-server[22559]:       uuid: '4cec7b52-32a0-4bb0-8dcb-20429fd703a2',
márc 18 14:07:08 xoa xo-server[22559]:       name_label: 'testpool-xcpng',
márc 18 14:07:08 xoa xo-server[22559]:       name_description: 'Default install',
márc 18 14:07:08 xoa xo-server[22559]:       memory_overhead: 2050654208,
márc 18 14:07:08 xoa xo-server[22559]:       allowed_operations: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       current_operations: {},
márc 18 14:07:08 xoa xo-server[22559]:       API_version_major: 2,
márc 18 14:07:08 xoa xo-server[22559]:       API_version_minor: 16,
márc 18 14:07:08 xoa xo-server[22559]:       API_version_vendor: 'XenSource',
márc 18 14:07:08 xoa xo-server[22559]:       API_version_vendor_implementation: {},
márc 18 14:07:08 xoa xo-server[22559]:       enabled: true,
márc 18 14:07:08 xoa xo-server[22559]:       software_version: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       other_config: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       capabilities: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       cpu_configuration: {},
márc 18 14:07:08 xoa xo-server[22559]:       sched_policy: 'credit',
márc 18 14:07:08 xoa xo-server[22559]:       supported_bootloaders: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       resident_VMs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       logging: {},
márc 18 14:07:08 xoa xo-server[22559]:       PIFs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       suspend_image_sr: 'OpaqueRef:6bee4b3d-5a25-482f-8727-0d6264034bef',
márc 18 14:07:08 xoa xo-server[22559]:       crash_dump_sr: 'OpaqueRef:6bee4b3d-5a25-482f-8727-0d6264034bef',
márc 18 14:07:08 xoa xo-server[22559]:       crashdumps: [],
márc 18 14:07:08 xoa xo-server[22559]:       patches: [],
márc 18 14:07:08 xoa xo-server[22559]:       updates: [],
márc 18 14:07:08 xoa xo-server[22559]:       PBDs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       host_CPUs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       cpu_info: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       hostname: 'testpool-xcpng',
márc 18 14:07:08 xoa xo-server[22559]:       address: '192.168.200.42',
márc 18 14:07:08 xoa xo-server[22559]:       metrics: 'OpaqueRef:e108f744-0233-4f77-a8ce-4bf07710e9f8',
márc 18 14:07:08 xoa xo-server[22559]:       license_params: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       ha_statefiles: [],
márc 18 14:07:08 xoa xo-server[22559]:       ha_network_peers: [],
márc 18 14:07:08 xoa xo-server[22559]:       blobs: {},
márc 18 14:07:08 xoa xo-server[22559]:       tags: [],
márc 18 14:07:08 xoa xo-server[22559]:       external_auth_type: '',
márc 18 14:07:08 xoa xo-server[22559]:       external_auth_service_name: '',
márc 18 14:07:08 xoa xo-server[22559]:       external_auth_configuration: {},
márc 18 14:07:08 xoa xo-server[22559]:       edition: 'xcp-ng',
márc 18 14:07:08 xoa xo-server[22559]:       license_server: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       bios_strings: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       power_on_mode: '',
márc 18 14:07:08 xoa xo-server[22559]:       power_on_config: {},
márc 18 14:07:08 xoa xo-server[22559]:       local_cache_sr: 'OpaqueRef:6bee4b3d-5a25-482f-8727-0d6264034bef',
márc 18 14:07:08 xoa xo-server[22559]:       chipset_info: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       PCIs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       PGPUs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       PUSBs: [],
márc 18 14:07:08 xoa xo-server[22559]:       ssl_legacy: false,
márc 18 14:07:08 xoa xo-server[22559]:       guest_VCPUs_params: {},
márc 18 14:07:08 xoa xo-server[22559]:       display: 'enabled',
márc 18 14:07:08 xoa xo-server[22559]:       virtual_hardware_platform_versions: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       control_domain: 'OpaqueRef:01867040-9d1c-4604-97fa-f1f7cde6b2f3',
márc 18 14:07:08 xoa xo-server[22559]:       updates_requiring_reboot: [],
márc 18 14:07:08 xoa xo-server[22559]:       features: [],
márc 18 14:07:08 xoa xo-server[22559]:       iscsi_iqn: 'iqn.2022-12.com.example:b6b66d4e',
márc 18 14:07:08 xoa xo-server[22559]:       multipathing: false,
márc 18 14:07:08 xoa xo-server[22559]:       uefi_certificates: '',
márc 18 14:07:08 xoa xo-server[22559]:       certificates: [],
márc 18 14:07:08 xoa xo-server[22559]:       editions: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       https_only: false
márc 18 14:07:08 xoa xo-server[22559]:     },
márc 18 14:07:08 xoa xo-server[22559]:     SR: SR {
márc 18 14:07:08 xoa xo-server[22559]:       uuid: '347ec16f-d788-d2e0-eaf7-4981aae3f352',
márc 18 14:07:08 xoa xo-server[22559]:       name_label: 'Local storage',
márc 18 14:07:08 xoa xo-server[22559]:       name_description: '',
márc 18 14:07:08 xoa xo-server[22559]:       allowed_operations: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       current_operations: {},
márc 18 14:07:08 xoa xo-server[22559]:       VDIs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       PBDs: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       virtual_allocation: 756061044736,
márc 18 14:07:08 xoa xo-server[22559]:       physical_utilisation: 169835184128,
márc 18 14:07:08 xoa xo-server[22559]:       physical_size: 349799133184,
márc 18 14:07:08 xoa xo-server[22559]:       type: 'ext',
márc 18 14:07:08 xoa xo-server[22559]:       content_type: 'user',
márc 18 14:07:08 xoa xo-server[22559]:       shared: false,
márc 18 14:07:08 xoa xo-server[22559]:       other_config: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       tags: [],
márc 18 14:07:08 xoa xo-server[22559]:       sm_config: [Object],
márc 18 14:07:08 xoa xo-server[22559]:       blobs: {},
márc 18 14:07:08 xoa xo-server[22559]:       local_cache_enabled: true,
márc 18 14:07:08 xoa xo-server[22559]:       introduced_by: 'OpaqueRef:NULL',
márc 18 14:07:08 xoa xo-server[22559]:       clustered: false,
márc 18 14:07:08 xoa xo-server[22559]:       is_tools_sr: false
márc 18 14:07:08 xoa xo-server[22559]:     },
márc 18 14:07:08 xoa xo-server[22559]:     VDI: VDI {
márc 18 14:07:08 xoa xo-server[22559]:       uuid: 'bebc77f5-f68d-4422-8c6d-79b9d230b755',
márc 18 14:07:08 xoa xo-server[22559]:       name_label: 'XO CloudConfigDrive',
márc 18 14:07:08 xoa xo-server[22559]:       name_description: '',
márc 18 14:07:08 xoa xo-server[22559]:       allowed_operations: [Array],
márc 18 14:07:08 xoa xo-server[22559]:       current_operations: {},
márc 18 14:07:08 xoa xo-server[22559]:       SR: 'OpaqueRef:6bee4b3d-5a25-482f-8727-0d6264034bef',
márc 18 14:07:08 xoa xo-server[22559]:       VBDs: [],
márc 18 14:07:08 xoa xo-server[22559]:       crash_dumps: [],
márc 18 14:07:08 xoa xo-server[22559]:       virtual_size: 10485760,
márc 18 14:07:08 xoa xo-server[22559]:       physical_utilisation: 3584,
márc 18 14:07:08 xoa xo-server[22559]:       type: 'user',
márc 18 14:07:08 xoa xo-server[22559]:       sharable: false,
márc 18 14:07:08 xoa xo-server[22559]:       read_only: false,
márc 18 14:07:08 xoa xo-server[22559]:       other_config: {},
márc 18 14:07:08 xoa xo-server[22559]:       storage_lock: false,
márc 18 14:07:08 xoa xo-server[22559]:       location: 'bebc77f5-f68d-4422-8c6d-79b9d230b755',
márc 18 14:07:08 xoa xo-server[22559]:       managed: true,
márc 18 14:07:08 xoa xo-server[22559]:       missing: false,
márc 18 14:07:08 xoa xo-server[22559]:       parent: 'OpaqueRef:NULL',
márc 18 14:07:08 xoa xo-server[22559]:       xenstore_data: {},
márc 18 14:07:08 xoa xo-server[22559]:       sm_config: {},
márc 18 14:07:08 xoa xo-server[22559]:       is_a_snapshot: false,
márc 18 14:07:08 xoa xo-server[22559]:       snapshot_of: 'OpaqueRef:NULL',
márc 18 14:07:08 xoa xo-server[22559]:       snapshots: [],
márc 18 14:07:08 xoa xo-server[22559]:       snapshot_time: '19700101T00:00:00Z',
márc 18 14:07:08 xoa xo-server[22559]:       tags: [],
márc 18 14:07:08 xoa xo-server[22559]:       allow_caching: false,
márc 18 14:07:08 xoa xo-server[22559]:       on_boot: 'persist',
márc 18 14:07:08 xoa xo-server[22559]:       metadata_of_pool: '',
márc 18 14:07:08 xoa xo-server[22559]:       metadata_latest: false,
márc 18 14:07:08 xoa xo-server[22559]:       is_tools_iso: false,
márc 18 14:07:08 xoa xo-server[22559]:       cbt_enabled: false
márc 18 14:07:08 xoa xo-server[22559]:     }
márc 18 14:07:08 xoa xo-server[22559]:   }
márc 18 14:07:08 xoa xo-server[22559]: }

bogikornel

@poddingue said:

Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out".

The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

bogikornel

@pierrebrunet I'll try to collect the logs.
However, I can't even backup VMs. What's interesting is that the error always occurs with only 1 VM or 1 Metadata backup. This morning's backup report:
Job ID: 3c17f0c1-85f5-4b98-b43a-0d117c812a9a
Run ID: 1783461900005
Mode: full
Start time: Wednesday, July 8th 2026, 12:05:00 am
End time: Wednesday, July 8th 2026, 1:22:44 am
Duration: an hour
Successes: 14 / 15
Transfer size: 276.37 GiB

Or with another host and another XO:
Job ID: 53c582e5-ba01-4a82-8a45-948a991add99
Job name: metadata
Run ID: 1783483200004
Start time: Wednesday, July 8th 2026, 6:00:00 am
End time: Wednesday, July 8th 2026, 6:05:00 am
Duration: 5 minutes
Successes: 5 / 6
Error: backup task failed with undefined error

bogikornel

@pierrebrunet said:

@bogikornel Can you tell us on which versions you are please?
the ones where it works and the ones where it does not work.
Thanks for your help!

Not working: commit bb4da build date: 2026.07.01
Not working: commit 3bc70 build date: 2026.07.02
Working:: commit 71fa8 build date: ~ 2025.11.29
Working: commit 96b76 build date: ~ 2024.07.22

bogikornel

@Bambos It's possible, but with the old version, there was no such problem, and nothing else changed except xen-orchestra.

bogikornel

Unfortunately, I also get "body timeout error" errors during backups. I run several xen-orchestra servers, but it only occurs where the version is up to date. It also occurs with random VMs, and even with metadata backups. I've attached this morning's log. [xo-server.log](Invalid MIME type)

bogikornel

@pkgw I tested it with a cluster size of 2 megabytes. I got similar results to those with the default size.

bogikornel

@olivierlambert said:

Might be interesting to test a different cluster size and see the impact

I tested it with a cluster size of 2 megabytes, and nothing changed

bogikornel

@stormi XCP-ng QCOW2 vs. VHD Performance Feedback on NVMe
First of all, I would like to thank the team for all the hard work in bringing QCOW2 support to a production-ready state. It is a very welcome feature.

I have performed some quick I/O benchmarks comparing the new QCOW2 format against the traditional VHD. In my tests, QCOW2 appears significantly slower than VHD on my hardware.

Test Environment
Hypervisor: Dell PowerEdge R420

CPU: Intel Xeon E5-2470 v2

Storage: Intel SSDPELKX010T8 NVMe

VM OS: Debian 13

VM Specs: 2 vCPUs, 1GB RAM

Setup: One 10GB VHD and one 10GB QCOW2 disk, both pre-filled from /dev/random.

Methodology
I used a custom test suite available here: https://vm01.unsoft.hu/~ventura/fio/fio_test_20250408.tar.gz

I also ran a simplefio loop with the following results:
VHD:

root@Debian-13-CloudInit-20250810:/mnt/vhd# for mode in read write; do for jobs in 1 16; do for bs in 4 64; do for t in "" rand; do printf "%2i qd %2ik % 4s " $jobs $bs $t; fio --name=random-write --rw=$t$mode --bs=${bs}k --numjobs=1 --size=1g --iodepth=$jobs --runtime=10 --time_based --direct=1 --ioengine=libaio|grep -e BW -e runt ; done; done; done; done
 1 qd  4k        read: IOPS=9625, BW=37.6MiB/s (39.4MB/s)(376MiB/10001msec)
 1 qd  4k rand   read: IOPS=5414, BW=21.2MiB/s (22.2MB/s)(212MiB/10001msec)
 1 qd 64k        read: IOPS=2657, BW=166MiB/s (174MB/s)(1661MiB/10001msec)
 1 qd 64k rand   read: IOPS=2575, BW=161MiB/s (169MB/s)(1610MiB/10001msec)
16 qd  4k        read: IOPS=45.7k, BW=178MiB/s (187MB/s)(1785MiB/10001msec)
16 qd  4k rand   read: IOPS=45.9k, BW=179MiB/s (188MB/s)(1794MiB/10001msec)
16 qd 64k        read: IOPS=16.7k, BW=1041MiB/s (1092MB/s)(10.2GiB/10001msec)
16 qd 64k rand   read: IOPS=16.7k, BW=1042MiB/s (1093MB/s)(10.2GiB/10001msec)
 1 qd  4k        write: IOPS=8842, BW=34.5MiB/s (36.2MB/s)(345MiB/10001msec); 0 zone resets
 1 qd  4k rand   write: IOPS=8880, BW=34.7MiB/s (36.4MB/s)(347MiB/10001msec); 0 zone resets
 1 qd 64k        write: IOPS=6095, BW=381MiB/s (399MB/s)(3810MiB/10001msec); 0 zone resets
 1 qd 64k rand   write: IOPS=6006, BW=375MiB/s (394MB/s)(3755MiB/10001msec); 0 zone resets
16 qd  4k        write: IOPS=49.3k, BW=193MiB/s (202MB/s)(1928MiB/10001msec); 0 zone resets
16 qd  4k rand   write: IOPS=47.3k, BW=185MiB/s (194MB/s)(1848MiB/10001msec); 0 zone resets
16 qd 64k        write: IOPS=14.3k, BW=891MiB/s (934MB/s)(8910MiB/10001msec); 0 zone resets
16 qd 64k rand   write: IOPS=15.5k, BW=966MiB/s (1013MB/s)(9663MiB/10001msec); 0 zone resets

QCOW2

root@Debian-13-CloudInit-20250810:/mnt/qcow2# for mode in read write; do for jobs in 1 16; do for bs in 4 64; do for t in "" rand; do printf "%2i qd %2ik % 4s " $jobs $bs $t; fio --name=random-write --rw=$t$mode --bs=${bs}k --numjobs=1 --size=1g --iodepth=$jobs --runtime=10 --time_based --direct=1 --ioengine=libaio|grep -e BW -e runt ; done; done; done; done
 1 qd  4k        read: IOPS=5866, BW=22.9MiB/s (24.0MB/s)(229MiB/10001msec)
 1 qd  4k rand   read: IOPS=4000, BW=15.6MiB/s (16.4MB/s)(156MiB/10001msec)
 1 qd 64k        read: IOPS=2229, BW=139MiB/s (146MB/s)(1394MiB/10001msec)
 1 qd 64k rand   read: IOPS=2161, BW=135MiB/s (142MB/s)(1351MiB/10001msec)
16 qd  4k        read: IOPS=16.9k, BW=66.2MiB/s (69.4MB/s)(662MiB/10001msec)
16 qd  4k rand   read: IOPS=17.6k, BW=68.8MiB/s (72.1MB/s)(688MiB/10001msec)
16 qd 64k        read: IOPS=7244, BW=453MiB/s (475MB/s)(4529MiB/10002msec)
16 qd 64k rand   read: IOPS=6994, BW=437MiB/s (458MB/s)(4372MiB/10002msec)
 1 qd  4k        write: IOPS=5551, BW=21.7MiB/s (22.7MB/s)(217MiB/10001msec); 0 zone resets
 1 qd  4k rand   write: IOPS=5159, BW=20.2MiB/s (21.1MB/s)(202MiB/10001msec); 0 zone resets
 1 qd 64k        write: IOPS=4024, BW=252MiB/s (264MB/s)(2515MiB/10001msec); 0 zone resets
 1 qd 64k rand   write: IOPS=4027, BW=252MiB/s (264MB/s)(2517MiB/10001msec); 0 zone resets
16 qd  4k        write: IOPS=14.5k, BW=56.8MiB/s (59.6MB/s)(568MiB/10002msec); 0 zone resets
16 qd  4k rand   write: IOPS=14.0k, BW=54.7MiB/s (57.4MB/s)(547MiB/10001msec); 0 zone resets
16 qd 64k        write: IOPS=6360, BW=398MiB/s (417MB/s)(3976MiB/10002msec); 0 zone resets
16 qd 64k rand   write: IOPS=6090, BW=381MiB/s (399MB/s)(3807MiB/10002msec); 0 zone resets

I would be interested to know if I'm overlooking something, or if the qcow2 format simply provides lower performance compared to VHD for the time being?

bogikornel

@anthoinebThe problem still persists; when can we expect it to be resolved?

bogikornel

@florent The following error occurred:

Mar 16 18:38:10 XOA xo-server[4001]: 2026-03-16T17:38:10. 246Z xo:rest-api:error-handler INFO [GET] /vms/0691be81-7ce9-7dba-9387-5620f8e0c52f/vdis (404)

XO version: Master, commit 15917
xcp-ng version: 8.3 with the latest updates.

What’s interesting is that there are two xcp-ng servers, and the problem only occurs on one of them.