Categories

  • All news regarding Xen and XCP-ng ecosystem

    142 Topics
    4k Posts
    rzrR
    Thank you every visitors and for those who did not had the chance to visit fosdem check this report: https://xcp-ng.org/blog/2026/02/19/fosdem-2026-follow-up/ [image: vates-xcp-ng-at-fosdem-2026-1.webp] One question, I promised to forward to the forum, how many VM are you running on XCP-ng ? Dozen, hundreds, thousands or more ?

  • Everything related to the virtualization platform

    1k Topics
    14k Posts
    olivierlambertO
    The thing is that it works very well and doesn't need fixes… I know, not usual right? It was so stable that we left it like that waiting for people coming for bugs and nothing happened… We'll cut a 1.0 in the next weeks/months.

  • All Xen Orchestra related questions/reports

    3k Topics
    27k Posts
    F
    Since installing todays XOA appliance update i am noticing strange behaviour on multiple XO instances while backups are running. (2 XOA and my own XOCE in my home lab) A number of VMs will display the error attached and then run a full backup. These jobs have been running flawlessly for quite sometime now. This wipes out the whole backup chain and leaves you with a single full backup in the end. [image: 1772166480795-screenshot-2026-02-26-at-8.15.09-pm.png] Looking at sudo journalctl -u xo-server -f -n 50 on XOA i see the following happening during the merge of each VM Feb 26 21:20:24 xoa xo-server[1731622]: 2026-02-27T03:20:24.207Z xo:backups:mergeWorker INFO merge in progress { Feb 26 21:20:24 xoa xo-server[1731622]: done: 2647, Feb 26 21:20:24 xoa xo-server[1731622]: parent: '/xo-vm-backups/d6befe36-fea7-04f3-25f1-feada684702b/vdis/8ca7c07e-4509-463d-8867-87e6b2026ad6/5afe2bcd-42d4-4b58-ba95-2c77e5107692/20260207T014453Z.vhd', Feb 26 21:20:24 xoa xo-server[1731622]: progress: 93, Feb 26 21:20:24 xoa xo-server[1731622]: total: 2845 Feb 26 21:20:24 xoa xo-server[1731622]: } Feb 26 21:20:32 xoa xo-server[1731622]: 2026-02-27T03:20:32.199Z xo:fs:abstract WARN retrying method on fs { Feb 26 21:20:32 xoa xo-server[1731622]: method: '_write', Feb 26 21:20:32 xoa xo-server[1731622]: attemptNumber: 0, Feb 26 21:20:32 xoa xo-server[1731622]: delay: 100, Feb 26 21:20:32 xoa xo-server[1731622]: error: Error: EBADF: bad file descriptor, write Feb 26 21:20:32 xoa xo-server[1731622]: From: Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.addSyncStackTrace (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:21:26) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler._writeFd (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:209:35) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler._write (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/abstract.js:680:25) Feb 26 21:20:32 xoa xo-server[1731622]: at loopResolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:83:46) Feb 26 21:20:32 xoa xo-server[1731622]: at new Promise (<anonymous>) Feb 26 21:20:32 xoa xo-server[1731622]: at loop (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:85:22) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.retry (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:87:10) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler._write (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:103:18) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.write (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/abstract.js:482:16) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.write (/usr/local/lib/node_modules/xo-server/node_modules/limit-concurrency-decorator/index.js:97:24) { Feb 26 21:20:32 xoa xo-server[1731622]: errno: -9, Feb 26 21:20:32 xoa xo-server[1731622]: code: 'EBADF', Feb 26 21:20:32 xoa xo-server[1731622]: syscall: 'write' Feb 26 21:20:32 xoa xo-server[1731622]: }, Feb 26 21:20:32 xoa xo-server[1731622]: file: { Feb 26 21:20:32 xoa xo-server[1731622]: fd: 17, Feb 26 21:20:32 xoa xo-server[1731622]: path: '/xo-vm-backups/d6befe36-fea7-04f3-25f1-feada684702b/vdis/8ca7c07e-4509-463d-8867-87e6b2026ad6/5afe2bcd-42d4-4b58-ba95-2c77e5107692/20260206T014545Z.vhd' Feb 26 21:20:32 xoa xo-server[1731622]: }, Feb 26 21:20:32 xoa xo-server[1731622]: retryOpts: {} Feb 26 21:20:32 xoa xo-server[1731622]: } Feb 26 21:20:32 xoa xo-server[1731622]: 2026-02-27T03:20:32.200Z vhd-lib:VhdFile INFO double close { Feb 26 21:20:32 xoa xo-server[1731622]: path: { Feb 26 21:20:32 xoa xo-server[1731622]: fd: 18, Feb 26 21:20:32 xoa xo-server[1731622]: path: '/xo-vm-backups/d6befe36-fea7-04f3-25f1-feada684702b/vdis/8ca7c07e-4509-463d-8867-87e6b2026ad6/5afe2bcd-42d4-4b58-ba95-2c77e5107692/20260207T014453Z.vhd' Feb 26 21:20:32 xoa xo-server[1731622]: }, Feb 26 21:20:32 xoa xo-server[1731622]: firstClosedBy: 'Error\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at VhdFile.dispose (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/Vhd/VhdFile.js:134:22)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDisk.dispose (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/Vhd/VhdFile.js:104:26)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDisk.close (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/RemoteVhdDisk.mjs:101:24)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDisk.unlink (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/RemoteVhdDisk.mjs:426:16)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDiskChain.unlink (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/RemoteVhdDiskChain.mjs:268:18)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at #step_cleanup (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/MergeRemoteDisk.mjs:279:23)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at async MergeRemoteDisk.merge (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/MergeRemoteDisk.mjs:159:11)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at async _mergeVhdChain (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_cleanVm.mjs:56:20)', Feb 26 21:20:32 xoa xo-server[1731622]: closedAgainBy: 'Error\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at VhdFile.dispose (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/Vhd/VhdFile.js:129:24)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDisk.dispose (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/Vhd/VhdFile.js:104:26)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDisk.close (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/RemoteVhdDisk.mjs:101:24)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/RemoteVhdDiskChain.mjs:85:52\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at Array.map (<anonymous>)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at RemoteVhdDiskChain.close (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/RemoteVhdDiskChain.mjs:85:35)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at #cleanup (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/disks/MergeRemoteDisk.mjs:309:21)\n' + Feb 26 21:20:32 xoa xo-server[1731622]: ' at async _mergeVhdChain (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_cleanVm.mjs:56:20)' Feb 26 21:20:32 xoa xo-server[1731622]: } Feb 26 21:20:32 xoa xo-server[1731622]: 2026-02-27T03:20:32.274Z xo:backups:mergeWorker FATAL ENOENT: no such file or directory, rename '/run/xo-server/mounts/cd3f089c-7064-412e-86e3-b62c5647021a/xo-vm-backups/.queue/clean-vm/_20260227T020329Z-ukxolxktxyl' -> '/run/xo-server/mounts/cd3f089c-7064-412e-86e3-b62c5647021a/xo-vm-backups/.queue/clean-vm/__20260227T020329Z-ukxolxktxyl' { Feb 26 21:20:32 xoa xo-server[1731622]: error: Error: ENOENT: no such file or directory, rename '/run/xo-server/mounts/cd3f089c-7064-412e-86e3-b62c5647021a/xo-vm-backups/.queue/clean-vm/_20260227T020329Z-ukxolxktxyl' -> '/run/xo-server/mounts/cd3f089c-7064-412e-86e3-b62c5647021a/xo-vm-backups/.queue/clean-vm/__20260227T020329Z-ukxolxktxyl' Feb 26 21:20:32 xoa xo-server[1731622]: From: Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.addSyncStackTrace (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:21:26) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler._rename (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:191:35) Feb 26 21:20:32 xoa xo-server[1731622]: at /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/utils.js:29:26 Feb 26 21:20:32 xoa xo-server[1731622]: at new Promise (<anonymous>) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.<anonymous> (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/utils.js:24:12) Feb 26 21:20:32 xoa xo-server[1731622]: at loopResolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:83:46) Feb 26 21:20:32 xoa xo-server[1731622]: at new Promise (<anonymous>) Feb 26 21:20:32 xoa xo-server[1731622]: at loop (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:85:22) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler.retry (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:87:10) Feb 26 21:20:32 xoa xo-server[1731622]: at LocalHandler._rename (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/retry.js:103:18) { Feb 26 21:20:32 xoa xo-server[1731622]: errno: -2, Feb 26 21:20:32 xoa xo-server[1731622]: code: 'ENOENT', Feb 26 21:20:32 xoa xo-server[1731622]: syscall: 'rename', Feb 26 21:20:32 xoa xo-server[1731622]: path: '/run/xo-server/mounts/cd3f089c-7064-412e-86e3-b62c5647021a/xo-vm-backups/.queue/clean-vm/_20260227T020329Z-ukxolxktxyl', Feb 26 21:20:32 xoa xo-server[1731622]: dest: '/run/xo-server/mounts/cd3f089c-7064-412e-86e3-b62c5647021a/xo-vm-backups/.queue/clean-vm/__20260227T020329Z-ukxolxktxyl' Feb 26 21:20:32 xoa xo-server[1731622]: } Feb 26 21:20:32 xoa xo-server[1731622]: } This files do exist prior to the job running, but are erased during the job execution. Remotes are all NFS based, my home lab and one location are using TrueNAS, another location is using NFS backed by Ubuntu 24.04 LTS. This seems to not make any difference. I submitted a ticket for this since i have pro support but wanted to put this out to the community to see if anyone else is running into this? I have reverted back to 6.1.2 and things are operating smoothly again.

  • Our hyperconverged storage solution

    41 Topics
    717 Posts
    DanpD
    @tmnguyen You can open a support ticket and request that we reactivate your XOSTOR trial licenses to match your existing XOA trial.

  • Non-English speakers

    32 Topics
    94 Posts
    olivierlambertO
    Difficile de parler de « réalité » avec les benchmarks. Vérifiez aussi l'iodepth (qui dépend du type matériel que vous avez, sur du flash/NVMe vous pouvez monter à 128 ou 256), la latence entre en compte aussi bien sûr. Le bottleneck principal est le monothreading de tapdisk, si vous testez sur plusieurs VMs différentes la somme va monter de manière assez régulière.