Posts made by acebmxer | XCP-ng and XO forum

acebmxer

I would advise not backing up the same vms with both XO and veeam. One or the other. No did not see the update yet. I will look and check.

Edit - thanks or pointing out the new version. I was still on 13.1.0.411. Downloading update now.

acebmxer

@zorro

Hello if you have not seen this post - https://xcp-ng.org/forum/topic/12407/veeam-13.1-rocky9-linux-appliance-potential-data-loss-with-cbt-and-workers-with-expired-tokens Myself and another user reported issues with Veeam and CBT.

My latest update from Veeam was no eta on the next update release to fix the issues. That has been some time and there might be a eta. I have not followed back up as of yet.

On thing that seems to have helped was using Active Full backup opposed to Synthetic full backups.

acebmxer

Last night I finished migrating to new to me cisco 25gb switch. Other changes I made was remove an ISCSI SR I was not utilizing and an NFS SR. Also migrated all vms to newly created QCOW2 NFS SR (in XO) and migrated all vms to the new SR. Checked that NBD and CBD were enabled in the backup job. I ran a full backup and it completed successfully.

Just to be sure I decided to re-install xcp-ng 8.3 on both host and re-apply the recent version of the updates. Ran delta backup and it passed.

I dont believe i missed anything in between last failed backup and issue i posted. Maybe I has having issues with the previous 10gb switch that were not visible with out taken deeper look?

If there is something specific you would like for me to check to test. But starting from clean slate / new switch seems to have helped?

acebmxer

@anthoineb
Since I had to physical power cycle to clear the I/O block and recover the host, the live kernel state and any active blocked PIDs from the window of the deadlock were cleared upon reboot.
Checking the host right now, there are currently no processes sitting in an uninterruptible sleep (D) state, and tap-ctl list shows the standard operational mappings for the running VMs:

# tap-ctl list
    5175    0    0        vhd /var/run/sr-mount/4991d4aa-ed84-599b-7d19-97f2f943a366/8d3ce0d6-7d4d-40b0-a99f-09daab8d55e1.vhd
    5720    1    0        vhd /var/run/sr-mount/4991d4aa-ed84-599b-7d19-97f2f943a366/1c200822-5a23-4afa-93cd-d0d89c076191.vhd
    6941    2    0        vhd /var/run/sr-mount/4991d4aa-ed84-599b-7d19-97f2f943a366/0fac203e-4740-4f95-a78c-a3630e591e76.vhd

# ps aux | awk '$8 ~ /D/'
(Returned empty)

Got it regarding the lock retry behavior in SMlog—thanks for clarifying that component. I am pulling the full un-truncated kern.log.1 from yesterday to add to the Google Drive folder so you can inspect the historical kernel call traces right when the backup stream hit the wall.

I have uploaded the raw kern.log.1 file directly to the shared folder.

Let me know if you want to add the exact file name context anywhere else in the message.

acebmxer

@willamcorlin,

If you are using my installer script (https://github.com/acebmxer/install_xen_orchestra), there are two quick ways to verify the active Node.js version. @poddingue can you confirm this is correct?

1. During the Update/Install Process
The script explicitly pulls the active environment version and prints it out against your NODE_VERSION target in the logs:

❯ node -v
v24.22.1

[INFO] Node.js v24.22.1 satisfies configured version 24 — no runtime change needed
[INFO] Installing Node.js 24...
[INFO] Node.js 24 requirement satisfied (installed: v24.22.1)

2. Verifying the Live Service Runtime
To be absolutely certain that the running systemd daemon is executing the correct binary instance after a restart, you can check the status of the xo-server unit:

sudo systemctl status xo-server.service

Look at the CGroup line in the output. It maps directly to the active path and flags (such as the heap memory allocations set by the script):

CGroup: /system.slice/xo-server.service
        └─18540 /usr/bin/node --max-old-space-size=5410 /opt/xen-orchestra/packages/xo-server/dist/cli.mjs

You can then run /usr/bin/node -v (or whatever path is reflected there) to double-check that the active service binary is indeed on the patched release line (v24.22.1, v22.23.4, etc.).

acebmxer

@dinhngtu

Here are the extracted Xen hypervisor, Dom0 kernel, and toolstack/storage manager logs tracking the exact window of the NFS storage deadlock on Oct 27 between 13:00 and 15:00.

During this window, xe task-list was completely blank, and user VMs went into an uninterruptible B H state (turning into zombie D P H domains upon an xl destroy attempt). Physical hardware interfaces showed zero errors or discards via ethtool.

Looking closely at crash_window_SMlog.log, there is a clear locking deadlock right at the start of the backup run (13:00:38):
"Failed to lock /var/lock/sm/96fd7d5a-1985-80be-5776-6ace7f651a41/sr on first attempt, blocked by PID 86371"

It looks like the parallel multi-stream VHD storage exports/NBD operations lock-contended the SR control handles so badly that the underlying sm/blktap layer deadlocked entirely.

I have attached the full log bundle containing:

xen_hypervisor.log (Xen ring buffer / xl dmesg)
crash_window_kernel.log (Dom0 kernel events from /var/log/kern.log)
crash_window_SMlog.log (Storage manager tracks from /var/log/SMlog.1)
xapi_crash.log (Toolstack logs from /var/log/xensource.log.1)

https://drive.google.com/drive/folders/1uCnWXArfgyvS1gPl3mViUl8G1-DBOGzD?usp=sharing - due to file size and type i could not upload here. So i posted it on my google drive.

acebmxer

@lmolina

of course i just downgraded again prior to this release... Sorry for the post. This was assisted with AI. I was also trying to do this while at work . I tried to verify this was not caused be my actions or configuration. If this is the results of upgrading and downgrading then everyelse use this a learning point.

[13:10 xcp-ng-disjqdnc ~]# yum history
Loaded plugins: fastestmirror
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    16 | update --enablerepo=xcp- | 2026-10-27 13:04 | Update         |   35 EE
    15 | history undo 14          | 2026-10-27 10:37 | Downgrade      |   35 EE
    14 | update --enablerepo=xcp- | 2026-10-22 19:00 | Update         |   35 EE
    13 | update --disablerepo=* - | 2026-10-22 15:40 | Update         |    1   
    12 | history undo 11          | 2026-10-22 13:37 | Downgrade      |   35 EE
    11 | update --enablerepo=xcp- | 2026-10-22 05:58 | Update         |   35 EE
    10 | update --disablerepo=* - | 2026-10-13 13:13 | Update         |    6   
     9 | update --disablerepo=* - | 2026-10-05 15:04 | Update         |   20   
     8 | update --enablerepo=xcp- | 2026-10-01 09:12 | Update         |   45 EE
     7 | update --disablerepo=* - | 2026-09-08 16:06 | Update         |    6   
     6 | update --enablerepo=xcp- | 2026-08-26 08:39 | Update         |   23 EE
     5 | update --enablerepo=xcp- | 2026-08-14 13:23 | I, U           |   53 EE
     4 | update                   | 2026-07-28 10:46 | Update         |    5   
     3 | update --enablerepo=xcp- | 2026-07-10 11:27 | Update         |   23 EE
     2 | install -y /root/twinsto | 2026-07-05 20:16 | Install        |   12   
     1 | update --disablerepo=* - | 2026-07-05 01:56 | I, O, U        |  138 EE

history list

updates installed no issues. I did a full backup job prior to new updates. after updates I did a delta backup. (vms are in VHD format)

migrated VMS back to QCOW2 and ran another backup. Because of migration this was a full backup..

@stormi @lmolina

Ran into a massive storage deadlock testing this batch of candidates on my NFS setups with Mellanox ConnectX links.

Here is the exact timeline and raw outputs of what went down:

Clean baseline: Rolled back previous stuff via yum history undo so the host was at a completely clean slate first.
Pre-test: Ran a full backup on standard VHD layout on the stable build. Zero issues, finished perfectly fine.
The update: Upgraded to the testing candidate repo (ID 16 in my history output above).
The trigger: Cleared a delta backup right after, then kicked off a full backup job. No other out-of-band tasks or live migrations were happening.

The Breakdown:
Mid-way through that full VHD backup, the entire storage layer completely locked up.

Toolstack went blind: xe task-list was completely blank. The host totally lost track of the job it was supposed to be running.
VMs deadlocked: Every user VM instantly frozen into a hard Blocked/Halted (B H) state. Trying a low-level xl destroy just turned them into permanent un-killable zombie domains (D P H).

[13:33 xcp-ng-rngyaiub ~]# xl destroy 1
[13:33 xcp-ng-rngyaiub ~]# list_domains
id |                                 uuid |  state
 0 | 2083b5bd-7878-4598-960f-bb425814d338 |     R 
 1 | 49f51614-c105-4fea-3103-566613ec79f2 | D P  H
 2 | dcbfaff1-d782-b42b-6eed-3d6b6d1ca032 | D P  H
 3 | 8cc17a78-ef65-1c06-4ac1-8e56b609d714 | D P  H
 4 | 2f724b50-5b16-9975-839b-fd85dd8ed065 | D P  H

Hardware / Config Diagnostics:
Before pointing fingers at the candidate code, I checked if my Mellanox links or the network switch choked under the backup stream load. Ran ethtool immediately following recovery and the physical interfaces are pristine:

[14:16 xcp-ng-disjqdnc ~]# ethtool -S eth1 | grep -E "discard|drop|filter|error"
     rx_xdp_drop: 0
     tx_queue_dropped: 0
     rx_oversize_pkts_sw_drop: 0
     rx_pet_hdr_lookup_drop: 0
     rx_pet_mdata_lookup_drop: 0
     rx_xsk_xdp_drop: 0
     rx_xsk_oversize_pkts_sw_drop: 0
     rx_crc_errors_phy: 0
     rx_in_range_len_errors_phy: 0
     rx_discards_phy: 0
     tx_discards_phy: 0
     tx_errors_phy: 0
     rx_prio0_discards: 0
     rx_prio1_discards: 0
     rx_prio2_discards: 0
     rx_prio3_discards: 0
     rx_prio4_discards: 0
     rx_prio5_discards: 0
     rx_prio6_discards: 0
     rx_prio7_discards: 0
     tx_pause_storm_error_events: 0

[14:18 xcp-ng-disjqdnc ~]# ethtool -k eth1 | grep -i vlan
rx-vlan-offload: on
tx-vlan-offload: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]

Conclusion:
Network layout config problems, buffer overflows, or silent packet drops are 100% ruled out here (all hardware discards are flat zero).

This wasn't a user mistake or an accidental configuration mismatch. The new sm or blktap components hit a deep kernel lockup handling the multi-stream VHD storage exports. The file locks deadlocked the kernel handles so badly that the hosts ignored standard and low-level CLI commands entirely, forcing a hard out-of-band physical power cycle of the XCP-ng hosts and the TrueNAS storage server just to clear the uninterruptible I/O sleep queue.

I haven't touched a thing since getting the hardware back up and verifying what choked. The system is sitting exactly as it is right now. Let me know what specific debugging paths you want me to look at or what logs you need from that window to trace this down.

acebmxer

@blueh2o

Are you using XOA or XO from sources? If the later what Commit are you on? Did you install the recent beta updates for your hosts?

Can you post the log of the the failed backups.

acebmxer

@rzr

I reinstalled the updates, and confirm backups work with NBD disablled in backup job. - I missed the part about NBD in the notes..

However the task stay at 0% while i can see traffic on my trunas..

acebmxer

@rzr said:

@acebmxer said:
(...)
yum history undo <ID> --enablerepo=xcp-ng-testing
(...)

Your're lucky ! this is a risky operation we don't recommend to downgrade because some packages like xapi are not handling this properly, prefer other workarounds if there are any (like disabling nbd until @storage team suggest a better option)

Let me thank you again @flakpyro for this regression report it is very valuable to us,

I didn't run the commands I stated. As I was asking first.. I did the command @flakpryo

Of course this is was before i saw your post. As this is the home lab i am fine with if it blows up. but for others this is good to know.

acebmxer

@flakpyro

Same i am continuing to have backup issues.. multiple reboots, disabled NBD backup getting stuck.

Is this the correct procedure to downgrade from the updates?

yum history list | head -n 6
yum history undo <ID> --enablerepo=xcp-ng-testing
reboot
rpm -q xapi-core blktap xen-hypervisor sm

Or do you advise doing - yum downgrade blktap-3.55.5-11.3.xcpng8.3 to just downgrade the blktap?

Edit - Actually looks like diabling NBD might have worked... I thought backup was still runing.. no the full backup completed this time and a scheduled delta job is running.... NBD is enabled on the delta job.

acebmxer

@lmolina

installed updates.... by accident i ran the linstore-testing updates. but no issues with the udpate itself.

I did download the snapshot-fixer.py in dry run it found to vms. I ran a full backup first and was going fine till here...

Here are some of the logs - if you have a place i can send the rest as i am not able to post all of them...

tapdisk-28167-daemonlog.txt
tapdisk-probe.txt
tapdisk-28167-fd.txt
tapdisk-28167-threads.txt
versions.txt

Dug into it some more. The stuck process is tapdisk pid 28167. tap-ctl list just hangs and tap-ctl list -p 28167 times out, but the other 3 tapdisks answer fine. It’s in S state, not D, so it doesn’t look like it’s waiting on the disk.

28167 is the one doing the NBD export of the backup snapshot for my Dockerthings_anepi VM (snapshot bf2a422e-9df3-4b3d-83f0-0fa3ff191efb, VHD on SR 4991d4aa...). That’s the export that stopped at 75%. Everything after it is stuck at 0% (VBD.plug, importing raw VDI, the XO-CE-root import), and when I tried force unplug those got stuck too. The backup job said it finished though, and the VM itself is still running fine.

Ran the snapshot-fixer dry run and it found 2 VMs. I did not run the rewrite before doing the backup.

Update went through fine at 05:58, all from xcp-ng-testing, and I rebooted at 06:00, so this happened on the new packages. The linstor-testing thing didn’t install anything, rpm -qa | grep -i linstor comes back empty. Attached the yum history.

I have the full logs (xensource, SMlog, daemon, kern) if you want them, just tell me where to send them.
yum-history-11.txt

Update: kill 28167 (plain SIGTERM, not -9) cleared it, then the stuck tasks cancelled without a reboot. Will post if it happens again on the next backup.

acebmxer

@ma_ron said:

Hi, Xen Orchestra, Community 51bb8 - back up logs and statuses are missing in the GUI. backups are done as seen exist available at the restore page. what can be the issue?

Hello @ma_ron - That commit is 30 commits behind the current master commit. I suggest updating and trying again.

acebmxer

@coolsport00

Glad to help. Yes you can switch back to master and wait for vates to push the changes to master branch. I also started out with ronivays script from Tom Lawrence's video also.

Feel free to check out my script if you like.

acebmxer

@coolsport00

If XOS = Xen Orchestra from sources. what script did you use to install it? My script is here - https://github.com/acebmxer/install_xen_orchestra Possible use used ronivay's script - https://github.com/ronivay/XenOrchestraInstallerUpdater?

I dont remember how to switch branches with ronivays script. You can deploy a new xo with my script. take a backup of your current xo's config from the settings > xo config. Deploy new xo and import the config. To swith branchs on my script use the steps i stated above - edit the xo-config.cfg file from master to xo6/fix-opening-console-in-new-tab and then run update and update to that xo branch.