Categories

  • All news regarding Xen and XCP-ng ecosystem

    146 Topics
    5k Posts
    acebmxerA
    @lmolina of course i just downgraded again prior to this release... Sorry for the post. This was assisted with AI. I was also trying to do this while at work . I tried to verify this was not caused be my actions or configuration. If this is the results of upgrading and downgrading then everyelse use this a learning point. [13:10 xcp-ng-disjqdnc ~]# yum history Loaded plugins: fastestmirror ID | Command line | Date and time | Action(s) | Altered ------------------------------------------------------------------------------- 16 | update --enablerepo=xcp- | 2026-10-27 13:04 | Update | 35 EE 15 | history undo 14 | 2026-10-27 10:37 | Downgrade | 35 EE 14 | update --enablerepo=xcp- | 2026-10-22 19:00 | Update | 35 EE 13 | update --disablerepo=* - | 2026-10-22 15:40 | Update | 1 12 | history undo 11 | 2026-10-22 13:37 | Downgrade | 35 EE 11 | update --enablerepo=xcp- | 2026-10-22 05:58 | Update | 35 EE 10 | update --disablerepo=* - | 2026-10-13 13:13 | Update | 6 9 | update --disablerepo=* - | 2026-10-05 15:04 | Update | 20 8 | update --enablerepo=xcp- | 2026-10-01 09:12 | Update | 45 EE 7 | update --disablerepo=* - | 2026-09-08 16:06 | Update | 6 6 | update --enablerepo=xcp- | 2026-08-26 08:39 | Update | 23 EE 5 | update --enablerepo=xcp- | 2026-08-14 13:23 | I, U | 53 EE 4 | update | 2026-07-28 10:46 | Update | 5 3 | update --enablerepo=xcp- | 2026-07-10 11:27 | Update | 23 EE 2 | install -y /root/twinsto | 2026-07-05 20:16 | Install | 12 1 | update --disablerepo=* - | 2026-07-05 01:56 | I, O, U | 138 EE history list updates installed no issues. I did a full backup job prior to new updates. after updates I did a delta backup. (vms are in VHD format) migrated VMS back to QCOW2 and ran another backup. Because of migration this was a full backup.. @stormi @lmolina Ran into a massive storage deadlock testing this batch of candidates on my NFS setups with Mellanox ConnectX links. Here is the exact timeline and raw outputs of what went down: Clean baseline: Rolled back previous stuff via yum history undo so the host was at a completely clean slate first. Pre-test: Ran a full backup on standard VHD layout on the stable build. Zero issues, finished perfectly fine. The update: Upgraded to the testing candidate repo (ID 16 in my history output above). The trigger: Cleared a delta backup right after, then kicked off a full backup job. No other out-of-band tasks or live migrations were happening. The Breakdown: Mid-way through that full VHD backup, the entire storage layer completely locked up. Toolstack went blind: xe task-list was completely blank. The host totally lost track of the job it was supposed to be running. VMs deadlocked: Every user VM instantly frozen into a hard Blocked/Halted (B H) state. Trying a low-level xl destroy just turned them into permanent un-killable zombie domains (D P H). [13:33 xcp-ng-rngyaiub ~]# xl destroy 1 [13:33 xcp-ng-rngyaiub ~]# list_domains id | uuid | state 0 | 2083b5bd-7878-4598-960f-bb425814d338 | R 1 | 49f51614-c105-4fea-3103-566613ec79f2 | D P H 2 | dcbfaff1-d782-b42b-6eed-3d6b6d1ca032 | D P H 3 | 8cc17a78-ef65-1c06-4ac1-8e56b609d714 | D P H 4 | 2f724b50-5b16-9975-839b-fd85dd8ed065 | D P H Hardware / Config Diagnostics: Before pointing fingers at the candidate code, I checked if my Mellanox links or the network switch choked under the backup stream load. Ran ethtool immediately following recovery and the physical interfaces are pristine: [14:16 xcp-ng-disjqdnc ~]# ethtool -S eth1 | grep -E "discard|drop|filter|error" rx_xdp_drop: 0 tx_queue_dropped: 0 rx_oversize_pkts_sw_drop: 0 rx_pet_hdr_lookup_drop: 0 rx_pet_mdata_lookup_drop: 0 rx_xsk_xdp_drop: 0 rx_xsk_oversize_pkts_sw_drop: 0 rx_crc_errors_phy: 0 rx_in_range_len_errors_phy: 0 rx_discards_phy: 0 tx_discards_phy: 0 tx_errors_phy: 0 rx_prio0_discards: 0 rx_prio1_discards: 0 rx_prio2_discards: 0 rx_prio3_discards: 0 rx_prio4_discards: 0 rx_prio5_discards: 0 rx_prio6_discards: 0 rx_prio7_discards: 0 tx_pause_storm_error_events: 0 [14:18 xcp-ng-disjqdnc ~]# ethtool -k eth1 | grep -i vlan rx-vlan-offload: on tx-vlan-offload: on rx-vlan-filter: on vlan-challenged: off [fixed] tx-vlan-stag-hw-insert: on rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: on [fixed] Conclusion: Network layout config problems, buffer overflows, or silent packet drops are 100% ruled out here (all hardware discards are flat zero). This wasn't a user mistake or an accidental configuration mismatch. The new sm or blktap components hit a deep kernel lockup handling the multi-stream VHD storage exports. The file locks deadlocked the kernel handles so badly that the hosts ignored standard and low-level CLI commands entirely, forcing a hard out-of-band physical power cycle of the XCP-ng hosts and the TrueNAS storage server just to clear the uninterruptible I/O sleep queue. I haven't touched a thing since getting the hardware back up and verifying what choked. The system is sitting exactly as it is right now. Let me know what specific debugging paths you want me to look at or what logs you need from that window to trace this down.

  • Everything related to the virtualization platform

    1k Topics
    16k Posts
    dicode-nlD
    @nick.lloyd if there is anything that isn't working just let me know. I do advise the SMAPIv3 solution at that implementation offers the full feature set!

  • All Xen Orchestra related questions/reports

    3k Topics
    29k Posts
    B
    @acebmxer Of course, sorry. We have XO from sources, currently commit 0be90. I'm not aware of any beta updates? I'm attaching the latest log for one of the jobs. 2026-10-27T06_00_00.022Z - backup NG.json.txt

  • Our hyperconverged storage solution

    55 Topics
    827 Posts
    C
    @poddingue Hi, we have a ticket at Vates regarding this. (Ticket#7765809)

  • Non-English speakers

    37 Topics
    137 Posts
    S
    @jqueiroz said: Bom dia a todos, Estou com um problema em uma VM, ela está ligando/desligando sozinha, repetidamente. Não encontro nenhuma mensagem de erro explicando o motivo. Alguém pode me dar alguma dica sobre como solucionar isso? Uso o XCP-NG 8.0 em um cluster de dois servidores HP Proliant DL-360, com o HA-Lizard. Tudo vinha funcionando bem até alguns dias atrás, quando houve uma queda de energia no datacenter da localidade. Tenho o mesmo arranjo de VMs/servidores em mais seis localidades, nenhuma apresenta esse problema. Analisar esse comportamento instável do cluster é quase como prever resultados complexos, parecendo a análise de riscos que a malta faz ao dar palpites no https://mostbetbrasil.lat. Grato. Como o problema começou depois da queda de energia e só acontece nessa localidade, eu começaria verificando os logs do XCP-ng e do HA-Lizard para entender quem está disparando os reinícios. Pode ser alguma política de HA tentando recuperar a VM repetidamente, algum problema de armazenamento ou até inconsistência no estado do host após a interrupção. Também vale conferir o estado do pool e os logs do Xen Orchestra, se estiver usando. Se possível, desative temporariamente a automação de recuperação dessa VM, com cuidado para não comprometer a disponibilidade, e veja se ela continua reiniciando sozinha. Isso ajuda a isolar a causa.