@gduperrey @rzr Hello!

Sorry for the delayed response and thanks a lot for raising that topic internally.

Due to these security patches being very critical/important I disabled DMC for all VMs that showed this issue (I think it was 6 Windows VMs) and patched the pool manually.

After that everything worked perfectly fine.

I wanted to react fast due to these security vulnerabilities and did not take the time to look at logs in /var/log etc.

Interestingly for the second update you guys published one day after (the kernel TCP vulnerability patch) RPU worked without any issues.

For now everything is stable and I will leave DMC disabled for these Windows VMs for now.

I was just suprised because this exact configuration, that included DMC was working stable for quite a few months.

However I can live without DMC on these VMs and am happy that everything is running stable.

Thank you very much for reacting fast on these security vulnerabilities!
I think you guys have a great process when it comes to this.

Best regards

Aaaaand it happened for the next VM.
Apart from updating XO I did not change anything...
It looks like RPU is failing on every single VM that has DMC enabled...
To me it looks like a regression

//EDIT:
the weird thing is that it fails with the same error even with DMC disabled...
It also fails when manually trying to live migrate the VM.
This particular VM is a Windows Server VM, XCP-ng tools version: Management agent 9.2.385-0
What leaves me baffled is that all of this has been working just fine for several months...

vm.migrate
{
  "vm": "25cf2537-c8f9-a3ef-ca68-6621351544c1",
  "migrationNetwork": "1ee81d68-6614-f134-f4b4-6bb5399bf66a",
  "targetHost": "d984660e-c9a9-4981-aac8-de3df67833a9"
}
{
  "code": "BALLOONING_TIMEOUT_BEFORE_MIGRATION",
  "params": [
    "OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813"
  ],
  "task": {
    "uuid": "dd29dc7b-62a6-1820-79d7-f447b7a5dcf6",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20261103T16:17:54Z",
    "finished": "20261103T16:20:44Z",
    "status": "failure",
    "resident_on": "OpaqueRef:73f0c67e-d88c-6ed0-9070-ca96f899319f",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "BALLOONING_TIMEOUT_BEFORE_MIGRATION",
      "OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2802))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2859))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2122))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2121))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 239))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 244))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2257))((process xenopsd-xc)(filename result.ml)(line 23))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2250))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2853))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2589))((process xenopsd-xc)(filename list.ml)(line 121))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2582))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2973))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 439))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2986))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 439))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2885))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3432))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3442))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3463))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 192))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3483))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3654))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 355))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 361))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 386))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 527))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3492))((process xapi)(filename lib/backtrace.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 551))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2605))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "BALLOONING_TIMEOUT_BEFORE_MIGRATION(OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813)",
  "name": "XapiError",
  "stack": "XapiError: BALLOONING_TIMEOUT_BEFORE_MIGRATION(OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813)
    at Function.wrap (file:///etc/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///etc/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1374:24)
    at file:///etc/xen-orchestra/packages/xen-api/index.mjs:1408:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1398:12)
    at Xapi._watchEvents (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1605:14)"
}

Hello,

thanks for publishing these updates!

However did you guys change anything regarding rolling pool update?

I am using latest XO from sources.

It's the third time I am getting this:

I am using dynamic memory control.

During last few rounds of updates (last few months) rolling pool update was working just fine.
This is the third VM today for which RPU is complaining about ballooning / DMC.

For the first two VMs I disabled DMC and re-tried RPU.
It progressed to the next VM but then failed again with the error shown on the screenshot.

In the past RPU was working fine with DMC hence the question regarding RPU changes?

Thanks and best regards

Last time I tested CBT enabled backups the backup job would sometimes create a full backup for a given VM instead of an incremental one.
This bug got triggered by live migrations. When a VM got live migrated it got processed as full backup by backup jobs even after "normal" live migrations (no storage live migration).

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

@acebmxer Very interesting. Thanks for sharing! I will wait until Veeam issued a new (hotfix) release before evaluating Veeam as a replacement for Xen Orchestra Backup.

@gleh Installed those RC packages in my test environment.
Tested basic operation. Looks good so far.
(Booted, all 3 SR connected, VMs able to start, both VHD and QCOW2)
Best regards

@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

Hello, so I got access to the test environment back and was able to install this set of patches.

I actually installed them on the XCP-ng host before you guys released them to the stable repository.

I ran:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates

and rebooted the system.

Unfortunately I have to say that this is the first time that patches broke my system.

While the XCP-ng host is still able to boot, it is not longer able to mount my SRs.

I have 2 SR in this test environment:

• 1x Linstor vSAN iSCSI configured as QCOW2
• 1x TrueNAS Core NFSv3 configured as VHD

I spent some hours troubleshooting this and appearently it is caused by jumbo frames no longer working after applying these XCP-ng patches.

The hypervisor / storage network in this testing environment is using jumbo frames everywhere (all switches involved, all storage systems).
Prior to installing the updates everything was working fine.

Now I can not ping the storage systems anymore using jumbo frames.
(ping -M do -s 8972 ...)

Hence the tasks that are meant to mount the SRs are stuck forever:

[14:38 xcpng-test01 ~]# xe task-list
uuid ( RO)                : a6ba1324-bfe1-8aca-c5d1-d7cc57d37cca
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : f9c76606-e516-553f-706c-ff52bc303e2d
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

The interesting thing is that "ip a" is still showing MTU 9000:

[14:44 xcpng-test01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:1c:22:40:4a:d2 brd ff:ff:ff:ff:ff:ff
8: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
9: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
10: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
11: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
12: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
13: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
    inet 10.10.160.24/24 brd 10.10.160.255 scope global xapi1
       valid_lft forever preferred_lft forever

Can't start XO VM right now as it lives on one of the SR but XO Lite is also still showing jumbo frames being enabled:

Was something changed in this set of patches that could cause this issue?

Thanks and best regards

Hmmm I have been reading quite often in the last months that people have issues with RPU.

I tried RPU during every single update within the last year and for me it only failed when one (or multiple) VMs have not been live migrateable.

That is not really the "fault" of the RPU process but rather related to the VMs.

Finding out the cause has been a bit finicky though. (reading through log file)

Maybe the biggest improvement to RPU would be some form of better error reporting?

That would make it more clear to people what actually caused the issues rather than thinking "RPU is broken".

@gleh Wow! This sounds like a big release especially in the QCOW2 / storage department.
A huge thanks to the whole XCP-ng team!
The platform keeps improving and improving which is awesome to see.

I will test this batch of packages next week when my test environment is available again.

Best regards

@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

@msupport @acebmxer
Hello guys.
Thanks for sharing your experience with veeam and reporting these issues.
My team is also planning on evaluating veeam and I was wondering: did you get a response from veeam?
Did they give you a time line on when they plan to release a fix?

@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.

@pierrebrunet Oh one more thing that I just remembered.
If I recall correctly the first time I saw this issue was right after I patched my XCP-ng hosts without updating Xen Orchestra.
So maybe this is caused by XAPI code changes?

@pierrebrunet Hello, thank you for working on this issue.

Are you completely sure that this isn't something that got introduced within Xen Orchestra code?

Since I started using XCP-ng / Xen Orchestra I have been using these backup repositories via SMB.
All backup jobs (including this one) have been working fine for almost a year.

This repeated body timeout error is a rather new issue which is why I was wondering if this could have been introduced by XO code changes?

I did not switch to a different linux kernel major version or similar. So the SMB client kernel module is at the same version number.

Best regards

@olivierlambert said:

Hi! Can you try to replicate on the latest commit?

Hello!
Yes I was able to replicate this issue after updating my Xen Orchestra.

The thing is that the "XO-config-and-pool-metadata" does not always fail.
It fails from time to time then it looks exactly like shown in the screenshot.
Attached logs please find above in my earlier post.

This exact job has been working for a long time before.
After some Xen Orchestra commit this issue got introduced.

I use the same backup remotes for multiple backup jobs.
Only the "XO-config-and-pool-metadata" job is failing with this body timeout error.

Best regards

@acebmxer Hello, thanks for sharing this here!

Me and my team have been waiting for this release!

I signed up for the Veeam newsletter hoping I would get notified once Veeam 13.1 (an XCP-ng plugin) gets released but they seem to be a bit slow.

So I found out about this release due to your post.

Will start testing!