@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

Hello, so I got access to the test environment back and was able to install this set of patches.

I actually installed them on the XCP-ng host before you guys released them to the stable repository.

I ran:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates

and rebooted the system.

Unfortunately I have to say that this is the first time that patches broke my system.

While the XCP-ng host is still able to boot, it is not longer able to mount my SRs.

I have 2 SR in this test environment:

• 1x Linstor vSAN iSCSI configured as QCOW2
• 1x TrueNAS Core NFSv3 configured as VHD

I spent some hours troubleshooting this and appearently it is caused by jumbo frames no longer working after applying these XCP-ng patches.

The hypervisor / storage network in this testing environment is using jumbo frames everywhere (all switches involved, all storage systems).
Prior to installing the updates everything was working fine.

Now I can not ping the storage systems anymore using jumbo frames.
(ping -M do -s 8972 ...)

Hence the tasks that are meant to mount the SRs are stuck forever:

[14:38 xcpng-test01 ~]# xe task-list
uuid ( RO)                : a6ba1324-bfe1-8aca-c5d1-d7cc57d37cca
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : f9c76606-e516-553f-706c-ff52bc303e2d
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

The interesting thing is that "ip a" is still showing MTU 9000:

[14:44 xcpng-test01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:1c:22:40:4a:d2 brd ff:ff:ff:ff:ff:ff
8: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
9: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
10: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
11: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
12: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
13: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
    inet 10.10.160.24/24 brd 10.10.160.255 scope global xapi1
       valid_lft forever preferred_lft forever

Can't start XO VM right now as it lives on one of the SR but XO Lite is also still showing jumbo frames being enabled:

Was something changed in this set of patches that could cause this issue?

Thanks and best regards

Hmmm I have been reading quite often in the last months that people have issues with RPU.

I tried RPU during every single update within the last year and for me it only failed when one (or multiple) VMs have not been live migrateable.

That is not really the "fault" of the RPU process but rather related to the VMs.

Finding out the cause has been a bit finicky though. (reading through log file)

Maybe the biggest improvement to RPU would be some form of better error reporting?

That would make it more clear to people what actually caused the issues rather than thinking "RPU is broken".

@gleh Wow! This sounds like a big release especially in the QCOW2 / storage department.
A huge thanks to the whole XCP-ng team!
The platform keeps improving and improving which is awesome to see.

I will test this batch of packages next week when my test environment is available again.

Best regards

@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

@msupport @acebmxer
Hello guys.
Thanks for sharing your experience with veeam and reporting these issues.
My team is also planning on evaluating veeam and I was wondering: did you get a response from veeam?
Did they give you a time line on when they plan to release a fix?

@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.

@pierrebrunet Oh one more thing that I just remembered.
If I recall correctly the first time I saw this issue was right after I patched my XCP-ng hosts without updating Xen Orchestra.
So maybe this is caused by XAPI code changes?

@pierrebrunet Hello, thank you for working on this issue.

Are you completely sure that this isn't something that got introduced within Xen Orchestra code?

Since I started using XCP-ng / Xen Orchestra I have been using these backup repositories via SMB.
All backup jobs (including this one) have been working fine for almost a year.

This repeated body timeout error is a rather new issue which is why I was wondering if this could have been introduced by XO code changes?

I did not switch to a different linux kernel major version or similar. So the SMB client kernel module is at the same version number.

Best regards

@olivierlambert said:

Hi! Can you try to replicate on the latest commit?

Hello!
Yes I was able to replicate this issue after updating my Xen Orchestra.

The thing is that the "XO-config-and-pool-metadata" does not always fail.
It fails from time to time then it looks exactly like shown in the screenshot.
Attached logs please find above in my earlier post.

This exact job has been working for a long time before.
After some Xen Orchestra commit this issue got introduced.

I use the same backup remotes for multiple backup jobs.
Only the "XO-config-and-pool-metadata" job is failing with this body timeout error.

Best regards

@acebmxer Hello, thanks for sharing this here!

Me and my team have been waiting for this release!

I signed up for the Veeam newsletter hoping I would get notified once Veeam 13.1 (an XCP-ng plugin) gets released but they seem to be a bit slow.

So I found out about this release due to your post.

Will start testing!

Hello,

the issue is back on my end.
Pool and metadata backup failed on all remotes due to "body timeout error".

Attached please find full backup log:
https://paste.vates.tech/?f781d8cb757e56b1#8MN4cyjHhkWKzhhvvggckf8mqjT9gxDY6t4vkSVQcWUY

I can confirm what @christopher-petzel said. Before commit 63f8d got pushed this issue did not occur.

I am currently running commit 1d82c (feat: release 6.6.2).

Best regards

Installed these patches on a pool of 4 hosts, can confirm everything looking good!

@rzr Thanks! How long does it take for these patches go be available?

[17:00 xcpng01 ~]# yum update
Geladene Plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

//EDIT: Nevermind. After running yum clean metadata they are being picked up.

@Jonathon I think @mathieu-l asked it you rebooted after installing drbd 9.2.18 from test repository. Your message sounds like you rebooted after installing 9.2.16 but not after applying the 9.2.18 update from test repository.

@jonathon are you using your LINSTOR/DRBD stack in the form of licensed XOSTOR? Your issue looks a case for pro support. Did you consider opening a case with pro support?

@rzr Since the targeted test window is over by now, I wonder: what are your plans regarding this set of patches? Will it be released soon?

@Tackyone said:

@MajorP93 said:

@Tackyone said:
live migrates between non-pooled [...]

Hi,

AFAIK this is not possible. Live migration does only work within the same pool. So you can not live migrate VMs to another pool / host that is not part of the pool.

But we do that all the time? - i.e. Two non-pool hosts, live migrate between them (these happen to have the same CPU / family etc.)

Okay I assumed that you use shared storage.

From the XenServer 8.4 "Migrate VMs" docs:
"Storage can only be shared between hosts in the same pool. As a result, VMs can only be migrated to hosts in the same pool."

So for classic live migrations that do not transfer the virtual disk you have to be on the same pool.

What you are describing must be a storage live migration then (XenMotion). That should work across pools, yes. I stand corrected because I did not factor in this variant in my initial answer.

//EDIT: also there is no such thing as non-pool hosts. Even a single host setup has it's own pool.

@Tackyone said:
live migrates between non-pooled [...]

Hi,

AFAIK this is not possible. Live migration does only work within the same pool. So you can not live migrate VMs to another pool / host that is not part of the pool.

Regarding mixing and matching different CPUs within the same pool: do note that you can not mix AMD/Intel together. Different CPU models of the same brand within a pool will work though. And it behaves just like you said: XCP-ng hosts will agree on the least common denominator regarding CPU features and use that across the pool in order to ensure that (live) migrations will work.