XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XCP-ng 8.3 updates announcements and testing

    Scheduled Pinned Locked Moved News
    664 Posts 55 Posters 566.0k Views 74 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Online
      MajorP93
      last edited by MajorP93

      Hello, so I got access to the test environment back and was able to install this set of patches.

      I actually installed them on the XCP-ng host before you guys released them to the stable repository.

      I ran:

      yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
      yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates

      and rebooted the system.

      Unfortunately I have to say that this is the first time that patches broke my system.

      While the XCP-ng host is still able to boot, it is not longer able to mount my SRs.

      I have 2 SR in this test environment:

      • 1x Linstor vSAN iSCSI configured as QCOW2
      • 1x TrueNAS Core NFSv3 configured as VHD

      I spent some hours troubleshooting this and appearently it is caused by jumbo frames no longer working after applying these XCP-ng patches.

      The hypervisor / storage network in this testing environment is using jumbo frames everywhere (all switches involved, all storage systems).
      Prior to installing the updates everything was working fine.

      Now I can not ping the storage systems anymore using jumbo frames.
      (ping -M do -s 8972 ...)

      Hence the tasks that are meant to mount the SRs are stuck forever:

      [14:38 xcpng-test01 ~]# xe task-list
      uuid ( RO)                : a6ba1324-bfe1-8aca-c5d1-d7cc57d37cca
                name-label ( RO): PBD.plug
          name-description ( RO): 
                    status ( RO): pending
                  progress ( RO): 0.000
      
      
      uuid ( RO)                : f9c76606-e516-553f-706c-ff52bc303e2d
                name-label ( RO): PBD.plug
          name-description ( RO): 
                    status ( RO): pending
                  progress ( RO): 0.000
      
      

      The interesting thing is that "ip a" is still showing MTU 9000:

      [14:44 xcpng-test01 ~]# ip a
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
             valid_lft forever preferred_lft forever
      2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
          link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
      3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
          link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
      4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
          link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
      5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
          link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
      6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
          link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
      7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
          link/ether 4a:1c:22:40:4a:d2 brd ff:ff:ff:ff:ff:ff
      8: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
      9: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
      10: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
      11: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
      12: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
      13: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
          link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
          inet 10.10.160.24/24 brd 10.10.160.255 scope global xapi1
             valid_lft forever preferred_lft forever
      
      

      Can't start XO VM right now as it lives on one of the SR but XO Lite is also still showing jumbo frames being enabled:
      749d3302-73f7-4534-8d57-d8bc4cec47b4-image.jpeg

      Was something changed in this set of patches that could cause this issue?

      Thanks and best regards

      A gduperreyG 2 Replies Last reply Reply Quote 0
      • B Offline
        bufanda @gduperrey
        last edited by

        @gduperrey said:

        Thank you everyone for your tests and your feedback!

        The updates are live now: https://xcp-ng.org/blog/2026/08/18/august-2026-updates-1-for-xcp-ng-8-3-lts/

        Installed on my prod pool and instantly got an SMART error. What a timing πŸ˜„ But other than that updates working withut issues.

        1 Reply Last reply Reply Quote 1
        • A Offline
          anthoineb Vates πŸͺ XCP-ng Team @MajorP93
          last edited by anthoineb

          @MajorP93 Try first to fix the ping issue with jumboframe before anything. If the issue persists then:
          what are the types of your SRs unable to be mounted? Is there any error in SMlog or xensouce.log?

          1 Reply Last reply Reply Quote 0
          • gduperreyG Offline
            gduperrey Vates πŸͺ XCP-ng Team @MajorP93
            last edited by

            Hello @MajorP93,

            Several things in your message raise questions for me.

            First, this SR:

            • 1x Linstor vSAN iSCSI configured as QCOW2

            What is this storage and how is it configured?
            Currently, we do not support QCOW2 on Linstor/Xostor. So, is it simply an iSCSI device to which you have applied the QCOW2 format, or is it a Xostor on the XCP-ng pool?

            Regarding jumbo frames, we do not use them internally and are not currently testing them. It is often recommended not to use them for the management interface, as indicated in the XCP-ng documentation: https://docs.xcp-ng.org/networking/

            Non-standard MTUs (such as jumbo frames) are not supported on management interfaces. Using them can lead to serious issues, including failed pool member joins or unexpected network outages.
            

            Is that the case here?

            1 Reply Last reply Reply Quote 0
            • bleaderB Offline
              bleader Vates πŸͺ XCP-ng Team
              last edited by

              @majorp93 no, these updates should not change anything on the MTU behavior. From your screenshots and ip a output, I assume eth4 is the only link used, than your vlan1060 network is indeed your management network. I guess with some confidence that:

              • eth4 is your actual NIC
              • xenbr4 is the pool-wide network without vlan using eth4
              • xapi1 is the bridge for vlan1060
                If that's right indeed on your host side, the MTUs are properly set.

              As GaΓ«l said, jumbo on management network is not officially supported, because we always end up in situation similar to yours, where something stops working for "some" reason πŸ™‚

              But as you said, when everything is setup properly, it does work.

              I have test hosts at home up to date but no similar setup to yours for SR. I tried with a pool-wide and a pool-wide + vlan as management with 9000 and the ping -M do -s 8972 does work fine in both cases, so nothing I can see here.

              When it fails, what do you see? Message too long? mtu=1500 ? something else?

              Do you have multiple hosts in that pool? Can you try that between the hosts as well and not toward the storage systems? I would also check on ovs with ovs-vsctl list interface that each of eth4, xennbr4 and xapi1 report 9000 mtu.

              I know you said it was properly setup everywhere, but I would still be tempted to think there is a setup issue somewhere.

              1 Reply Last reply Reply Quote 1
              • M Online
                MajorP93
                last edited by

                @anthoineb @gduperrey @bleader
                Hi!

                Thank you very much for your responses.

                I actually found the issue and it was not caused by the XCP-ng patches.
                Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
                It seems like this happened in the time frame between XCP-ng updates .
                Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

                Anyways sorry for taking your time in this regard-

                Best regards

                1 Reply Last reply Reply Quote 2
                • marcoiM Offline
                  marcoi
                  last edited by

                  for the latest updates. the test pool didnt move the vms when i went to reboot. so i moved them manually. rebooted master, move vms back patched and reboot 2nd test server in the pool.
                  Skipped trying rolling reboot in prod and followed the same process. both pools went well without other issues.

                  1 Reply Last reply Reply Quote 1
                  • glehG Offline
                    gleh Vates πŸͺ XCP-ng Team
                    last edited by

                    New maintenance update candidates for XCP-ng 8.3 LTS

                    This batch of updates focuses on storage bug fixes and performance improvements.

                    What changed

                    Storage

                    • blktap:
                      When CBT is enabled, an issue occurred where tapdisk opened cbtlog instead of the primary QCOW2 image. This previously caused commit operations (ENOTSUPP) and leaf-coalesce routines to fail, resulting in excessive storage consumption on the SR. Tapdisk now correctly bypasses the cbtlog image to allow proper coalescing.

                    • sm: Fix a bug where the coalesce on a secondary host of a QCOW2 VDI on LVMSR could corrupt LVM metadata and would need manual intervention to restore them.

                    • xapi:
                      - Migration speeds are now improved for sparse QCOW2-backed VDIs, as empty sectors are no longer transferred.
                      - Preserve VDI tags on migration. This is a foundational fix for kubernetes integration.

                    Versions

                    • blktap: 3.55.5-9.3.xcpng8.3 -> 3.55.5-9.4.xcpng8.3
                    • sm: 3.2.12-23.4.xcpng8.3 -> 3.2.12-23.5.xcpng8.3
                    • xapi: 26.1.16-1.1.xcpng8.3 -> 26.1.16-1.2.xcpng8.3

                    Test on XCP-ng 8.3

                    yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
                    yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
                    reboot
                    

                    The usual update rules apply: pool coordinator first, etc.

                    What to test

                    As usual, normal use and anything else you want to test.

                    Test window before official release of the updates

                    ~24 hrs

                    We would like to thank users who shared feedback since our last call for testing: @Andrew, @JeffBerntsen, @MajorP93, @XCP-ng-JustGreat, @acebmxer, @bufanda, @flakpyro

                    B acebmxerA M 3 Replies Last reply Reply Quote 4
                    • B Offline
                      bufanda @gleh
                      last edited by

                      @gleh installed on my lab pool. Haven’t seen any issues so far.

                      1 Reply Last reply Reply Quote 2
                      • acebmxerA Online
                        acebmxer @gleh
                        last edited by

                        @gleh

                        Installed...

                        Updated:
                          blktap.x86_64 0:3.55.5-9.4.xcpng8.3                forkexecd.x86_64 0:26.1.16-1.2.xcpng8.3                
                          message-switch.x86_64 0:26.1.16-1.2.xcpng8.3       qcow-stream-tool.x86_64 0:26.1.16-1.2.xcpng8.3         
                          rrdd-plugins.x86_64 0:26.1.16-1.2.xcpng8.3         sm.x86_64 0:3.2.12-23.5.xcpng8.3                       
                          sm-cli.x86_64 0:26.1.16-1.2.xcpng8.3               sm-fairlock.x86_64 0:3.2.12-23.5.xcpng8.3              
                          squeezed.x86_64 0:26.1.16-1.2.xcpng8.3             varstored-guard.x86_64 0:26.1.16-1.2.xcpng8.3          
                          vhd-tool.x86_64 0:26.1.16-1.2.xcpng8.3             wsproxy.x86_64 0:26.1.16-1.2.xcpng8.3                  
                          xapi-core.x86_64 0:26.1.16-1.2.xcpng8.3            xapi-nbd.x86_64 0:26.1.16-1.2.xcpng8.3                 
                          xapi-rrd2csv.x86_64 0:26.1.16-1.2.xcpng8.3         xapi-storage-script.x86_64 0:26.1.16-1.2.xcpng8.3      
                          xapi-tests.x86_64 0:26.1.16-1.2.xcpng8.3           xapi-xe.x86_64 0:26.1.16-1.2.xcpng8.3                  
                          xcp-networkd.x86_64 0:26.1.16-1.2.xcpng8.3         xcp-rrdd.x86_64 0:26.1.16-1.2.xcpng8.3                 
                          xenopsd.x86_64 0:26.1.16-1.2.xcpng8.3              xenopsd-cli.x86_64 0:26.1.16-1.2.xcpng8.3              
                          xenopsd-xc.x86_64 0:26.1.16-1.2.xcpng8.3
                        
                        1 Reply Last reply Reply Quote 0
                        • M Online
                          MajorP93 @gleh
                          last edited by

                          @gleh Installed those RC packages in my test environment.
                          Tested basic operation. Looks good so far.
                          (Booted, all 3 SR connected, VMs able to start, both VHD and QCOW2)
                          Best regards

                          1 Reply Last reply Reply Quote 0

                          Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                          Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                          With your input, this post could be even better πŸ’—

                          Register Login
                          • First post
                            Last post