@olivierlambert Classic blunder It's working since rebooting, thanks

@archw Yes, if you installed any 8.3 beta or RC and have recently performed a yum update and reboot, then you are up to date with the packages that are included in the 8.3-rc2 release.

@BenjiReis I've finally taken the time to review this again now that I've updated to 8.3-rc1 via yum update, so here's some follow-up on the points I brought up previously:

There is no way to configure IPv6 on the management interface via xsconsole, such as if one wants to switch between static configuration, autoconf via RAs, or DHCPv6.

True but we'll soon release an new version of xsconsole adapted for IPV6 allowing to configure IPv6 for management interface

There is apparently no support for IPv6 DNS servers, only IPv4. For example, if I try to add an IPv6 address like fd00::1 or [fd00::1] as a DNS server via xsconsole, there is apparently no change to the configuration. Editing /etc/resolv.conf works to achieve this (e.g. adding the line nameserver fd00::1), but this is known not to persist across reboots.

Should be solved by the future xsconsole release as well

Still not seeing any enhancements/changes in behaviour as of xsconsole 11.0.6-1.1.xcpng8.3.

There is apparently no support for RDNSS (advertisement of DNS servers in RAs rather than via DHCPv6).

DHCPv6 is one of the major blindspot for now indeed, I'm working on it but I don't have much knowledge on this so any hints are welcome if you spot if something is missing somewhere.

Just to clarify, this isn't related to DHCPv6, but RAs (Router Advertisement packets). I personally don't have a DHCPv6 server on my network at all. RDNSS is described in RFC8106.

Others may want to advertise DNS servers using DHCPv6, though, so that should still be tested as well.

The "autoconf" option (available during installation, after choosing IPv6-only or dual-stack, and then being asked which mode to use to configure IPv6 addresses) appears to only be used at installation time to determine values such as the gateway's link-local address, the available address prefixes, and perform SLAAC and DAD, but then the resulting values are hard-coded and don't change according to changes in the environment, such as an upstream change in network prefix. (I will need to do some more testing to really confirm this, but this seems to be the case in my experience.) Compare this to when IPv4 is configured to use DHCP(v4), in which the management interface may have a different IPv4 address at different times, namely if it's assigned a different address by the DHCP server when it attempts to get or renew a lease.

I'm not aware of this issue, i'll try to reproduce in our env.

I haven't been able to reproduce this either, and my prefix has changed a couple of times since I said this was an issue. Perhaps I just imagined it, hit a weird edge case, or didn't wait for the valid lifetime of the old prefix to expire; my router doesn't reliably advertise the fact that an old prefix is no longer valid.

Some repos are unreachable in IPv6-only environments, which I'm aware is already known, and I can get around this by using NAT64 (either with CLAT to perform 464XLAT; or with DNS64), but this fact is currently a blocker for me to move to being IPv6-only.

We contacted the mirrors many times, still trying to have'em all advertising IPv4 and 6 and also trying to find a solution that could "smartly" redirect towards a compatible mirror.

@stormi said in IPv6 support in XCP-ng for the management interface - feedback wanted:

FYI, I have finally reviewed all mirrors that provide updates for XCP-ng and disabled the remaining 6 which didn't support IPv6 (and notified their owners. I'll enable them again if they enable IPv6).

So, if you experience any issues installing updates via IPv6, tell us so that we investigate faulty mirrors.

I personally haven't had any issues reaching repos since then, but I haven't explicitly tested this or looked through the mirrorlist. I also don't think this is much of an issue in practice, since 464XLAT can be used; this is no longer a blocker from me, as I've reviewed the way I'm deploying IPv6-only. It's very nice to see you motivate / put pressure on mirror maintainers to make their sites accessible over IPv6 though, especially indirectly by simply removing such sites from the mirrorlist.

Speaking of NAT64, this is just a question, I haven't tested or looked into this myself: Does XCP-ng include a CLAT daemon and support for auto-configuring 464XLAT using either the "PREF64" RA option (RFC8781) or resolution of ipv4only.arpa via a DNS64 server (RFC7050)?

Haven't tested either for now, feel free to do and report if you get here before me.

I've got this working pretty easily by manually installing clatd from GitHub and its dependencies from EPEL and the other RHEL repos. It works, but isn't native. That being said, I don't know of any other Linux distros that natively support this yet. To my knowledge, there is ongoing work to implement this directly in Systemd. Clatd supports RFC7050, but doesn't support PREF64/RFC8781 as it's not particualrly feasible for it to do so, but hopefully Systemd is able to if/when it implements a CLAT.

This also isn't reliable across reboots / DHCP lease renewals because I have no simple way to disable IPv4 on the management interface. I haven't tried this with an installation where I've selected "IPv6-only" in the installer.

One practical issue I've experienced when using 464XLAT in this way is that XO Lite tries to contact the pool server in the frontend / client / web browser using JS fetch calls for URLs falling under https://localhost/, which would instead usually be under https://<pool server IPv4 address>/. These are the addresses that XO Lite will prompt the user to ensure that the browser trusts TLS certificates for if they are self-signed and no known CA has issued/signed them. As such, these don't work, since "localhost" from the XO Lite user's perspective isn't the same machine as the "localhost" that XO Lite is running on. If XO Lite supported making these calls using any of the pool servers' routable IPv6 addresses (e.g. ULAs or GUAs, but not LLAs), this would work just fine.

I may find some time to test these things on an "IPv6-only" installation, but I expect that will be after 8.3 has reached general release.

@gb-123

I believe there is no 'upgrade' path from BIOS to UEFI? ( I would need to completely re-install the host again... right ?)

That is correct, and the installation documentation mentions this:

WARNING
NEVER switch from UEFI to BIOS (or vice-versa) after you installed XCP-ng. Stick to the mode that you chose during the install.

---

Does XCP-ng have the option of secure boot when installing from iso?

Not yet, though that feature seems to be tracked on GitHub here and mentions a talk discussing the scope of the problem (YouTube video). Unfortunately, it looks like there hasn't been any progress on this in the last 2 or 3 years.

Feedback/comments relating to IPv6 after some usage of beta1 and beta2:

• There is no way to configure IPv6 on the management interface via xsconsole, such as if one wants to switch between static configuration, autoconf via RAs, or DHCPv6.

• There is apparently no support for IPv6 DNS servers, only IPv4. For example, if I try to add an IPv6 address like fd00::1 or [fd00::1] as a DNS server via xsconsole, there is apparently no change to the configuration. Editing /etc/resolv.conf works to achieve this (e.g. adding the line nameserver fd00::1), but this is known not to persist across reboots.

• There is apparently no support for RDNSS (advertisement of DNS servers in RAs rather than via DHCPv6).

• The "autoconf" option (available during installation, after choosing IPv6-only or dual-stack, and then being asked which mode to use to configure IPv6 addresses) appears to only be used at installation time to determine values such as the gateway's link-local address, the available address prefixes, and perform SLAAC and DAD, but then the resulting values are hard-coded and don't change according to changes in the environment, such as an upstream change in network prefix. (I will need to do some more testing to really confirm this, but this seems to be the case in my experience.) Compare this to when IPv4 is configured to use DHCP(v4), in which the management interface may have a different IPv4 address at different times, namely if it's assigned a different address by the DHCP server when it attempts to get or renew a lease.

• Some repos are unreachable in IPv6-only environments, which I'm aware is already known, and I can get around this by using NAT64 (either with CLAT to perform 464XLAT; or with DNS64), but this fact is currently a blocker for me to move to being IPv6-only.

• Speaking of NAT64, this is just a question, I haven't tested or looked into this myself: Does XCP-ng include a CLAT daemon and support for auto-configuring 464XLAT using either the "PREF64" RA option (RFC8781) or resolution of ipv4only.arpa via a DNS64 server (RFC7050)?

@stormi @BenjiReis I thought I'd document my upgrade process here, as I did a bunch of testing this week on a spare laptop before finally doing it for real last night, and it all went very smoothly in the end. Perhaps all of this can be done by the installer as a user-friendly means of upgrading to add IPv6 support without needing any changes in XAPI:

1. Make note of the current partition table, because it will be wiped and the SR partition will not be recreated during the installation process. Mine was as follows:

# lsblk /dev/sda
NAME                                                                                              MAJ:MIN RM  SIZE
sda                                                                                                 8:0    0 21.8T  0 disk 
├─sda4                                                                                              8:4    0  512M  0 part /boot/efi
├─sda2                                                                                              8:2    0   18G  0 part 
├─sda5                                                                                              8:5    0    4G  0 part /var/log
├─sda3                                                                                              8:3    0 21.8T  0 part 
│ └─XSLocalEXT--d62dbe0a--b8b8--143f--6f29--3829124d35d4-d62dbe0a--b8b8--143f--6f29--3829124d35d4 253:0    0 21.8T  0 lvm  /run/sr-mount/d62dbe0a-b8b8-143f-6f29-3829124d35d4
├─sda1                                                                                              8:1    0   18G  0 part /
└─sda6                                                                                              8:6    0    1G  0

# gdisk -l /dev/sda
[...]
First usable sector is 34, last usable sector is 46875541470
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1        46139392        83888127   18.0 GiB    0700  
   2         8390656        46139391   18.0 GiB    0700  
   3        87033856     46875541470   21.8 TiB    8E00  
   4        83888128        84936703   512.0 MiB   EF00  
   5            2048         8390655   4.0 GiB     0700  
   6        84936704        87033855   1024.0 MiB  8200  

2. Ensure that you have an instance of XO (XenOrchestra) running on a different machine. Use that instance to create a backup of the pool metadata of the machine you'll be adding IPv6 support to.

3. Install XCP-ng 8.3 from scratch on the machine, overwriting the existing installation. Ensure that no disks are selected for use as an SR. This will wipe the partition table and create new partitions for the OS, but leave unpartitioned space where the SR partition would otherwise be. Since versions 8.2 and 8.3 use the same partition layout, you should get the same partition sizes, thereby leaving the SR filesystem intact on the disk, but inaccessible. Since you opted not to create an SR partition, the partition numbers will differ slightly. Immediately after installation, mine was as follows:

# lsblk /dev/sda
NAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda        8:0    0 21.8T  0 disk 
├─sda2     8:2    0   18G  0 part 
├─sda5     8:5    0    4G  0 part /var/log
├─sda3     8:3    0  512M  0 part /boot/efi
├─sda1     8:1    0   18G  0 part /
└─sda6     8:6    0    1G  0 part [SWAP]

# gdisk -l /dev/sda
[...]
First usable sector is 34, last usable sector is 46875541470
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1        46139392        83888127   18.0 GiB    0700  
   2         8390656        46139391   18.0 GiB    0700  
   3        83888128        84936703   512.0 MiB   EF00  
   5            2048         8390655   4.0 GiB     0700  
   6        84936704        87033855   1024.0 MiB  8200  

4. Reboot into the new installation, and then recreate the SR partition using gdisk:

   1. Run gdisk /dev/sda (or other device node name as appropriate).
   2. Create a new partition by entering n, then use the default values for the start and end sector (these should automatically match those of the SR partition as it appeared in the original partition table prior to reinstallation), and use 8e00 for the partition type.
   3. Remove the partition label by entering c, then the partition number (should be 4), then enter nothing for the name.
   4. Check the new partition table by entering p; the start and end sector values should match those of the original partition table, but the partition numbers may differ.
   5. Write the changes with w, or quit without writing changes with q.

5. Connect to the new installation using the remote XO instance, then create a new backup of this fresh installation's pool metadata.

6. Alter the first backup's file data (which is an XML file) as follows:

   1. In the section <table name="PBD">, replace the occurrence of the device node path for the SR with the correct path as it would be for the new installation. In particular, the disk's SCSI or other ID may have changed, and the SR partition's number in the partition table has probably changed from 3 to 4. In my case, I had to change it from /dev/disk/by-id/scsi-36...fa-part3 to /dev/disk/by-id/scsi-36...a9-part4.

   2. In the second backup's file data, find the section <table name="PIF">. Within it, find the <row> pertaining to the management interface. Copy the values of the following <row> attributes, overwriting the corresponding attributes in the first backup's file data with their values, so that the new installation's values for the IPv4- and IPv6-related configuration parameters are used:

      • DNS
      • IP
      • IPv6
      • gateway
      • ip_cofiguration_mode
      • ipv6_configuration_mode
      • ipv6_gateway
      • netmask
      • primary_address_type

7. Use XO to restore the now-altered first backup to the new installation. It will automatically reboot, and all storage backends, virtual disk metadata, VMs, and VM metadata should be restored and working, along with IPv6 on the management interface.

For those coming here from Google who are still unsure of some of the details, including what the static lower bound is actually for, here is the corresponding documentation for XenServer: https://docs.xenserver.com/en-us/citrix-hypervisor/vms/vm-memory.html

@archw Yes, if you installed any 8.3 beta or RC and have recently performed a yum update and reboot, then you are up to date with the packages that are included in the 8.3-rc2 release.

@BenjiReis I've finally taken the time to review this again now that I've updated to 8.3-rc1 via yum update, so here's some follow-up on the points I brought up previously:

There is no way to configure IPv6 on the management interface via xsconsole, such as if one wants to switch between static configuration, autoconf via RAs, or DHCPv6.

True but we'll soon release an new version of xsconsole adapted for IPV6 allowing to configure IPv6 for management interface

There is apparently no support for IPv6 DNS servers, only IPv4. For example, if I try to add an IPv6 address like fd00::1 or [fd00::1] as a DNS server via xsconsole, there is apparently no change to the configuration. Editing /etc/resolv.conf works to achieve this (e.g. adding the line nameserver fd00::1), but this is known not to persist across reboots.

Should be solved by the future xsconsole release as well

Still not seeing any enhancements/changes in behaviour as of xsconsole 11.0.6-1.1.xcpng8.3.

There is apparently no support for RDNSS (advertisement of DNS servers in RAs rather than via DHCPv6).

DHCPv6 is one of the major blindspot for now indeed, I'm working on it but I don't have much knowledge on this so any hints are welcome if you spot if something is missing somewhere.

Just to clarify, this isn't related to DHCPv6, but RAs (Router Advertisement packets). I personally don't have a DHCPv6 server on my network at all. RDNSS is described in RFC8106.

Others may want to advertise DNS servers using DHCPv6, though, so that should still be tested as well.

The "autoconf" option (available during installation, after choosing IPv6-only or dual-stack, and then being asked which mode to use to configure IPv6 addresses) appears to only be used at installation time to determine values such as the gateway's link-local address, the available address prefixes, and perform SLAAC and DAD, but then the resulting values are hard-coded and don't change according to changes in the environment, such as an upstream change in network prefix. (I will need to do some more testing to really confirm this, but this seems to be the case in my experience.) Compare this to when IPv4 is configured to use DHCP(v4), in which the management interface may have a different IPv4 address at different times, namely if it's assigned a different address by the DHCP server when it attempts to get or renew a lease.

I'm not aware of this issue, i'll try to reproduce in our env.

I haven't been able to reproduce this either, and my prefix has changed a couple of times since I said this was an issue. Perhaps I just imagined it, hit a weird edge case, or didn't wait for the valid lifetime of the old prefix to expire; my router doesn't reliably advertise the fact that an old prefix is no longer valid.

Some repos are unreachable in IPv6-only environments, which I'm aware is already known, and I can get around this by using NAT64 (either with CLAT to perform 464XLAT; or with DNS64), but this fact is currently a blocker for me to move to being IPv6-only.

We contacted the mirrors many times, still trying to have'em all advertising IPv4 and 6 and also trying to find a solution that could "smartly" redirect towards a compatible mirror.

@stormi said in IPv6 support in XCP-ng for the management interface - feedback wanted:

FYI, I have finally reviewed all mirrors that provide updates for XCP-ng and disabled the remaining 6 which didn't support IPv6 (and notified their owners. I'll enable them again if they enable IPv6).

So, if you experience any issues installing updates via IPv6, tell us so that we investigate faulty mirrors.

I personally haven't had any issues reaching repos since then, but I haven't explicitly tested this or looked through the mirrorlist. I also don't think this is much of an issue in practice, since 464XLAT can be used; this is no longer a blocker from me, as I've reviewed the way I'm deploying IPv6-only. It's very nice to see you motivate / put pressure on mirror maintainers to make their sites accessible over IPv6 though, especially indirectly by simply removing such sites from the mirrorlist.

Speaking of NAT64, this is just a question, I haven't tested or looked into this myself: Does XCP-ng include a CLAT daemon and support for auto-configuring 464XLAT using either the "PREF64" RA option (RFC8781) or resolution of ipv4only.arpa via a DNS64 server (RFC7050)?

Haven't tested either for now, feel free to do and report if you get here before me.

I've got this working pretty easily by manually installing clatd from GitHub and its dependencies from EPEL and the other RHEL repos. It works, but isn't native. That being said, I don't know of any other Linux distros that natively support this yet. To my knowledge, there is ongoing work to implement this directly in Systemd. Clatd supports RFC7050, but doesn't support PREF64/RFC8781 as it's not particualrly feasible for it to do so, but hopefully Systemd is able to if/when it implements a CLAT.

This also isn't reliable across reboots / DHCP lease renewals because I have no simple way to disable IPv4 on the management interface. I haven't tried this with an installation where I've selected "IPv6-only" in the installer.

One practical issue I've experienced when using 464XLAT in this way is that XO Lite tries to contact the pool server in the frontend / client / web browser using JS fetch calls for URLs falling under https://localhost/, which would instead usually be under https://<pool server IPv4 address>/. These are the addresses that XO Lite will prompt the user to ensure that the browser trusts TLS certificates for if they are self-signed and no known CA has issued/signed them. As such, these don't work, since "localhost" from the XO Lite user's perspective isn't the same machine as the "localhost" that XO Lite is running on. If XO Lite supported making these calls using any of the pool servers' routable IPv6 addresses (e.g. ULAs or GUAs, but not LLAs), this would work just fine.

I may find some time to test these things on an "IPv6-only" installation, but I expect that will be after 8.3 has reached general release.

@gb-123

I believe there is no 'upgrade' path from BIOS to UEFI? ( I would need to completely re-install the host again... right ?)

That is correct, and the installation documentation mentions this:

WARNING
NEVER switch from UEFI to BIOS (or vice-versa) after you installed XCP-ng. Stick to the mode that you chose during the install.

---

Does XCP-ng have the option of secure boot when installing from iso?

Not yet, though that feature seems to be tracked on GitHub here and mentions a talk discussing the scope of the problem (YouTube video). Unfortunately, it looks like there hasn't been any progress on this in the last 2 or 3 years.

@gb-123 Using BIOS for a hypervisor generally complicates things related to disk and partition management, since a BIOS machine boots from an MBR-formatted disk, and MBR does not directly support drives greater than 2TiB. For a hypervisor, the main benefits of UEFI are:

• support for booting from GPT-formatted drives (which thus can be larger than 2TiB);
• power management (which is generally better in a UEFI-native OS than one being booted in CSM (a.k.a Legacy BIOS) mode; and
• Secure Boot, which you can use to ensure that the OS being booted is cryptographically signed by an OS vendor that you trust.

See here for a more detailed comparison of BIOS and UEFI. The comments about Fast Boot aren't relevant here.

I personally have XCP-ng installed in UEFI mode and then run all of my VMs in BIOS mode, because I want those benefits of UEFI on the host system, but don't want all of my guests/VMs' boot disks to have the overhead of an EFI System Partition.

@olivierlambert Classic blunder It's working since rebooting, thanks

@olivierlambert No change in behaviour; just updated XO from 25982ca to dcdf288, and checked that dom0 is up to date also:

[16:12 harmonium ~]# yum clean expire-cache
Loaded plugins: fastestmirror
Cleaning repos: xcp-ng-base xcp-ng-updates
2 metadata files removed
[16:12 harmonium ~]# yum update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
xcp-ng-base/signature                        |  473 B  00:00:00     
xcp-ng-base/signature                        | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/signature                     |  473 B  00:00:00     
xcp-ng-updates/signature                     | 3.0 kB  00:00:00 !!! 
xcp-ng-base/primary_db                       | 3.1 MB  00:00:15     
No packages marked for update

As mentioned here: https://xcp-ng.org/forum/post/72590

I was able to get the new version of xo-lite via yum update (and it's looking good!), so I have everything up to date, though I'm still running Xen 4.13, not Xen 4.17. However, stats in XO (not XO Lite) still aren't working for me for some reason; I'm running XO v5.92.0 (commit 25982ca, from today [29 Feb 2024]), built from source. All VMs are running xe-guest-utilities / management agent 7.30.0-2, as that is the version provided by my installation of XCP-ng.

I have several different VM types (Debian 11, Debian 12, Debian 12 from cloud images initialised using cloud-init, Ubuntu 22.04), all of which stats were working for under XCP-ng 8.2, I believe with the same version of xe-guest-utilities, which I installed from the ISO available under the "XCP-ng Tools" iso-repo. All VMs show "Management agent 7.30.0-2 detected" and I can see other data, such as IP addresses assigned to VIFs, but the Stats subpage just shows "No stats."

@stormi I was able to get the new version of xo-lite via yum update (and it's looking good!), so I have everything up to date, though I'm still running Xen 4.13, not Xen 4.17. However, stats in XO (not XO Lite) still aren't working for me for some reason; I'm running XO v5.92.0 (commit 25982ca, from today), built from source. All VMs are running xe-guest-utilities / management agent 7.30.0-2, as that is the version provided by my installation of XCP-ng.

@TheFrisianClause No, my remark is about wanting to resolve any and all DNS queries using a DNS server that is accessible over IPv6 (such as wanting to use Google's 2001:4860:4860::8888 rather than Google's 8.8.8.8, or in my case wanting to use my LAN's DNS server at fd00::1 rather than 10.0.0.1), whereas yours is about your host attempting to connect to IPv6 addresses that it obtained by resolving AAAA queries, despite the host only having IPv4 connectivity.

The behaviour you're observing is definitely odd. Are you sure that no IPv6 addresses are configured? What is the output of ip a and ip -6 route? If you share the output here, you may want to redact any geographically identifying info in it.

Feedback/comments relating to IPv6 after some usage of beta1 and beta2:

• There is no way to configure IPv6 on the management interface via xsconsole, such as if one wants to switch between static configuration, autoconf via RAs, or DHCPv6.

• There is apparently no support for IPv6 DNS servers, only IPv4. For example, if I try to add an IPv6 address like fd00::1 or [fd00::1] as a DNS server via xsconsole, there is apparently no change to the configuration. Editing /etc/resolv.conf works to achieve this (e.g. adding the line nameserver fd00::1), but this is known not to persist across reboots.

• There is apparently no support for RDNSS (advertisement of DNS servers in RAs rather than via DHCPv6).

• The "autoconf" option (available during installation, after choosing IPv6-only or dual-stack, and then being asked which mode to use to configure IPv6 addresses) appears to only be used at installation time to determine values such as the gateway's link-local address, the available address prefixes, and perform SLAAC and DAD, but then the resulting values are hard-coded and don't change according to changes in the environment, such as an upstream change in network prefix. (I will need to do some more testing to really confirm this, but this seems to be the case in my experience.) Compare this to when IPv4 is configured to use DHCP(v4), in which the management interface may have a different IPv4 address at different times, namely if it's assigned a different address by the DHCP server when it attempts to get or renew a lease.

• Some repos are unreachable in IPv6-only environments, which I'm aware is already known, and I can get around this by using NAT64 (either with CLAT to perform 464XLAT; or with DNS64), but this fact is currently a blocker for me to move to being IPv6-only.

• Speaking of NAT64, this is just a question, I haven't tested or looked into this myself: Does XCP-ng include a CLAT daemon and support for auto-configuring 464XLAT using either the "PREF64" RA option (RFC8781) or resolution of ipv4only.arpa via a DNS64 server (RFC7050)?