XCP-ng 8.3 betas and RCs feedback 🚀

lyan

Hi, I saw we are still using 4.13.5-10.42.1 version of Xen in beta, however, I noticed that there was plan to use 4.17.x by https://en.wikipedia.org/wiki/XCP-ng. I am wondering what is the roadmap of xen in xcp-ng eventually? Do we have upgrade plan for Xen?

POleszkiewicz

Great work,

BTW, would it be possible to add nvme-cli to the installer image? It would be nice if we could actually attach NVMeOF at install time and install to NVMeOF volumes (While keeping /boot either on USB/SD locally or on iSCSI), this way we could easily provision a cluster of diskless hosts, while keeping system storage redundancy by using MD RAID between two NVMEoF volumes located on different target hosts.

(with some manual work to attach NVMeOF before mounting root)

olivierlambert

@lyan Yes, for a version 9.0, because a major number change means we can make huge bump in kernel and Xen versions

olivierlambert

@POleszkiewicz Interesting, might worth doing a PoC with this to see if it works correctly. Can you open an issue on the main XCP Github repo?

stormi

To pave the way, you can also modify the installation ISO with https://github.com/xcp-ng/xcp/tree/master/scripts/iso-remaster: add nvme-cli inside install.img, rebuild the ISO, and see how the installer behaves.

lyan

@olivierlambert nice, out of curiosity, what is the rough timeline for the 9.0 release,

laurentm

Hello, I have encountered a crash with kernel panic and loop reboot on a working 8.2 install. XCP-NG was on a RAID1 SSD partition and datas on a RAID 10 SSD partition. I found no error on the SSDs (which are Enterprise quality) and had no idea how to get a simple way to have the VMs up and running if I wiped and completely reinstall XCP-NG.

So, I did a quick and dirty repair remotely through the DELL R420 IDRAC : upgrading the broken 8.2 with the 8.3 beta ISO and it worked perfectly.
Nevertheless, IDRAC is showing 192 Gb ram and XCP-NG 8.3 see only 144 Gb .
I suspect a RAM issue unless there is something wrong in the beta release.

My concern is: how can I move the VMs to another DELL R820 server which should be running 8.2 (since running a beta on production is a very bad practice) ?
Should I have to export in XVA files the VMs or should 8.2 could accept motion move ? or backup and restore with XenOrchestra ?

Thanks for your advices.

Kind regards,

Laurent

stormi

Live migration from 8.3 to 8.2 is not possible. You can't migrate to a lower release. Warm migration using XenOrchestra is possible.

See this blog post which explains the principle: https://xen-orchestra.com/blog/warm-migration-with-xen-orchestra/

laurentm

@stormi Thanks for the advice. I was pretty sure of no downgrade ability but I did not think to all of the XOA capabilities.

By the way, is there a release scheduled for the final 8.3 iso ?

stormi

@laurentm The schedule follows XenServer's schedule, with an extra delay for adaptation work, and there isn't any strongly defined ETA for it yet. We hope it will be before the end of the year.

Andrew

@stormi You also can't export XVA 8.3 and import into 8.2.... OVA did work.

Thunder-Cloud

Have I understood this correct when I think that:
XCP-NG 8.2 is same "source" as Citrix Hypervisor 8.2?
XCP-NG 8.3 is same "source" as the XenServer 8.0?

stormi

@Thunder-Cloud this is not far from accurate, if by "same source" you mean based on. Not everything from Citrix Hypervisor / XenServer is taken identical in XCP-ng, so we don't share 100% of the source code. There are various proprietary components we removed or replaced, and we also have our specific additions.

Also, the name "XenServer 8.0" is not accurate : it's called simply XenServer 8.

john.c

@stormi It's currently getting stuck on finishing startup on XCP-ng 8.3 beta 1. I have completed the installation of the 8.3 beta via netinstall media. Configured for dual ipv4 and ipv6 network stack.

However it seems to be taking a really long time to bring up the management network or not bringing it up. Alternatively something else is preventing it from finishing the start up.

If attempt to confirm the ethernet port for the management network, it's currently timing out during the attempt.

management connection issue - xcp-ng 8.3 beta1.jpg

management connection issue - xcp-ng 8.3 beta1 - message displayed.jpg

dell poweredge r620 system inventory.png

Can anyone please help me deduce which part of the finishing start-up is getting stuck?

gb.123

@john-c

Can you try installing ipv4 only and alternatively ipv6 only to see if it works ?
(Just wanted to confirm something)

laurentm

@john-c Maybe you could add an intel NIC if you have a spare one.
I like DELL servers but not much the Broadcom they provide to save some bucks.

bc-23

Hi,

I have a issue starting a PV VM on a fresh installed XCP-ng 8.3 server.
The VM was created from a template I exported from our XCP-ng 8.2 cluster and imported into the new 8.3 server.
The template creates a empty PV VM containing the PV boot information to do a network installation.

The error message I get is:
xenopsd internal error: VM = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error

In the xensource.log I don't see much more information:

Sep 28 08:20:12 X xapi: [error||26203 |Async.VM.start R:5c82647ea60e|xenops] Re-raising as INTERNAL_ERROR [ xenopsd internal error: VM
 = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ]
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] Async.VM.start R:5c82647ea60e failed with exception Server_error(INTERNAL_ERROR, [ 
xenopsd internal error: VM = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ])
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] Raised Server_error(INTERNAL_ERROR, [ xenopsd internal error: VM = fb7977de-aa28-27
3b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ])
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] 1/39 xenopsd-xc Raised at file ocaml/xenopsd/xc/xenops_server_xen.ml, line 2201
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] 2/39 xenopsd-xc Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 2
4
...

I skipped the remaining 36 lines from the backtrace, as this only seems to be the ocaml stack trace I it doesn't some to contain any additional relevant information.

When I compare two newly created VMs based on the PV template in the 8.2 and 8.3 environment, the look equal.
The PV elements from vm-param-list on both VMs looks like:

xe vm-param-list uuid=<UUID> | grep PV
                             PV-kernel ( RW): 
                            PV-ramdisk ( RW): 
                               PV-args ( RW): preseed/url=<install specific information>
                        PV-legacy-args ( RW): 
                         PV-bootloader ( RW): eliloader
                    PV-bootloader-args ( RW): 
                    PV-drivers-version (MRO): <not in database>
    PV-drivers-up-to-date ( RO) [DEPRECATED]: <not in database>
                   PV-drivers-detected ( RO): <not in database>

I see a difference on the bios-strings parameter, which is empty in 8.2 but contains the following in 8.3:

bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d

Do you have a hint what could case this error, or where I could find additional information, as the error message does not contain a lot of information.

Thanks.

olivierlambert

PV guests should be avoided as possible, it's not officially supported anymore.

Is it a 64 or 32 bits PV guest?
Can you convert it to a HVM guest and see the result?

bc-23

@olivierlambert

It is a 64bit PV guest.
The reason still using PV guests is that our install environment used to add boot parameters for the installer which was quite a nice way to manage the installation.

Running as HVM guest works without issue.

But when PV guests are not supported anymore then it seems now I should start to migrate our environment to use HVM guests.

May I ask what would be a good way to insert install information into a HVM guest? At the moment I would think about using xenstore to add information like IP to use and other information and create a custom Debian installer reading the information from the xenstore.
I know also about cloudinit, but what I have read so far I don't think it would fit into our environment.

Thanks,

olivierlambert

Packer is probably what you need @AtaxyaNetwork worked on it for some VMs here, feel free to ask her if you have question.