XCP-ng 8.3 betas and RCs feedback 🚀

[laurentm]

Hello, I have encountered a crash with kernel panic and loop reboot on a working 8.2 install. XCP-NG was on a RAID1 SSD partition and datas on a RAID 10 SSD partition. I found no error on the SSDs (which are Enterprise quality) and had no idea how to get a simple way to have the VMs up and running if I wiped and completely reinstall XCP-NG.

So, I did a quick and dirty repair remotely through the DELL R420 IDRAC : upgrading the broken 8.2 with the 8.3 beta ISO and it worked perfectly.
Nevertheless, IDRAC is showing 192 Gb ram and XCP-NG 8.3 see only 144 Gb .
I suspect a RAM issue unless there is something wrong in the beta release.

My concern is: how can I move the VMs to another DELL R820 server which should be running 8.2 (since running a beta on production is a very bad practice) ?
Should I have to export in XVA files the VMs or should 8.2 could accept motion move ? or backup and restore with XenOrchestra ?

Thanks for your advices.

Kind regards,

Laurent

[stormi]

Live migration from 8.3 to 8.2 is not possible. You can't migrate to a lower release. Warm migration using XenOrchestra is possible.

See this blog post which explains the principle: https://xen-orchestra.com/blog/warm-migration-with-xen-orchestra/

[laurentm]

@stormi Thanks for the advice. I was pretty sure of no downgrade ability but I did not think to all of the XOA capabilities.

By the way, is there a release scheduled for the final 8.3 iso ?

[stormi]

@laurentm The schedule follows XenServer's schedule, with an extra delay for adaptation work, and there isn't any strongly defined ETA for it yet. We hope it will be before the end of the year.

[Andrew]

@stormi You also can't export XVA 8.3 and import into 8.2.... OVA did work.

[Thunder-Cloud]

Have I understood this correct when I think that:
XCP-NG 8.2 is same "source" as Citrix Hypervisor 8.2?
XCP-NG 8.3 is same "source" as the XenServer 8.0?

[stormi]

@Thunder-Cloud this is not far from accurate, if by "same source" you mean based on. Not everything from Citrix Hypervisor / XenServer is taken identical in XCP-ng, so we don't share 100% of the source code. There are various proprietary components we removed or replaced, and we also have our specific additions.

Also, the name "XenServer 8.0" is not accurate : it's called simply XenServer 8.

[john.c]

@stormi It's currently getting stuck on finishing startup on XCP-ng 8.3 beta 1. I have completed the installation of the 8.3 beta via netinstall media. Configured for dual ipv4 and ipv6 network stack.

However it seems to be taking a really long time to bring up the management network or not bringing it up. Alternatively something else is preventing it from finishing the start up.

If attempt to confirm the ethernet port for the management network, it's currently timing out during the attempt.

Can anyone please help me deduce which part of the finishing start-up is getting stuck?

[gb.123]

@john-c

Can you try installing ipv4 only and alternatively ipv6 only to see if it works ?
(Just wanted to confirm something)

[laurentm]

@john-c Maybe you could add an intel NIC if you have a spare one.
I like DELL servers but not much the Broadcom they provide to save some bucks.

[bc-23]

Hi,

I have a issue starting a PV VM on a fresh installed XCP-ng 8.3 server.
The VM was created from a template I exported from our XCP-ng 8.2 cluster and imported into the new 8.3 server.
The template creates a empty PV VM containing the PV boot information to do a network installation.

The error message I get is:
xenopsd internal error: VM = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error

In the xensource.log I don't see much more information:

Sep 28 08:20:12 X xapi: [error||26203 |Async.VM.start R:5c82647ea60e|xenops] Re-raising as INTERNAL_ERROR [ xenopsd internal error: VM
 = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ]
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] Async.VM.start R:5c82647ea60e failed with exception Server_error(INTERNAL_ERROR, [ 
xenopsd internal error: VM = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ])
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] Raised Server_error(INTERNAL_ERROR, [ xenopsd internal error: VM = fb7977de-aa28-27
3b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ])
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] 1/39 xenopsd-xc Raised at file ocaml/xenopsd/xc/xenops_server_xen.ml, line 2201
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] 2/39 xenopsd-xc Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 2
4
...

I skipped the remaining 36 lines from the backtrace, as this only seems to be the ocaml stack trace I it doesn't some to contain any additional relevant information.

When I compare two newly created VMs based on the PV template in the 8.2 and 8.3 environment, the look equal.
The PV elements from vm-param-list on both VMs looks like:

xe vm-param-list uuid=<UUID> | grep PV
                             PV-kernel ( RW): 
                            PV-ramdisk ( RW): 
                               PV-args ( RW): preseed/url=<install specific information>
                        PV-legacy-args ( RW): 
                         PV-bootloader ( RW): eliloader
                    PV-bootloader-args ( RW): 
                    PV-drivers-version (MRO): <not in database>
    PV-drivers-up-to-date ( RO) [DEPRECATED]: <not in database>
                   PV-drivers-detected ( RO): <not in database>

I see a difference on the bios-strings parameter, which is empty in 8.2 but contains the following in 8.3:

bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d

Do you have a hint what could case this error, or where I could find additional information, as the error message does not contain a lot of information.

Thanks.

[olivierlambert]

PV guests should be avoided as possible, it's not officially supported anymore.

1. Is it a 64 or 32 bits PV guest?
2. Can you convert it to a HVM guest and see the result?

[bc-23]

@olivierlambert

It is a 64bit PV guest.
The reason still using PV guests is that our install environment used to add boot parameters for the installer which was quite a nice way to manage the installation.

Running as HVM guest works without issue.

But when PV guests are not supported anymore then it seems now I should start to migrate our environment to use HVM guests.

May I ask what would be a good way to insert install information into a HVM guest? At the moment I would think about using xenstore to add information like IP to use and other information and create a custom Debian installer reading the information from the xenstore.
I know also about cloudinit, but what I have read so far I don't think it would fit into our environment.

Thanks,

[olivierlambert]

Packer is probably what you need @AtaxyaNetwork worked on it for some VMs here, feel free to ask her if you have question.

[bc-23]

Thanks for the hint, I will take a look into it.

[olivierlambert]

Link to the repo: https://github.com/ddelnano/packer-plugin-xenserver

[john.c]

@gb-123 said in XCP-ng 8.3 beta :

@john-c

Can you try installing ipv4 only and alternatively ipv6 only to see if it works ?
(Just wanted to confirm something)

@stormi @gb-123 When I tried installing as IPv4 only it was OK and came up correctly successfully. About to try IPv6 only will report back.

[john.c]

@john-c said in XCP-ng 8.3 beta :

@gb-123 said in XCP-ng 8.3 beta :

@john-c

Can you try installing ipv4 only and alternatively ipv6 only to see if it works ?
(Just wanted to confirm something)

@stormi When I tried installing as IPv4 only it was OK and came up correctly successfully. About to try IPv6 only will report back.

@stormi @gb-123 When I tried installing as IPv6 only its currently trying to bring up the management network.

EDIT - Update its still trying to bring up the connection with no success.

[john.c]

@stormi @gb-123 Just done a reboot from maintenance mode, to see if this gets it to bring up the IPv6 only management network.

UPDATE: Definitely didn't help still trying to bring up the IPv6 only management network.

[john.c]

@stormi @gb-123 The issue with management network being brought up on XCP-ng 8.3 beta1, dual stack (IPv4 and IPv6). Specifically it taking a really long time or not coming up at all, likely is due to the IPv6 side not completing successfully or in a reasonable timespan.

As shown by the problem on the IPv6 only install.