XCP-ng 8.2.0 beta now available!

hoerup

@stormi whats the opinion on using the new gluster & ceph drivers for a hyper converged solution? Ok / should be ok but not tested / not recommended ??

stormi

@hoerup Should be ok but not tested enough.

olivierlambert

This is not trivial if you don't have enough Ceph/Gluster knowledge and a little bit of toolstack knowledge. Also, it won't survive an upgrade.

krakers

@olivierlambert do you have similar timeline plans to 8.1 for 8.2? I mean ~2 months between beta and final release?

olivierlambert

This will depend on what's going on, but that sound a reasonable timeframe (no promises though )

technot

Is there a Xcp-NG Center compatible version compiled yet ?

I had a little problems upgrading to the 8.2 beta.
The main tty showed a normal bootup untill a point where it showed a bunch of ACPI errors,
it worked for 10-15 min on starting previously disabled devices, and then just stopped.
I let it work another 10-15 min but all that came was this error, over and over again:

[  639.270804] EDAC sbridge: Seeking for: PCI ID 8086:3cf5
[  639.270804] EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
[  639.270804] EDAC sbridge: Couldn't find mci handler
[  639.270804] EDAC sbridge: Failed to register device with error -19.

I switched to tty2 and lunched the init script from /opt/xensources and managed to upgrade from there. Wasnt too easy as the screen kept getting garbled from debug info that I assume default to tty2 output.

After the upgrade and reboot, it boots. But it takes a long long time.
Here is a small snippit from the first boot dmesg:

[   29.618412] usb 1-1.1.1: Product: G19 Gaming Keyboard
[   29.623569] input: G19 Gaming Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.0/0003:046D:C228.0003/input/input7
[   30.290247] hid-generic 0003:046D:C228.0003: input,hidraw2: USB HID v1.10 Keyboard [G19 Gaming Keyboard] on usb-0000:00:1a.0-1.1.1/input0
[   30.293954] input: G19 Gaming Keyboard Consumer Control as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.1/0003:046D:C228.0004/input/input8
[   30.958193] hid-generic 0003:046D:C228.0004: input,hiddev97,hidraw3: USB HID v1.10 Device [G19 Gaming Keyboard] on usb-0000:00:1a.0-1.1.1/input1
[  129.058404] EXT4-fs (sdb1): mounting ext3 file system using the ext4 subsystem
[  129.063909] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[  169.228203] e1000e 0000:00:19.0 0000:00:19.0 (uninitialized): registered PHC clock
[  169.356510] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) 8c:89:a5:a3:b6:f3
[  169.356512] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection

And its not done and ready untill it tries to initialize samba SR:

[  453.242190] xenbr1: port 1(eth1) entered blocking state
[  453.242190] xenbr1: port 1(eth1) entered forwarding state
[  476.236967] FS-Cache: Loaded
[  476.254307] FS-Cache: Netfs 'cifs' registered for caching
[  476.254307] Key type cifs.spnego registered
[  476.254307] Key type cifs.idmap registered

All in all this is much better then 8.1 and 8.0 where I had similar problems with ACPI, but it never got any further, and just locked up, no tty2 or such.
I actually had to use ACPI=off and/or ACPI=strict, and either way, the whole experience after upgrading was super slow vms, but with no aparant error messages so I returned to 7.6.
Well see how 8.2 fares, but so far so good
I have attached both the dmesg and xl-dmesg from the installer, incase you want to review what takes so long during boot.
xl-dmesg-log.txt
dmesg-log.txt

Thanks for all you hard work

brezlord

I upgraded to 8.2.0-beta to fix this issue nfs-storage-latency. Bellow are some tests done, if you need any specific tests just let me know.

I’m running XO from sources xo-server 5.67.0 xo-web 5.71.0

XEN-HOST-01 Dell R720
XEN-HOST-02 Dell R710
TrueNAS NFS Storage over 10GbE

I upgraded from 8.1.0 patched to 8.2.0-beta via ISO install. Install went through with no troubles. The host booted with no problems. All networks and storage was present.

All VMs booted and came back up with no issues and guest tools running. VMs consist of CentOS 7 & 8, Debian 9 & 10, Windows 10Pro

Successfully live migrated VMs from 8.1.0 host to 8.2.0-beta with no issues.

Attempted VM migration from 8.2.0-beta to 8.1.0 host failed as expected.

Successfully migrated VMs between 8.2.0-beta hosts.

Could not change pool master from 8.2.0-beta to 8.1.0 host. Get Unknown error from peer.

Successfully changed pool master with both hosts on 8.2.0-beta.

Both host on 8.2.0-beta, shut down host-02 which was not the pool master and the shared storage shows partially connected and no stats available for storage. Reboot host-02 and shared storage is now healthy and stats are back. Disabled host-02 and shutdown and still had the same issue as above. The only was to get stats back and get shares storage back to healthy and stats working is to forget the storage attached to host-02 that was shut down.

XCP-ng Center v20.04.00.32 will not connect to 8.2.0-beta hosts.

Had to reinstall xscontainer package to have docker container visibility in XO.

stormi

@technot said in XCP-ng 8.2.0 beta now available!:

Is there a Xcp-NG Center compatible version compiled yet ?

I don't think there is one. It depends on the community. Previous releases were built by @borzel. I don't know if he will have the time to do it this time.

demanzke

I just updated using the ISO-method because the yum method didn't boot successfully.
What would be the best way to migrate from a ZFS file-SR to ZFS-SR? I assume setting the SR type after creation isn't intended and I can't find that in the XAPI docs either.
Is the cleanest solution to create a new ZFS dataset, ZFS-SR on top and move the virtual drives?

stormi

I would have been interested in debugging why the system didn't boot successfully after an update using yum. Can you give us more details?

To migrate the ZFS SR, the only clean way I am aware of is to create a new one. There may be a hacky way but it would probably involve modifying XAPI database objects dans metadata written in the SR itself so I wouldn't attempt it.

stormi

@brezlord said in XCP-ng 8.2.0 beta now available!:

Could not change pool master from 8.2.0-beta to 8.1.0 host. Get Unknown error from peer.

When you get such error in XO, there's usually more information in XO's log page. However, it is expected that you can't move the master to a host running a lower release.

Both host on 8.2.0-beta, shut down host-02 which was not the pool master and the shared storage shows partially connected and no stats available for storage. Reboot host-02 and shared storage is now healthy and stats are back. Disabled host-02 and shutdown and still had the same issue as above. The only was to get stats back and get shares storage back to healthy and stats working is to forget the storage attached to host-02 that was shut down.

Ok, I don't think it's a regression but it may be interesting to create a GitHub issue with all the details.

Had to reinstall xscontainer package to have docker container visibility in XO.

That is expected, unless you upgrade using yum.

if you need any specific tests just let me know.

If your host has an nVIDIA GPU, we would be interested in knowing it because there used to be bugs specific to hosts with such a GPU, even when not used.

And be ready for testing UEFI VMs soon! I'll post here when the updated uefistored package is available.

brezlord

Thanks for the info. So doing a ISO update is essentially like a fresh install but keeps settings.

I don't have any nVIDIA GPUs, sorry can't help there.

I'll boot the second host and get more info to raise an issue for the SR problem.

brezlord

I have raised issue https://github.com/xcp-ng/xcp/issues/439 for the above SR problem.

brezlord created this issue in xcp-ng/xcp

open Shared storage & XCP-ng tools show partially connected when 1 of 2 hosts shutdown. #439

technot

And be ready for testing UEFI VMs soon! I'll post here when the updated uefistored package is available.

Does the host have to be booted in UEFI mode for UEFI VMs to function?

If it does, is there an easy way to make the change from legacy to UEFI?
All I know, is the install docs clearly state you should never change mode when upgrading.
Is the only way to export all the vms, and reinstall, then import the vms back?

stormi

@technot said in XCP-ng 8.2.0 beta now available!:

Does the host have to be booted in UEFI mode for UEFI VMs to function?

No, it's not related to the host's mode.

technot

@stormi
Is it only lacking to get built/compiled? If so I could do it sometime during the weekend. Or has the code not been updated to reflect any changes done with xcp-ng 8.2. If the latter, I am afraid I cant help.

stormi

It's not ready yet so you can't help now. Our developer who wrote that piece of code (uefistored) is chasing the last known bugs and then I'll build an update and push it to our RPM repository for XCP-ng 8.2. I'll announce it when it's ready.

technot

I dont know if this is expected behaviour, after upgrading all the way from 7.6 to 8.2 beta. But on a windows guest I have an AMD gpu in full pci passthrough, the device didnt show up after boot. And instead there was another device showing fault.
Intel 8237sb pci to usb universal host controller, was throwing an error 43.
And under Display adapters, there was only the microsoft remote emulated and a "Microsoft Basic" devices showing.
I reinstalled the AMD driver, and rebooted. And it came back to life.

Also, I must say the vms feel snappier. Faster boots, and less delay/latency. But I have not meassured pre/post, so its only a feeling!:D

technot

@stormi
Im sorry, it wasnt clear. What I meant is XCP-ng Center. If that code has been updated, or just a question of compiling it.

borzel

@stormi Heyho! I'll try to find some time. But first I'll setup an XCP-ng 8.2 host