@Andrew that's planned

People with FreeBSD 13.2 or later as a guest are also now able to use Netlink to get their IP address changes reported instantly (rather than having to wait for as much as 1 minute).

This is however not yet part of our CI jobs, as we rely on cross-rs, which for now only provides FreeBSD 12 cross-build support. Your choices to try it:

• build it yourself
• manually-prebuilt binary

Don't forget to kldload netlink first, the Netlink support is not active by default!

@sb2014 this address reporting looks buggy, can you please share some guest information (OS type and version, guest agent version reported by XO)?

@Dataslak can you please request a commandline from GRUB (hit c on the boot menu), and issue the following commands:

echo $root
search --label --set root root-eqjpzg
echo $root

Raised this on IRC, resulting in:

I'll see what I can do, we may still have some time to include this before the release

I guess that's pretty much all we can do on our side (and that's nice from them ).

Catching up with the subject...

The good news first: the patch is already in the 6.8 stable branch (as c8b7b2f158d9d4fb89cd2f68244af154f7549bb4), and part of v6.8.5.

The Ubuntu situation is:

• master kernel branch OTOH has been at v6.8.4 for 12 days, with only minor packaging adjustments since then
• latest tag on master is 6.8.0-25 (only used for lowlatency kernel as of today), based on v6.8.1
• they also have a 6.8.0-26 and changelog entries for -27 on master-next (all about s390)
• standard kernel at 6.8.0-22, based on v6.8.1 too

I'm not familiar with their internal processes, but I suspect they're probably in a validation cycle for upcoming upcoming version right now, and any such fix would have to wait for next one.

The tracker ticket for 6.8.0-27 seems to imply -27 packaging is "in progress". Still digging...

@Tristis-Oris and with a 3-host pool I can reproduce you issue on 2nd attempt: new master loses its xapi process

When the master is changed, the xapi service is stopped and then restarted but something seems to get wrong this time.

Among issues I realize that my former-master shows this at the time of the failing switch:

Oct 14 15:20:50 xcpng83-bzkcpvhy xsh: [ warn||0 ||xsh] TLS verification is disabled on this host: /var/xapi/verify-certificates is absent

(while both other hosts do have that file)

daemon.log on new-master shows systemd desperately trying to restart xapi:

Oct 14 15:20:59 xcp-ng-hqerhcgv xapi-init[1244028]: Stopping xapi: [  OK  ]
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Unit xapi.service entered failed state.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: xapi.service failed.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: xapi.service holdoff time over, scheduling restart.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Cannot add dependency job for unit lvm2-activation.service, ignoring: Unit is masked.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Cannot add dependency job for unit lvm2-activation-early.service, ignoring: Unit is masked.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Starting XenAPI server (XAPI)...
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Started XenAPI server (XAPI).
Oct 14 15:20:59 xcp-ng-hqerhcgv xapi-init[1244047]: Starting xapi:
Oct 14 15:21:00 xcp-ng-hqerhcgv systemd[1]: xapi.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 14 15:21:00 xcp-ng-hqerhcgv xapi-init[1244078]: Stopping xapi: [  OK  ]

and xensource.log is very similar to yours. Congrats, that's a nice bug

@Tristis-Oris Oh OK. Had a try to run it several times myself (though on a 2-host pool), and I was able to see the operation performed twice, though apparently the second op did finish.
Running from the shell, if launch a second xe pool-designate-new-master while the first has not returned yet, it gets a DESIGNATE_NEW_MASTER_IN_PROGRESS error, but once it has returned there seem to be a window to do strange things.

@puneet336 you need to make your connection to your pool master (not your XOA guest), that's where the XAPI endpoint is.

If you did not deploy TLS certificates on your pool, XenAPI.Session(URL) will complain that the SSL certificate is self-signed (which you can workaround by adding ignore_ssl=True as you did above, but don't keep this in production).

You will also find a few guidelines to write a robust XenAPI client at https://xapi-project.github.io/xen-api/usage.html

@puneet336 @olivierlambert the lib is still working, but I'm usually use it with a local connection, running scripts inside dom0, with this code:

def get_xapi_session():
    session = XenAPI.xapi_local()
    try:
        session.xenapi.login_with_password('root', '', '', 'xcp-ng-tests session')
    except Exception as e:
        raise Exception('Cannot get XAPI session: {{}}'.format(e))
    return session

Will check what happens with a remote session.

@Ajmind-0 this was an install not an upgrade of 8.2.1, right? Was your previous working install of 8.2.1 in UEFI mode?

@Ajmind-0 does it still freeze if you add atexit=shell on the linux commandline? In some error situations the installer attempts to reboot the system after a few seconds, this will make it drop into a shell instead.

@Ajmind-0 there may be more details in the log a bit before those last lines, and they would not be visible because of the screen size and volume of logs. You can switch to the console with a shell and have a look into /tmp/install-log, there may be some more useful info in there.

@Tristis-Oris I also reproduced the issue on 8.2.1, and record the issue. As for your pool left in a tricky state, would it be reasonable for you to reinstall the impacted hosts?

@Tristis-Oris and with a 3-host pool I can reproduce you issue on 2nd attempt: new master loses its xapi process

When the master is changed, the xapi service is stopped and then restarted but something seems to get wrong this time.

Among issues I realize that my former-master shows this at the time of the failing switch:

Oct 14 15:20:50 xcpng83-bzkcpvhy xsh: [ warn||0 ||xsh] TLS verification is disabled on this host: /var/xapi/verify-certificates is absent

(while both other hosts do have that file)

daemon.log on new-master shows systemd desperately trying to restart xapi:

Oct 14 15:20:59 xcp-ng-hqerhcgv xapi-init[1244028]: Stopping xapi: [  OK  ]
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Unit xapi.service entered failed state.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: xapi.service failed.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: xapi.service holdoff time over, scheduling restart.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Cannot add dependency job for unit lvm2-activation.service, ignoring: Unit is masked.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Cannot add dependency job for unit lvm2-activation-early.service, ignoring: Unit is masked.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Starting XenAPI server (XAPI)...
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Started XenAPI server (XAPI).
Oct 14 15:20:59 xcp-ng-hqerhcgv xapi-init[1244047]: Starting xapi:
Oct 14 15:21:00 xcp-ng-hqerhcgv systemd[1]: xapi.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 14 15:21:00 xcp-ng-hqerhcgv xapi-init[1244078]: Stopping xapi: [  OK  ]

and xensource.log is very similar to yours. Congrats, that's a nice bug

@Tristis-Oris Oh OK. Had a try to run it several times myself (though on a 2-host pool), and I was able to see the operation performed twice, though apparently the second op did finish.
Running from the shell, if launch a second xe pool-designate-new-master while the first has not returned yet, it gets a DESIGNATE_NEW_MASTER_IN_PROGRESS error, but once it has returned there seem to be a window to do strange things.

@Tristis-Oris very strange, your log seems to show that 2 designate_new_master requests were handled one after the other (at 12:11:33 and 12:12:45), both to switch to the same host - and we see in the logs that while the 1st one got all phases executed (1, 2.1, 2.2), the second one starts to have issues during "Phase 2.1: telling everyone but me to commit". Sending a second request should indeed not trigger the whole thing again, so something apparently went quite wrong, but what is indeed not obvious.

@Tristis-Oris thanks!

I realize one thing is not clear to me: is the "unreachable master" in your situation the one you switched to, or the original master? And how many hosts in total are in this pool when you launch this operation?

The logs we'll need to analyze will be the xensource.log from the first occurrence of designate_new_master on both old and new master, until the time of that exception.

@Tristis-Oris said in XCP-ng 8.3 betas and RCs feedback :

then changed master

Could you please detail that part a bit more? I understand you used XO, but which screen and which action?

@Tristis-Oris Pool_role.This_host_is_broken is a bit scary, and more logs will be needed to understand how it got there. I'm trying to follow the same procedure as you to request the right lines - how exactly did you perform the master change?