XCP-ng 8.3 betas and RCs feedback 🚀

yann

@Tristis-Oris and with a 3-host pool I can reproduce you issue on 2nd attempt: new master loses its xapi process

When the master is changed, the xapi service is stopped and then restarted but something seems to get wrong this time.

Among issues I realize that my former-master shows this at the time of the failing switch:

Oct 14 15:20:50 xcpng83-bzkcpvhy xsh: [ warn||0 ||xsh] TLS verification is disabled on this host: /var/xapi/verify-certificates is absent

(while both other hosts do have that file)

daemon.log on new-master shows systemd desperately trying to restart xapi:

Oct 14 15:20:59 xcp-ng-hqerhcgv xapi-init[1244028]: Stopping xapi: [  OK  ]
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Unit xapi.service entered failed state.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: xapi.service failed.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: xapi.service holdoff time over, scheduling restart.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Cannot add dependency job for unit lvm2-activation.service, ignoring: Unit is masked.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Cannot add dependency job for unit lvm2-activation-early.service, ignoring: Unit is masked.
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Starting XenAPI server (XAPI)...
Oct 14 15:20:59 xcp-ng-hqerhcgv systemd[1]: Started XenAPI server (XAPI).
Oct 14 15:20:59 xcp-ng-hqerhcgv xapi-init[1244047]: Starting xapi:
Oct 14 15:21:00 xcp-ng-hqerhcgv systemd[1]: xapi.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 14 15:21:00 xcp-ng-hqerhcgv xapi-init[1244078]: Stopping xapi: [  OK  ]

and xensource.log is very similar to yours. Congrats, that's a nice bug

Tristis Oris

@yann yay!

Strebor

Just a note to say thanks to all involved! Excellent job!

I yesterday upgraded our 8.3 release pool to 8.3 production. 2x Intel NUC11TNKi5. Running very nicely! The upgrade from release candidate with the ISO written to USB media went smoothly. Also the yum updates after the upgrade.

Now looking forward to see XO Lite be completed!
Keep up the great work!

yann

@Tristis-Oris I also reproduced the issue on 8.2.1, and record the issue. As for your pool left in a tricky state, would it be reasonable for you to reinstall the impacted hosts?

Tristis Oris

@yann yep, already reinstall it.

gb.123

Installed in my Homelab. Install went smoothly.

Small bugs I have noticed :
1.
XCP-NG Install Error.jpg

This ONLY happens when using KVM and 'scaling' the display to fit window.

XCP-NG Update Error.jpg

This happens ONLY on the first boot after applying patch ( 6 programs ) once release iso is installed.
This goes away from the second boot.

Everything seems to be working smoothly for now. Kudos to the Vates Team !

gb.123

This post is deleted!

stormi

Regarding xsconsole, I think it always had this issue where at some point it thinks (rightfully or not) that the screen is too small for it to run.

Tristis Oris

Already every VM at 8.3 pool stuck at control domain. Can't detach them without pool reboot. Any workaround?

vbd.delete
{
  "id": "7af4b39d-d6aa-2589-509f-f082a1f0f2bb

"
}
{
  "code": "OPERATION_NOT_ALLOWED",
  "params": [
    "VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

'"
  ],
  "call": {
    "method": "VBD.destroy",
    "params": [
      "OpaqueRef:65481d14-d3de-82d7-fa7d-95131f14a209

"
    ]
  },
  "message": "OPERATION_NOT_ALLOWED(VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

')",
  "name": "XapiError",
  "stack": "XapiError: OPERATION_NOT_ALLOWED(VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

')
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410181215/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202410181215/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Ajmind 0

I have tried to install an old Intel Server (SR2625/S5520UR / Dual Xeon E5645) running XCP-NG XCP-NG 8.3 beta with 8.3 release ISO:

The Server was first running XCP-NG 8.2.1 and was upgraded with 8.3 beta 2 ISO.

I could run the installation proccess until

completing installation 50%

And nothing happens further:
PXL_20241021_125706035.MP (Groß).jpg

Install media was freshly created and booted in EFI mode.
Any idea how to finish this process?

Finallf

@Ajmind-0
try running the installation in safe option.

jhansen

Version XCP-NG 8.3 RC
XO lastest version

Hello,
I use XO with Delta Backup to back up my VM's. I have activated NBD and CBT in the backup. The backups run via the management interface with NBD activated.
The first full backup runs error-free.
The second backup with delta gives me the error message cleanVm: incorrect backup size in metadata and VDI_IN_USE and the backup ends with an error.
After the delta backup, I then have snapshots on the storage that are blocked with "Control domain on host xxx". I can then no longer delete these. The only option is to reboot all Xen servers in the pool, after which the snapshots are still there, but can then be disconnected and forget to get them deleted.
I was able to reproduce this error on different pools with different servers, all 8.3. It is always different VMs that trigger the error and others that are backed up without errors.
If I delete all control domain blocked snapshots and also the purged snapshot from the storage, the next full backup runs error-free again.
Does anyone have the same problem and maybe a solution?
regards Joerg

jhansen

I saw that Tristis Oris seems to have a similar problem with snapshots.

Something else, when updating from 8.2.1 to 8.3 RC.
My 8.2.1 servers were still running on BIOS boot. When updating to 8.3 RC I got the message that there was still a DOS table on the hard drive and the update from the USB stick was aborted.
A newly installed 8.3 server with UEFI cannot be integrated into the old 8.2 pool.

Here is a small workaround:
2 USB sticks, one with 8.2.1 and one with 8.3 RC.
Take one server out of the pool, not the master.
Reinstall with UEFI on 8.2.1.
Integrate server into the old pool.
make server as new pool master.
The server can now be updated to 8.3 without any problems, the master is on UEFI and all pool information and VM are on 8.3.
Now just take all other servers out of the pool, do a clean 8.3 UEFI installation and put them back into the pool.

Maybe that will help others with the same problem.

regards Joerg

Mathieu

@Tristis-Oris
It seems that since updating to 8.3, I also have VDI attached to control domain I cannot delete, they appear after a backup.
The only solution to forget them is to a pool reboot.

stormi

@jhansen said in XCP-ng 8.3 betas and RCs feedback :

Something else, when updating from 8.2.1 to 8.3 RC.

Are you updating to 8.3 RC when the official 8.3.0 is now released?

stormi

Given the diversity of issues reported above, and the fact that XCP-ng 8.3 is now released, I strongly suggest you all create new topics in the forum. You can reference them once in this thread to help them get attention, no problem with that.

stormi

@jhansen said in XCP-ng 8.3 betas and RCs feedback :

My 8.2.1 servers were still running on BIOS boot. When updating to 8.3 RC I got the message that there was still a DOS table on the hard drive and the update from the USB stick was aborted.
A newly installed 8.3 server with UEFI cannot be integrated into the old 8.2 pool.

Here is a small workaround:
2 USB sticks, one with 8.2.1 and one with 8.3 RC.
Take one server out of the pool, not the master.
Reinstall with UEFI on 8.2.1.
Integrate server into the old pool.
make server as new pool master.
The server can now be updated to 8.3 without any problems, the master is on UEFI and all pool information and VM are on 8.3.
Now just take all other servers out of the pool, do a clean 8.3 UEFI installation and put them back into the pool.

Let's add that there must be no VMs on local storage on the server you reinstall, of course. Other than that, I like this workaround!

CC @yann, do you think it's worth adding this to the release notes?

jhansen

@stormi
Sorry, I was a bit confused this morning when I saw all the errors. Of course it wasn't an update to 8.3 RC but to 8.3.0 Official.
Very strange thing, I checked my test lab server, it's the only pool that doesn't have the error. Full backup and several delta backups, all OK.
The only difference between this and the other pools is that this one was running with 8.3 RC2 and was only updated to 8.3.0 official with yum update.
I have now cleaned a faulty pool, removed all snapshots, removed the NDB and CBT in the backup and removed the NBD interface in the pool. I'm currently doing a full backup on this pool and will then do several deltas to narrow down the error a bit.
Regards, Joerg

Ajmind 0

@Finallf

already done but with same result.

jhansen

@stormi
Of course, you shouldn't have a VM on the local storage.
It's also a good idea to write down the IPs, MACs and the number of the NICs beforehand. It can happen during the new installation that the order of the network cards changes.
Final tip: if you use iSCSI storage and use the iSCSI IQN as identification, you should also write this down beforehand and set it on the newly installed server before you put the server in the pool, otherwise you will have problems with the iSCSI storage.