XCP-ng 8.3 betas and RCs feedback 🚀

ThierryC01

After upgrading to 8.3. from 8.2, all backups have disappeared, I wanted to restore one VM but nothing is displayed at all. Even after importing the config file from 8.2, it is impossible to restore anything. There is not even a way to browse and direct the system to the location of previous backup files.

stormi

Where were the backups stored? The upgrade doesn't overwrite anything but the system partitions.

ThierryC01

@stormi My backups are stored on a remote SMB share.

ThierryC01

How can we re-attach an existing local SR that cannot be seen by the system? All my ISO files are local (/dev/sdb4) with no way to store them anywhere else.

Xcp-ng attach.png

florent

@ThierryC01 Can you browse the samba mount and check if the data are still there ?
The remote should have a xo-vm-backups folder containing one folder per VM

stormi

Regarding the local ISO SR, you need to mount /dev/sdb4 into a directory named the same as what you had done before the upgrade. The XAPI database should still have knowledge about this location and then be able to see the ISOs (possibly after a reboot, provided you added the mountpoint into fstab so that it's mounted at boot time).

ThierryC01

@florent Yes, all backup folders are present with all the data. All I did was just upgrade Xcp-ng from 8.2 to 8.3. I thought that performing backups before was a good idea... I have the backups but no mean to restore them

The backup page is blank in XOA, as well as the restore page. After importing the config backup, I noticed that the pool name was renamed (???) and only the backup list is displayed and disabled, after enabling all the backups, nothing is shown to restore.

florent

@ThierryC01 the backup aren't linked to the Sr , maybe there is a problem in the XO/Xoa

Can you show me a screenshot of the backup page ? I am not sure we're talking of the same thing

ThierryC01

@florent Sure, here are the screenshots after importing the config.json saved from 8.2 before the upgrade. The backup Jobs page was empty.

Xcp-ng backup_1.png

Xcp-ng backup_2.png

And this is the files on my remote share

Xcp-ng backups.png

florent

@ThierryC01 is the remote connected ? what happen when you click on "refresh backup list" ?

edit : it seems you have multiple remotes, each one with only one vm ?

ThierryC01

@florent Remote is connected all the time, it is a NAS and each VM has its own folder otherwise it is just a mess of mumble jumble folder names that no one knows which VM it is. Refreshing the backup jobs or restore page does nothing.

ThierryC01

@stormi I edited the fstab to mount my ISO SR and it is back online. Next time, I will remember to backup this file too before any upgrade.

florent

@ThierryC01 said in XCP-ng 8.3 beta :

@florent Remote is connected all the time, it is a NAS and each VM has its own folder otherwise it is just a mess of mumble jumble folder names that no one knows which VM it is. Refreshing the backup jobs or restore page does nothing.

ok, that's a first for me
The only way this can work is with multiples remote and job set in XO. Are they still there ? no error ?

ThierryC01

@florent I found the source of the problem, I already had it when I was trying the backup feature in 8.2 and it is with the remote settings: if there is a logon issue, there is no way to edit the password, save and go with it, you have to re-create the remote entirely which is extremely frustrating. For some reason, the password encryption does not behave as expected, when, editing the password, it is replaced by "obfuscated-xxxxxxx unique ID xxxxxxx" so it cannot work.

The message I get when trying to connect to a remote folder or just after editing the password and save
Xcp-ng remote_1.png

The obfuscated password displayed when editing
Xcp-ng remote_3png.png

And the error log for the logon attempt with the correct password:

remote.test
{
  "id": "365964ae-ea3c-41a5-80eb-0c5eb595f01d"
}
{
  "messageName": "session_setup_step2",
  "params": {},
  "message": "STATUS_LOGON_FAILURE (0xC000006D) : The attempted logon is invalid. This is either due to a bad username or authentication information.",
  "code": "STATUS_LOGON_FAILURE",
  "name": "Error",
  "stack": "Error: STATUS_LOGON_FAILURE (0xC000006D) : The attempted logon is invalid. This is either due to a bad username or authentication information.
    at SMB2Forge.request (/etc/xen-orchestra/node_modules/@marsaud/smb2/lib/tools/smb2-forge.js:22:15)
    at /etc/xen-orchestra/node_modules/@marsaud/smb2/lib/tools/smb2-connection.js:108:11
    at Object.<anonymous> (/etc/xen-orchestra/node_modules/@marsaud/smb2/lib/tools/message.js:15:15)
    at Socket.<anonymous> (/etc/xen-orchestra/node_modules/@marsaud/smb2/lib/tools/smb2-forge.js:71:31)
    at Socket.emit (events.js:400:28)
    at Socket.emit (domain.js:475:12)
    at Socket.patchedEmit [as emit] (/etc/xen-orchestra/@xen-orchestra/log/configure.js:118:17)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at Socket.Readable.push (internal/streams/readable.js:206:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
    at TCP.callbackTrampoline (internal/async_hooks.js:130:17)"
}

florent

@ThierryC01 the obfuscated- part is added to the data sent to the web client and the logs

But , in theory, editing a remote with a new password will change the password.

You can check the data saved in the XO database by executing yarn xo-server-db ls remote from your xo folder

olivierlambert

@ThierryC01 said in XCP-ng 8.3 beta :

mess of mumble jumble folder names that no one knows which VM it is

That's really not a good idea to do one BR per VM, as you discovered. The current folder structure is here for a reason, it's not a "mess of mumble jumble folder names": it's a stable structure meant to be parsed by XO, not by a human.

You are currently fighting/bending against the tool, this has consequences.

ThierryC01

@olivierlambert That is not true, a remote folder is still a remote folder and we should be able to backup any VM wherever we need/want. The issue here is when the login fails once, there is no way to edit the logon password and succeed. It was working very well until I upgraded from 8.2 to 8.3 and then had to restore a VM from backup. No grudge here, just a fact.

What I call a mumble jumble naming of folder is just that every VM is identified by a uuid in the same folder if all backups are stored in the same place which makes it difficult to sort should we need to either use a backup on another platform or delete to regain disk space.

I am just fighting to get a basic feature working: edit the password of a connection without having to delete the record and start over.

ThierryC01

@florent That is the theory yes, but in reality, editing the password fails.

ThierryC01

Okay... I have to apologize for the pollution of this topic, I was running a container version of XOA (Xen Orchestra, commit 422a2) which I use in emergency cases such as my actual up to date XOA VM which crashed due to an LVM issue and was trying to restore. I managed to get my actual XOA VM running and everything is fine, no problem whatsoever, all backups are here using the XOA VM, available to restore and all. I really feel bad for wasting everyone's time. I will update my emergency XOA container from now on, I thought that all information were stored on the Xcp-ng server, good to know.

laurentm

@ThierryC01

About your LVM crash, was it with NVMe U2 SSD ?
Running XenServer from more than ten years and XCP-NG from 7.5, the worst issue I encounted has been recently with a very brand new DELL R650 fitted with some excellent Kingston U2 NVMe DC1500M,
connected straight to the motherboard, without any hardware raid controller. It appears that NVMe can disappear randomly due to some stupid power save feature in Linux kernel. I have patched that and I'm using each ssd separetely, in order to reduce the risk, and without LVM at XCP-NG level, only in my Linux VMs.