dthenot

dthenot

Hello,

I'm a new developer on XCP-ng, I'll work on the Xen side to improve performance.
I'm a newly graduated of University of Versailles Saint-Quentin with a specialty in parallel computing and HPC, I have a big interest in operating systems.

dthenot

Hello,

As some of you may know, there is currently a problem with disks with blocksize of 4KiB not being compatible to be a SR disk.
It is an error with the vhd-util utilities that is not easily fixed.
As such, we quickly developed a SMAPI driver using losetup ability to emulate another sector size to be able to workaround the problem for the moment.

The real solution will involve SMAPIv3, which the first driver is available to test: https://xcp-ng.org/blog/2024/04/19/first-smapiv3-driver-is-available-in-preview/

To go back to the LargeBlock driver, it is available in 8.3 in sm 3.0.12-12.2.

To set it up, it is as simple as creating a EXT SR with xe CLI but with type=largeblock.

xe sr-create host-uuid=<host UUID> type=largeblock name-label="LargeBlock SR" device-config:device=/dev/nvme0n1

It does not support using multiple devices because of quirks with LVM and the EXT SR driver.

It automatically creates a loop device with a sector size of 512b on top of the 4KiB device and then creates a EXT SR on top of this emulated device.

This driver is a workaround, we have automated tests but they can't catch all things.
If you have any feedbacks or problems, don't hesitate to share here

dthenot

@ph7 It's only enabled for the two yum command with the --enablerepo explicitly used.
It's disabled in the config otherwise.
No need to do anything

dthenot

@gb.123 Hello,
The instruction in the first post are still the way to go

dthenot

@nasheayahu Hello,

I do this in my homelab with a disk since I imported it from a physical machine:

mkdir /srv/NAS
xe sr-create type=udev device-config:location=/srv/NAS name-label="NAS Disks" host-uuid=<UUID of the host with the disks>

Then you make a symlink to the device:

ln -s /dev/sda /srv/NAS/sda #although it might be better to use a stable identifier if you have multiple disks
xe sr-scan uuid=<UUID of the udev SR>

The disk will appear as a VDI in the SR that you can then plug to a VM.

I also renamed the VDI [NOSNAP][NOBAK] NAS Drive
[NOSNAP][NOBAK] instruct XO to not snapshot and export the disk in backups (since it can't).

It's not passthrough, as in the disk is not directly given to the VM, but tapdisk (the process virtualizing storage) will give access to the whole disk to the guest, so it can mount any FS on it.

The udevSR is usually used for plugging USB from what I could gather from existing code. But users in homelabs have been using it like this for a while.
Though you will likely not have native performance level since it's not exactly passthrough.

dthenot

@Andrew Hello,

I have been able to find the problem and make a fix, it's in the process of being packaged.
I can confirm it only happen for file based SR when using purge snapshots.
For some reason, the vdi type of CBT_metadata is cbtlog for FileSR but stays the image format it was for LVMSR
And it would make a condition fail during the list_changed_blocks call.

dthenot

@manilx Hi,

yum install plug-late-sr

Should do the trick to install it

dthenot

@olivierlambert In 8.2 yes, linstor sm version is separated, it's not the case in 8.3 anymore.

dthenot

@ccooke Hello,

We have a fix, we are aiming to validate it rapidly so it shouldn't happen for others.
Thank you for reporting the issue. I'll update the thread again when the update is available and it should be safe for other people going through here to update using the RPU.

dthenot

@olivierlambert I am

dthenot

@racom.jiristerba
Sorry, I missed answering your questions the first time

Is the tapdisk QCOW2 commit failure “The device is not writable: Permission denied” a known issue with these package versions on shared LVM/iSCSI storage?

Yes, it's a known issue, most of them will be fixed with the latest update, you might need to activate the VDI again but a migration during RPU will be enough

Why does rollback attempt to deactivate the active guest LV while tapdisk still holds it open?

It's because in the case of VHD, it's the case but with QCOW2 we don't stop the tapdisk process accessing the VDI, it's a change that was missed

Is the qcow2OLD_<UUID> naming seen in rollback expected, compared with QCOW2-OLD_<UUID> seen during successful cleanup?

It's a pre-existing bug that I already have on my TODO list

Is there a supported update, hotfix, or workaround for this configuration?

Installing the latest release sm-3.2.12-25.1 during updates (and maybe launching a xe sr-scan uuid=<SR UUI> after updating it so it auto-resolve the undo) should be enough

What additional logs are required to identify the original failure of the second QCOW2 VM?

I don't think we need any more logs since the errors I'm seeing should already be fixed.
If you have any more issues after installing the newest packages, I will take another look

What is the recommended recovery procedure without a guest outage, and how should we validate the disk chains before resuming snapshot backups?

The sr-scan after updating should do it automatically, it shouldn't need any manipulation. Looking at the storage logs in /var/log/SMlog for any irregularities could help to see problems.

dthenot

@zorro said:

*** finished leaf-coalesce successfully

This is misleading, this is printed by _finishInterruptedCoalesceLeaf that is handling cleanup after the leaf coalesce failed.

Thank you for the information, we have a fix for the interrupted coalesce leaf in the next release that might help, I'll also try to reproduce to see if it's the same behavior.

dthenot

@racom.jiristerba Hello, this appear to be bugs we are already fixing in the build currently in the testing repository. Please see this post for instruction to install the updates : https://xcp-ng.org/forum/post/108803

These two bugs are: the chain being RO sometime which the tapdisk process doing the coalesce can't access and a bug with the LV being in use when it's trying to refresh the VDI it was coalescing and it's parent following the cleanup. Both of those are fixed in sm-3.2.12-25.1.xcpng8.3

dthenot

@samuelolavo Hello,

Could the fact that this is a local LVM SR on a non-master host be relevant to how make_chain_rw is being handled?

Yes, it's indeed what's happening. The check for the SRMaster (for a shared SR it's the master but for a local it's supposed to be done on the host directly) is failing and always saying False. I'll fix this.

Thank you for the help

dthenot

@samuelolavo Hello,

It is indeed linked to QCOW2.

Is mixing VHD and QCOW2 VDIs on the same VM and on the same SR fully supported?

Yes, it's supposed to work.

Are there specific minimum versions of xapi, sm, blktap, or other storage components required for large QCOW2 VDIs?

Yes, but we would prefer for people to be uptodate, we have been fixing things.
But I don't think it's a problem for you at the moment.

make_chain_rw is necessary for the master to make the QCOW2 LV RW at the VDI activation time. So it's likely linked to this particular QCOW2.

make_chain_rw is executed on the master host and is logging in /var/log/SMlog, could you take a look at what it's erroring on?

dthenot

@Dan Hello,

So our investigation found that this change was introduced with commit https://github.com/xcp-ng/sm/commit/00637dd52e845d6016add9718fc9cc694aec9f0d
It was aimed at EXTSR in particular, it appear that the first SR to be plugged is the one choosing the mode of sr-mount since util.makedirs also create the parent directory with the given mode.

I have created a card for the issue on our side.

dthenot

@Dan Hello,

I can't answer from memory, we'll take a look at this and come back to you

Thanks for the report, I guess it could be considered a bug since even it's one of the other, it's not the expected behaviour to have the mode change between runs.

dthenot

@bvitnik It's specific to XCP-ng.

dthenot

@bvitnik Hello,

sm-config:vdi_type is the pre-existing one so we are still writing it for compatibility reasons.
But you might notice that vdi_type is only written for LVM-based SRs.
While image-format is always written in XCP-ng with the QCOW2 feature.

So it might be safe to just look at image-format if it exist and if it's not, it's likely a VHD.
On LVMSR, you can fallback on vdi_type (and it should work for XS too).

Indeed, ISO SRs do not advertise a VDI format. Technically, ISOs could be considered to be raw.

It should also be noted that our implementation of QCOW2 is very different from GFS2 QCOW2 of XenServer.
Our implementation uses the existing SR types to allow VHD and QCOW2 to co-exist.

dthenot

@probain Hello,

It's likely linked to the List index out of range bug.
That bug was linked to the SR scan failing to introduce CBT_metatadata VDI in the XAPI database, could you try to launch a xe sr-scan uuid=<SR UUID> and try again to disable CBT?
If it does not work, could you share the /var/log/SMlog of around the time you are trying to disable CBT?

dthenot

@dthenot