NVMe storage wrong detection

laurentm

I have a DELL R820 retrofitted with a U2 NVMe backplane (with one single U2 SSD for now) and an add-on cheap PCIe U2 adapter (with one other U2 SSD). I did that for testing the perfs, which are twice better than a SATA SSD but the same for the backplane and the add-on card, restricted to gen2 lanes.

I installed XCP-NG 8.3beta1 via IDRAC on a small SATA SSD (in a false cdrom drawer adapter) and, during installation, both U2 NVMe were properly detected and choosen as EXT storage for VMs.

In XOA, only one storage (7,6 TB) appears with this combined path
/dev/disk/by-id/nvme-KINGSTON_SEDC1500M7680G_50026B72830CEBE9,/dev/disk/by-id/nvme-KINGSTON_SEDC1500M3840G_50026B72830167A0
but the 3,8 TB disk is not showing its capacity.

So, I had to forget this storage and recreate one after one two EXT storage.

I know that my use case is outside an average setup but I do not understand why XCP-NG is aware of boths disks
but only takes in consideration the larger one.

Forza

@laurentm Xcp-ng doesn't support 4Kn sectors. Perhaps this is why only one one of the disks could be used.

Danp

Sounds similar to this recent thread, where the SR was spanning multiple drives.

davemcl

@laurentm
List the LBA format of drive
nvme id-ns -H /dev/nvme0n1

Youre probably not set to 512b sector size. Just format it using same cli tool.
nvme format /dev/nvme1n1 -l $ID
where $ID is the id of the desired sector size.

laurentm

@davemcl said in NVMe storage wrong detection:

List the LBA format of drive

Thanks, I'm new to NVMe storage in XCP-NG and did not even know the nvme-cli (which is not installed by default in CentOS and XCP-NG). It looks like I'm already using 512b sector size:

/dev/nvme0n1 50026B72830CEBE9 KINGSTON SEDC1500M7680G 1 7.68 TB / 7.68 TB 512 B + 0 B S67F0103
/dev/nvme1n1 50026B72830167A0 KINGSTON SEDC1500M3840G 1 3.84 TB / 3.84 TB 512 B + 0 B S67F0103

In the last 15 years, with my previous DELL servers, the RAID controller was doing everything and volumes were perfectly seen by XenServer and XCP-NG. When I bought a full NVMe RS650XS, I did not ordered any RAID controller and only one DELL U2 960 GB to save money because DELL U2 prices are crazy. Instead I bought 4 really cheap KINGSTON DC1500M, which are less expensive than their SATA counterpart (DC600M). Unfortunately, the LVM choice at install leads me to crash half of the VMs after one month !

Now, I'm using EXT and many separate disks, this is far for ideal because I waste room for more VMs but it seems reliable
and these large U2 SSDs are really cheap, while the slower SAS SSDs are still pricey for an unknown reason.

Danp

Have you tried removing the original SR and then creating two new SRs, one for each device?

laurentm

@Danp Yes I did that and it looks good now.