ISO modification with additional RPM for NIC

LennertvdBerg

I'm fairly new to XCP-NG and would like to build a custom ISO for XCP-NG where I can add an additional RPM for a Mellanox ConnectX-6 Lx 10/25GbE SFP28. The problem is that I don't have other NICs installed and I can't install XCP-NG 8.2.1 because it detects during installation that there's no NIC in the system. I can install XCP-NG 8.3 beta 2 as the drivers are included there. So, I would like to include the drivers for the Mellanox in the ISO so that during installation the process will automatically detect it and I can run the installation.

In xcp-ng-8.3.0-beta2, there's an additional mellanox-mlnxen-5.4_1.0.3.0-4.xcpng8.3.x86_64.rpm in the Packages/ directory. In xcp-ng-8.2.1-20231130, there is no mellanox-mlnxen*.rpm at all. I found two Mellanox RPMs at Koji;

mellanox-mlnxen-alt-5.4_1.0.3.0-1.xcpng8.2.x86_64.rpm (https://koji.xcp-ng.org/buildinfo?buildID=2620)
mellanox-mlnxen-alt-5.9_0.5.5.0-1.1.xcpng8.2.x86_64.rpm (https://koji.xcp-ng.org/buildinfo?buildID=2868)

I tried following the instructions for ISO modification mentioned in the XCP-NG ISO modification documentation

First, I extracted the ISO using the following commands:

mkdir tmpmountdir/
mount -o loop filename.iso tmpmountdir/ # as root
cp -a tmpmountdir/. iso
umount tmpmountdir/ # as root
chmod a+w iso/ -R

Then, I used wget to download the RPMs into the Packages/ directory. After this, I updated the repodata/ using the following command (remember to install createrepo-c first):"

sudo apt install createrepo-c
rm repodata/ -rf
createrepo_c . -o .

Finally, I built the ISO using the instructions given in the XCP-NG documentation:

#OUTPUT=/path/to/destination/iso/file # change me
OUTPUT=/home/xcp-ng/new_iso/xcp-ng-8.2.1-20231130-mod.iso
VERSION=8.2 # change me
genisoimage -o $OUTPUT -v -r -J --joliet-long -V "XCP-ng $VERSION" -c boot/isolinux/boot.cat -b boot/isolinux/isolinux.bin \
            -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e boot/efiboot.img -no-emul-boot .
isohybrid --uefi $OUTPUT

However, when I use this ISO, the Mellanox ConnectX-6 Lx drivers do not load during installation.

Also, I have seen on the Nvidia website that new drivers for the ConnectX-6 Lx are available for Citrix XenServer Host 8.2 in version mlnx-en-23.10-2.1.3.1-xenserver8.2-x86_64.

So my questions are:

What am I doing wrong with building the ISO and including the RPMs?
Is it possible to include the mlnx-en-23.10-2.1.3.1-xenserver8.2-x86_64 for XCP-NG 8.2?
What steps do I need to take, and how?

Danp

Have you tried booting with the alternate kernel? https://docs.xcp-ng.org/installation/hardware/#during-system-installation

Did you read the docs about creating a custom ISO? https://docs.xcp-ng.org/project/development-process/ISO-modification/

Edit: You could also press F9 during the install process to install custom network drivers.

olivierlambert

Adding @stormi in the loop

LennertvdBerg

@Danp, UPDATED: I tried booting with an alternate kernel in XCP-NG 8.2.1 and XCP-NG 8.3 beta 2, but it didn't load the Mellanox ConnectX-6 Lx 10/25GbE drivers.

Yes, I've read the documentation about creating a custom ISO and have detailed my procedure above. The only part I'm unsure about is this:
"you need to add new RPMs not just replace existing ones, they need to be pulled by another existing RPM as dependencies. If there's none suitable, you can add the dependency to the xcp-ng-deps RPM."
I couldn’t realize or understand this step.

LennertvdBerg

@stormi could you maybe advise what I'm doing wrong?

olivierlambert

@stormi is just back today, I'll come back to you soon

LennertvdBerg

@olivierlambert Thanks.

stormi

I haven't forgotten you, but have been busy. Still need help?

LennertvdBerg

@stormi Hi, some help is welcome Still haven’t found a solutions.

stormi

Do you absolutely need all in one ISO? Otherwise there's a rather simple solution with an unmodified ISO + a driver ISO. We even already provide one: https://updates.xcp-ng.org/isos/drivers/8.x/

LennertvdBerg

@stormi I thought it’s convenient to have all in one as it’s easy for installation. But I can check this options as well. So you recommend to extract the iso to a separate USB drive and load drivers from there?

stormi

Yes, without any network connectivity available, that's the main option.

Regarding modifying the installation ISO, you'd need two things:

modify install.img to put the driver in it. That's the easiest part and you can use https://github.com/xcp-ng/xcp/tree/master/scripts/iso-remaster (see the example script in samples/). We still need to update the online docs to mention this option (ping @yann). This will give you network connectivity during the installation, but won't install the driver on the system.
install the driver on the target system. If I had done things right, you could just add the RPM in a directory and modify an XML file. However I have not done this right so it's not possible at the moment. There are several options, but I think the simplest is:
- add the RPM in Packages/ and regenerate the repodata + modify the installer code in install.img so that it installs it (look up 'xcp-ng-deps' and add the package name next to it)

LennertvdBerg

@stormi I’ll be back on Wednesday (just short holiday now), I’ll try your advice and see how it works

Rafisto

Hi, I've recently tackled the same issue and can confirm @stormi's steps are enough to get things running. I have built the installimg using https://github.com/xcp-ng/create-install-image, and included the mellanox driver by modifying packages.lst.

echo "mellanox-mlnxen-alt" >> configs/8.2/packages.lst

I then modified the /opt/xensource/installer/repository.py python script to include:

<     _targets = ['xcp-ng-deps']
---
>     _targets = ['xcp-ng-deps', 'mellanox-mlnxen-alt']

Such created ISO provides mellanox-mlnxen-alt in both the installer and the destination system.

olivierlambert

Excellent news! I suppose we should document it somewhere, ping @thomas-dkmt

stormi

Also note that there is a simpler approach if you don't need an "all-in-one" ISO image: driver disks.

You can build driver disks using https://github.com/xcp-ng/driver-disks and load them during the installation.

You can also ask us to provide driver disks for alternate drivers for which we haven't built them yet. We make them available on https://mirrors.xcp-ng.org/isos/drivers/8.x/...

That's also something we need to document in the official docs. It's mentioned in the release notes for 8.3 but not in other sections of the documentation.

Rafisto

The initial reason we've chosen to create an "all-in-one" ISO image was to simplify further configuration of an automated installation with PXE / Answerfile.

We tried driver disks before, though in our special case of mellanox-mlnxen-alt and hardware without any other NICs, I believe it always ends up requiring manual intervention. I haven't found a way to load the driver disk in the installer that is not connected to the network.

I think, even though my approach requires a bit more effort, it can still prove useful in some cases.